Can't pass traffic after upgrade to Ubuntu 18.04

[fitig]

Problem description

1. I ran openconnect-gp as follows: openconnect --protocol=gp <gp_gateway_name> --dump -vvv
2. I get connected (it shows connected on the firewall side) but can't pass any traffic (ping and any other traffic don't work)
3. An interested data point is that the firewall says I'm connected with Tunnel Type IPSec but the client reports "Connected as , using SSL
4. I do see "WARNING: Server asked us to submit HIP report" and I've tried adding --csd-wrapper hipreport.sh but that gives the same results
5. I did recompile just to be sure
6. I was able to connect without issue before my upgrade to 18.04
7. I'm able to connect via a Windows GP client from a VM running on the same machine

Operating system and openconnect-gp version

openconnect-gp version:

OpenConnect version v7.08-294-g5691d958
Using GnuTLS. Features present: PKCS#11, HOTP software token, TOTP software token, System keys, DTLS, ESP
Supported protocols: anyconnect (default), nc, gp

operating system

Linux lt-tfg 4.13.0-39-generic #44-Ubuntu SMP Thu Apr 5 14:25:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

GlobalProtect VPN information

POST https://<gp_gateway_name>/ssl-vpn/login.esp
Attempting to connect to server <gp_gateway_ip>:443
Connected to <gp_gateway_ip>:443
SSL negotiation with <gp_gateway_name>
Connected to HTTPS on <gp_gateway_name>
> POST /ssl-vpn/login.esp HTTP/1.1
> Host: <gp_gateway_name>
> User-Agent: PAN GlobalProtect
> X-Pad: 00000000000000000000
> Content-Type: application/x-www-form-urlencoded
> Content-Length: 172
> 
> jnlpReady=jnlpReady&ok=Login&direct=yes&clientVer=4100&prot=https:&clientos=linux-64&server=<gp_gateway_name>&computer=<local_computer_name>&user=<gp_user_name>&passwd=<gp_password>
Got HTTP response: HTTP/1.1 200 OK
Date: Wed, 02 May 2018 14:42:28 GMT
Content-Type: application/xml; charset=UTF-8
Content-Length: 660
Connection: keep-alive
ETag: "facd2-2346-5a3ffe8f"
Pragma: no-cache
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires: Thu, 19 Nov 1981 08:52:00 GMT
X-FRAME-OPTIONS: DENY
Set-Cookie: PHPSESSID=<PHPSESSID>; secure; HttpOnly
Set-Cookie: PHPSESSID=<PHPSESSID>; secure; HttpOnly
Set-Cookie: PHPSESSID=<PHPSESSID>; secure; HttpOnly
Set-Cookie: PHPSESSID=<PHPSESSID>; secure; HttpOnly
Set-Cookie: PHPSESSID=<PHPSESSID>; secure; HttpOnly
Set-Cookie: PHPSESSID=<PHPSESSID>; secure; HttpOnly
Set-Cookie: PHPSESSID=<PHPSESSID>; secure; HttpOnly
Set-Cookie: PHPSESSID=<PHPSESSID>; secure; HttpOnly
Set-Cookie: PHPSESSID=<PHPSESSID>; secure; HttpOnly
Set-Cookie: PHPSESSID=<PHPSESSID>; secure; HttpOnly
Set-Cookie: PHPSESSID=<PHPSESSID>; secure; HttpOnly
Set-Cookie: PHPSESSID=<PHPSESSID>; secure; HttpOnly
Set-Cookie: PHPSESSID=<PHPSESSID>; secure; HttpOnly
Set-Cookie: PHPSESSID=<PHPSESSID>; secure; HttpOnly
Set-Cookie: PHPSESSID=<PHPSESSID>; secure; HttpOnly
Set-Cookie: PHPSESSID=<PHPSESSID>; secure; HttpOnly
Set-Cookie: PHPSESSID=<PHPSESSID>; secure; HttpOnly
Set-Cookie: PHPSESSID=<PHPSESSID>; secure; HttpOnly
Set-Cookie: PHPSESSID=<PHPSESSID>; secure; HttpOnly
HTTP body length:  (660)
< <?xml version="1.0" encoding="utf-8"?><jnlp><application-desc><argument>(null)</argument><argument>e94079743888a9deeeae847b7fed3536</argument><argument>ede5c71d981767b7f46be458838d69bf8f703754</argument><argument>Client_gateway_split-N</argument><argument><gp_user_name></argument><argument>LDAP_then_Local-VPN_Users</argument><argument>vsys1</argument><argument><user_domain></argument><argument>(null)</argument><argument></argument><argument></argument><argument></argument><argument>tunnel</argument><argument>-1</argument><argument>4100</argument><argument></argument><argument></argument><argument></argument><argument></argument></application-desc></jnlp>
GlobalProtect login returned authentication-source=LDAP_then_Local-VPN_Users
POST https://<gp_gateway_name>/ssl-vpn/getconfig.esp
> POST /ssl-vpn/getconfig.esp HTTP/1.1
> Host: <gp_gateway_name>
> User-Agent: PAN GlobalProtect
> Cookie: PHPSESSID=<PHPSESSID>
> X-Pad: 00000000000000000000000000000000000000000000000000000000000
> Content-Type: application/x-www-form-urlencoded
> Content-Length: 261
> 
> client-type=1&protocol-version=p1&app-version=3.0.1-10&os-version=linux-64&clientos=linux-64&hmac-algo=sha1%2cmd5&enc-algo=aes-128-cbc%2caes-256-cbc&authcookie=<cookie>&portal=Client_gateway_split-N&user=<gp_user_name>&domain=<user_domain>
Got HTTP response: HTTP/1.1 200 OK
Date: Wed, 02 May 2018 14:42:28 GMT
Content-Type: application/xml; charset=UTF-8
Content-Length: 1827
Connection: keep-alive
ETag: "faccc-1f3-5a3ffe8f"
Pragma: no-cache
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires: Thu, 19 Nov 1981 08:52:00 GMT
X-FRAME-OPTIONS: DENY
HTTP body length:  (1827)
< 
< 	<response status="success">
< 		<need-tunnel>yes</need-tunnel>
< 		<ssl-tunnel-url>/ssl-tunnel-connect.sslvpn</ssl-tunnel-url>
< 		<portal>Client_gateway_split-N</portal>
< 		<user><gp_user_name></user>
< 		<lifetime>691200</lifetime>
< 		<timeout>10800</timeout>
< 		<disconnect-on-idle>10800</disconnect-on-idle>
< 		<bw-c2s>1000</bw-c2s>
< 		<bw-s2c>1000</bw-s2c>
< 		<gw-address><gp_gateway_ip></gw-address>
< 		<ip-address><user_ip></ip-address>
< 		<netmask>255.255.255.255</netmask>
< 		<dns>
< 			<member><dns_server_1></member>
< 			<member><dns_server_2></member>
< 		</dns> 
< 		<wins>
< 		</wins> 
< 		<dns-suffix>
< 			<member><user_domain>.lan</member>
< 		</dns-suffix> 
< 		<default-gateway><user_ip></default-gateway>
< 		<mtu>0</mtu>
< 		<no-direct-access-to-local-network>no</no-direct-access-to-local-network>
< 		<access-routes>
< 			<member><access_route_1></member>
< 			<member><access_route_2></member>

< 			<member><dns_server_1>/32</member>
< 			<member><dns_server_2>/32</member>
< 		</access-routes> 
< 		<exclude-access-routes>
< 		</exclude-access-routes> 
< 		<ipsec>
< 			<udp-port>4501</udp-port>
< 			<ipsec-mode>esp-tunnel</ipsec-mode>
< 			<enc-algo>aes-128-cbc</enc-algo>
< 			<hmac-algo>sha1</hmac-algo>
< 			<c2s-spi>0x750C1E99</c2s-spi>
< 			<s2c-spi>0xE5464831</s2c-spi>
< 			<akey-s2c>
< 				<bits>160</bits>
< 				<val><key></val>
< 			</akey-s2c> 
< 			<ekey-s2c>
< 				<bits>128</bits>
< 				<val><key></val>
< 			</ekey-s2c> 
< 			<akey-c2s>
< 				<bits>160</bits>
< 				<val><key></val>
< 			</akey-c2s> 
< 			<ekey-c2s>
< 				<bits>128</bits>
< 				<val><key></val>
< 			</ekey-c2s> 
< 		</ipsec> 
< 	</response>
Tunnel timeout (rekey interval) is 180 minutes.
TCP_INFO rcv mss 1460, snd mss 1460, adv mss 1460, pmtu 1500
No MTU received. Calculated 1422 for ESP tunnel
POST https://<gp_gateway_name>/ssl-vpn/hipreportcheck.esp
> POST /ssl-vpn/hipreportcheck.esp HTTP/1.1
> Host: <gp_gateway_name>
> User-Agent: PAN GlobalProtect
> Cookie: PHPSESSID=<PHPSESSID>
> X-Pad: 0000000000000000000000000000000000000
> Content-Type: application/x-www-form-urlencoded
> Content-Length: 219
> 
> client-role=global-protect-full&authcookie=<cookie>&portal=Client_gateway_split-N&user=<gp_user_name>&domain=<user_domain>&computer=<local_computer_name>&client-ip=<user_ip>&md5=9186c68981cdeae3e747fc387224189a
Got HTTP response: HTTP/1.1 200 OK
Date: Wed, 02 May 2018 14:42:28 GMT
Content-Type: application/xml; charset=UTF-8
Content-Length: 107
Connection: keep-alive
ETag: "faccf-6a6-5a3ffe8f"
X-Content-Type-Options: nosniff
Pragma: no-cache
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Content-Security-Policy: default-src 'self'
Expires: Thu, 19 Nov 1981 08:52:00 GMT
X-FRAME-OPTIONS: DENY
HTTP body length:  (107)
< 
< 	<response status="success">
< 		<hip-report-needed>yes</hip-report-needed>
< 		<delay>0</delay>
< 	</response>
Gateway says HIP report submission is needed.
POST https://<gp_gateway_name>/ssl-vpn/hipreport.esp
> POST /ssl-vpn/hipreport.esp HTTP/1.1
> Host: <gp_gateway_name>
> User-Agent: PAN GlobalProtect
> Cookie: PHPSESSID=<PHPSESSID>
> X-Pad: 0000000000000000000000000000000
> Content-Type: application/x-www-form-urlencoded
> Content-Length: 7265
> 
> client-role=global-protect-full&authcookie=<cookie>&portal=Client_gateway_split-N&user=<gp_user_name>&domain=<user_domain>&computer=<local_computer_name>&client-ip=<user_ip>&report=<hip_report>
Got HTTP response: HTTP/1.1 200 OK
Date: Wed, 02 May 2018 14:42:28 GMT
Content-Type: application/xml; charset=UTF-8
Content-Length: 75
Connection: keep-alive
ETag: "facce-632-5a3ffe8f"
X-Content-Type-Options: nosniff
Pragma: no-cache
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Content-Security-Policy: default-src 'self'
Expires: Thu, 19 Nov 1981 08:52:00 GMT
X-FRAME-OPTIONS: DENY
HTTP body length:  (75)
< 
< 	<response status="success">
< 		<notification></notification>
< 	</response>
HIP report submitted successfully.
Parameters for incoming ESP: SPI <spi>
ESP encryption type AES-128-CBC (RFC3602) key <key>
ESP authentication type HMAC-SHA-1-96 (RFC2404) key <key>
Parameters for outgoing ESP: SPI <spi>
ESP encryption type AES-128-CBC (RFC3602) key <key>
ESP authentication type HMAC-SHA-1-96 (RFC2404) key <key>
Send ESP probes
Connected as <user_ip>, using SSL

[dlenski]

What happens after the part that you've logged?

Does the ESP tunnel also connect successfully, or does the connection stay SSL-only?

And does the vpnc-script run successfully? All of the routing setup is done by the script, not by the openconnect binary.

[fitig]

It hangs right where I ended the log. If I do a control-C I see the following:

^CFailed to spawn script '/usr/share/vpnc-scripts/vpnc-script' for connect: Interrupted system call
Send ESP probes
Received ESP packet of 84 bytes
Accepting later-than-expected ESP packet with seq 2 (expected 0)
ESP session established with server
Received ESP packet of 84 bytes
Accepting out-of-order ESP packet with seq 1 (expected 3)
Received ESP packet of 84 bytes
Accepting later-than-expected ESP packet with seq 3 (expected 2)
ESP tunnel connected; exiting HTTPS mainloop.
POST https://<gp_gateway_dns_name>/ssl-vpn/logout.esp
SSL negotiation with <gp_gateway_dns_name>
Connected to HTTPS on <gp_gateway_dns_name>

POST /ssl-vpn/logout.esp HTTP/1.1
Host: <gp_gateway_dns_name>
User-Agent: PAN GlobalProtect
Cookie: PHPSESSID=2efe123b2f9c5a55605f3a0fa1a5ed23
X-Pad: 0000000000000000000000000000000000000000000000000000000000000000
Content-Type: application/x-www-form-urlencoded
Content-Length: 128

computer=lt-tfg&authcookie=&portal=Client_gateway_split-N&user=<gp_user_name>&domain=<user_domain>
Got HTTP response: HTTP/1.1 200 OK
Date: Wed, 02 May 2018 15:01:23 GMT
Content-Type: application/xml; charset=UTF-8
Content-Length: 293
Connection: keep-alive
ETag: ""
Pragma: no-cache
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires: Thu, 19 Nov 1981 08:52:00 GMT
X-FRAME-OPTIONS: DENY
HTTP body length: (293)
<
<
<
< Client_gateway_split-N
< <user_domain>
< <gp_user_name>
< <local_computer_name>
<
<
<
Logout successful
RTNETLINK answers: No such process
RTNETLINK answers: No such process
User canceled (SIGINT); exiting;

[dlenski]

Looks like you've got your answer. Something is preventing the vpnc-script from running.

This likely has nothing to do with the GlobalProtect patches in this version of openconnect.

Things to check:

1. Is the vpnc-script actually installed at the location stated?
2. Does the Ubuntu-distributed open connect package have the same problem when connecting to an anyconnect VPN, if you have access to one?
3. Is this bug reported on Ubuntu Launchpad for the open connect or vpnc-script packages?

[maltris]

Hello,

I experienced the same problem and solved it by disabling systemd-resolved and unlinking /etc/resolv.conf, recreating it manually. I suspect this has something to do with resolvconf/systemd-resolved and DNS.

@github-t: Can you try to reproduce my problem?

[irockel]

I have the same problem, if I kill this line

run-parts --arg=-a --arg=tun0 /etc/resolvconf/update.d

from ps the connection succeeds.

This seems to be the related Ubuntu Issue:

https://bugs.launchpad.net/ubuntu/+source/openconnect/+bug/1752411

There's also a working workaround:

/usr/lib/avahi/avahi-daemon-check-dns.sh : dns_has_local()
  OUT=\`LC_ALL=C /usr/bin/timeout 5 host -t soa local. 2>&1\`

[vshalts]

I can confirm both the problem and that last solution with kill of run-parts work great. At least as temporary workaround solution. @irockel Thanks!

[fitig]

Can confirm that @irockel 's workaround fixes it for me.

[dlenski]

Thanks @maltris and @irockel for identifying the root cause and workarounds.

I'll close this one as "not OpenConnect's fault" and point anyone else to this if it comes up again.

I hope Ubuntu will fix this soon.