Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wsl 2 ubuntu 18.04 unable to connect to IP resources with Cisco Annyconnect active #4277

Open
eclay11 opened this issue Jul 6, 2019 · 224 comments
Labels
network wsl2 Issue/feature applies to WSL 2

Comments

@eclay11
Copy link

eclay11 commented Jul 6, 2019

Please fill out the below information:

  • Your Windows build number: (Type ver at a Windows Command Prompt)
    Microsoft Windows [Version 10.0.18932.1000]

  • What you're doing and what's happening: (Copy&paste the full set of specific command-line steps necessary to reproduce the behavior, and their output. Include screen shots if that helps demonstrate the problem.)
    I've installed ubuntu 18.04 and converted it to wsl v2 using the following command in windows powershell running as administrator.
    PS C:\WINDOWS\system32> wsl --set-version Ubuntu-18.04 2

When I start the Ubuntuu 18.04 instance I'm able to communicate with resources using IPs and hostnames.

eclay@wh-lpt-eclay:$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=49 time=115 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=49 time=95.4 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=49 time=77.3 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=49 time=92.6 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=49 time=90.6 ms
64 bytes from 8.8.8.8: icmp_seq=6 ttl=49 time=78.9 ms
64 bytes from 8.8.8.8: icmp_seq=7 ttl=49 time=84.1 ms
64 bytes from 8.8.8.8: icmp_seq=8 ttl=49 time=90.9 ms
64 bytes from 8.8.8.8: icmp_seq=9 ttl=49 time=80.7 ms
64 bytes from 8.8.8.8: icmp_seq=10 ttl=49 time=80.0 ms
^C
--- 8.8.8.8 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9081ms
rtt min/avg/max/mdev = 77.316/88.602/115.084/10.734 ms
eclay@wh-lpt-eclay:~$ trace
tracepath traceroute6 traceroute6.iputils tracerpt.exe
eclay@wh-lpt-eclay:$ tracepath 8.8.8.8
1?: [LOCALHOST] pmtu 1500
1: wh-lpt-eclay.mshome.net 0.497ms
1: wh-lpt-eclay.mshome.net 0.379ms
2: 10.179.10.1 2.890ms
3: 192.168.1.1 8.474ms
4: 192.168.1.1 10.395ms pmtu 1430
4: 172.26.96.161 71.223ms
5: 172.18.84.36 73.898ms asymm 6
6: 12.249.2.25 73.083ms asymm 7
7: 12.122.146.97 96.016ms asymm 14
8: 12.122.111.33 95.857ms asymm 13
9: sffca22crs.ip.att.net 95.209ms asymm 12
10: 12.122.163.61 90.288ms asymm 11
11: 12.255.10.236 103.555ms asymm 15
12: no reply
13: no reply
14: no reply
15: no reply
16: no reply
17: no reply
18: no reply
19: no reply
20: no reply
21: no reply
22: no reply
23: no reply
24: no reply
25: no reply
26: no reply
27: no reply
28: no reply
29: no reply
30: no reply
Too many hops: pmtu 1430 Resume: pmtu 1430

Once I connect to my work VPN using cisco annyconnect client v 4.7.00136 I'm not longer able to connect to IP resources.

eclay@wh-lpt-eclay:$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
8 packets transmitted, 0 received, 100% packet loss, time 7277ms

eclay@wh-lpt-eclay:$ tracepath 8.8.8.8
1?: [LOCALHOST] pmtu 1430
1: ??? 0.362ms
1: ??? 0.364ms
2: ??? 0.392ms pmtu 1200
2: no reply
3: no reply
4: no reply
5: no reply
6: no reply
7: no reply
8: no reply
9: no reply
10: no reply
11: no reply
12: no reply
13: no reply
14: no reply
15: no reply
16: no reply
17: no reply
18: no reply
19: no reply
20: no reply
21: no reply
22: no reply
23: no reply
24: no reply
25: no reply
26: no reply
27: no reply
28: no reply
29: no reply
30: no reply
Too many hops: pmtu 1200
Resume: pmtu 1200

  • What's wrong / what should be happening instead:
    I should be able to ping 8.8.8.8 or other outside IPs once the annyconnect vpn client is running.
@ghost
Copy link

ghost commented Jul 7, 2019

Not 100% sure, but it can be that annyconnecy blocks any dns server except provided by connection. I was observed something similar and was unable to disable this behaviour.

@eclay11
Copy link
Author

eclay11 commented Jul 7, 2019

I've tried the method of creating a manual named.conf and it didn't fix anything. I can ping the 1st dns server from windows but not from ubuntu.
Windows DNS servers after vpn connection established.

DNS Servers . . . . . . . . . . . : 172.24.0.25
8.8.8.8
C:\Users\eclay>ping 172.24.0.25

Pinging 172.24.0.25 with 32 bytes of data:
Reply from 172.24.0.25: bytes=32 time=131ms TTL=126
Reply from 172.24.0.25: bytes=32 time=103ms TTL=126
Reply from 172.24.0.25: bytes=32 time=109ms TTL=126

Ubuntu

$ ping 172.24.0.25
PING 172.24.0.25 (172.24.0.25) 56(84) bytes of data.

^C
--- 172.24.0.25 ping statistics ---
190 packets transmitted, 0 received, 100% packet loss, time 196530ms

It seems to be more of a routing problem than a name resolution. For instance I can ping www.microsoft.com from a cmd prompt but not from ubunutu.

windows

ping www.microsoft.com

Pinging e13678.dspb.akamaiedge.net [23.202.232.226] with 32 bytes of data:
Reply from 23.202.232.226: bytes=32 time=103ms TTL=51
Reply from 23.202.232.226: bytes=32 time=105ms TTL=51
Reply from 23.202.232.226: bytes=32 time=121ms TTL=51
Reply from 23.202.232.226: bytes=32 time=109ms TTL=51

Ping statistics for 23.202.232.226:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 103ms, Maximum = 121ms, Average = 109ms

Ubuntu

$ ping 23.202.232.226
PING 23.202.232.226 (23.202.232.226) 56(84) bytes of data.
^C
--- 23.202.232.226 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4176ms

@WsK
Copy link

WsK commented Aug 16, 2019

Having exactly same problem...Errrrr

@kobenauf
Copy link

kobenauf commented Aug 16, 2019

I have the same issue, that from within the WSL 2 shell, I'm unable to use AnyConnect running in Windows. However, in WSL I installed openconnect, and I run:
sudo openconnect --no-dtls -u myUsername vpn.myhost.com
Then I can access my company ip addresses from within WSL 2 sessions, but not from Windows, but this lets me at least ssh and run git, so that's an improvement.

Also, this only works if I disconnect the VPN in Windows first, which is annoying.

@chunkaichang
Copy link

Same issue here. I tried to add search list in /etc/resolv.conf followed the guide here but still failed.

@craigloewen-msft craigloewen-msft added network wsl2 Issue/feature applies to WSL 2 labels Aug 27, 2019
@craigloewen-msft
Copy link
Member

If you're still experiencing this issue could you please take some networking logs? It will help us diagnose what's going on.

You can find instructions on how to do so here!

@mcurole
Copy link

mcurole commented Sep 5, 2019

I'm definitely experiencing this issue and I'm running 18970. I can capture logs if needed. The problem is easy to reproduce.

@mcurole
Copy link

mcurole commented Sep 5, 2019

Well, cancel that. I don't have the Feedback Hub installed and I have another issue where I can't download app from the Microsoft Store because of some weird Microsoft Account issue that I'm working with support on.

@MichaelTong
Copy link

Also, if WSL 2 starts first, Cisco AnyConnect will fail to connect.

@MichaelTong
Copy link

I filed a feedback in FH. https://aka.ms/AA60j0u

@cmeiklejohn
Copy link

Same issue, I filed feedback using Feedback Hub.

@paulguerrero
Copy link

paulguerrero commented Oct 14, 2019

I also have the same issue with AnyConnect, also gave feedback in the Hub (although the option to post a recording was grayed out for me).

@neileadobe
Copy link

neileadobe commented Oct 31, 2019

I'm also having this problem and opened an issue for it in Feedback Hub with logs: https://aka.ms/AA6fthe

@uc-cjdavis
Copy link

I have the same issue, that from within the WSL 2 shell, I'm unable to use AnyConnect running in Windows. However, in WSL I installed openconnect, and I run:
sudo openconnect --no-dtls -u myUsername vpn.myhost.com
Then I can access my company ip addresses from within WSL 2 sessions, but not from Windows, but this lets me at least ssh and run git, so that's an improvement.

Also, this only works if I disconnect the VPN in Windows first, which is annoying.

I tried this route, but our AnyConnect uses 2FA which makes using openconnect impossible

@raisin-loaf
Copy link

I'm in the same boat. WSL 2 is a godsend while I'm in the office; it's so much faster and better integrated than using a VirtualBox (or similar) VM. Unfortunately I still have to use said VM when connecting remotely via VPN because, as others here have described, all network connectivity from WSL 2 stops the moment the VPN is up.

I use Cisco AnyConnect too although I imagine the problem is common to most VPN clients.

Like @haselton I'm unable to use OpenConnect as the company I work for enforces 2FA.

@craigloewen-msft - I'd dearly love to send you some logs but our workstation diagnostic data settings are locked down by Group Policy. If there's anything else I can do to help please let me know.

WSL 2 is fabulous. 🙂

@craigloewen-msft
Copy link
Member

@raisin-loaf thank you for the offer! I'll let you know if you can help out in any other way. We are investigating issues related to the VPN.

@AdonisLL
Copy link

AdonisLL commented Dec 4, 2019

I was able to resolve this by installing anyconnect pwa from the windows store the vpn connection now works with WSL when my stand alone installation of anyconnect 4.6 does not work... hope this helps someone

@raisin-loaf
Copy link

I was able to resolve this by installing anyconnect pwa from the windows store

It works! Thanks so much for posting this.

@dalgibbard
Copy link

dalgibbard commented Dec 10, 2019

I was having the same issue; i found that Anyconnect was setup to do full-tunnel, and therefore a route existed to throw all traffic through the VPN connection (likely including packets destined for the WSL2 vm).

WSL2 routes are configured with a metric of 5256, and anyconnect routes have a metric of 2, and therefore take priority. I amended the routes for anyconnect (where metric == 2) to be 5257 instead, which seems to largely work. I used "NetRouteView" run as Admin to do the changes because I was too lazy to find the powershell equivilent commands :)

This restored connectivity, but I found DNS to still be broken with the WSL default resolvconf (nameserver 172.xx.xx.x) so i dropped in nameserver 1.1.1.1 instead which seems OK. Will likely replace this with the work DNS for work resources to resolve nicely :)
and now full connectivity is working; no additional changes needed; but this likely will need applying each time I start anyconnect.

@chrisjones-brack3t
Copy link

Any movement on a fix for this? In my attempt to move away from macOS to Windows for dev, this has effectively stopped me completely as the company I work for uses AnyConnect.

@craigloewen-msft
Copy link
Member

Yes we are tracking this and are looking into solutions with the networking team! I'll post any updates here as soon as they become available.

@caal-15
Copy link

caal-15 commented Jan 10, 2020

Thanks a lot, this would be a life saver!

@kzeitz
Copy link

kzeitz commented Jan 18, 2020

For what it's worth I was having all the same issues described here.

I saw AdonisLL post and tried it. I installed the AnyConnect UWP version available on the Windows store. Now, it seems to work perfectly! No jacking around with resolv.conf, or turning off generateResolvCon in wsl.conf.

In fact the VPN seems much faster than the AnyConnect client 4.5 version I was using. Considering all the grief this was causing, this turned out to be a fast and easy fix for this!

@chrisjones-brack3t
Copy link

@kzeitz Company I work for requires 2FA authentication which it seems the Windows store version of AnyConnect doesn’t support.

@kzeitz
Copy link

kzeitz commented Jan 20, 2020 via email

@yaogee
Copy link

yaogee commented Oct 17, 2021

no one is assigned after all these struggles and time wasted.

@c3101
Copy link

c3101 commented Oct 18, 2021

no one is assigned after all these struggles and time wasted.

It's only been 2 years, I'm sure Microsoft is working on it. It's not like we pay for the produ.... oh, wait..

@tksingh-amr
Copy link

It (workaround) did not work for me either.

@balmeida-nokia
Copy link

@c3101
Copy link

c3101 commented Oct 20, 2021

That's nice, consolidated information, which still requires ADMINISTRATOR privileges.
Not everyone has that. Everyone with this problem however DO have a COMPANY MANAGED device which they are dependant on.
MICRO$OFT, ARE YOU LISTENING?

@balmeida-nokia
Copy link

@c3101 Is that comment about my workaround?
If so, which part needs admin privileges?

@NiklasBr
Copy link

Says right there in the linked Gist:

Run the following command in Powershell with Administrative permission.

@tksingh-amr
Copy link

tksingh-amr commented Oct 20, 2021

Try this workaround: https://gist.github.com/balmeida-nokia/122adf625c11c916902950e3255bd104
It doesn't work in my office laptop. I manually tried setting the metrics to 6000 as stated in the link. I set the nameserver as well in the resolve.conf file, but ping doesn't work. however I can ping to the name server from the Linux host, but not the outside world.

@balmeida-nokia
Copy link

Says right there in the linked Gist:

Run the following command in Powershell with Administrative permission.

Oh right... I forgot that part.

Try this workaround: https://gist.github.com/balmeida-nokia/122adf625c11c916902950e3255bd104

It doesn't work in my office laptop. I manually tried setting the metrics to 6000 as stated in the link. I set the nameserver as well in the resolve.conf file, but ping doesn't work. however I can ping to the name server from the Linux host, but not the outside world.

Pinging doesn't work with some AnyConnect settings. However, tcp (HTTP, ssh, etc... Using wget, for example) should work

@sakai135
Copy link

@c3101 Maybe this could work for you https://github.com/sakai135/wsl-vpnkit

@leosh64
Copy link

leosh64 commented Oct 26, 2021

After trying different things it does only work to me if I start WSL before activating AnyConnect VPN.

This also works for me, good workaround!

@mendrik
Copy link

mendrik commented Nov 4, 2021

is anybody going to fix this issue that has been open for more than 2 years on an OS I pay for?

same here, but what fixed it finally for me is using the anyconnect app from the ms store and then go to windows vpn settings and use the config like so:
image

@balmeida-nokia
Copy link

balmeida-nokia commented Nov 4, 2021

I partially fixed by using Anyconnect version 4.10.
I can now access all the internal (VPN "protected") addresses.
The only thing missing is having the data proxy working

@th0ger
Copy link

th0ger commented Nov 24, 2021

Has anyone been able to test the new "BypassVirtualSubnetsOnlyV4"/"BypassVirtualSubnetsOnlyV6" with Cisco AnyConnect 4.10?
Troubleshoot AnyConnect > Connectivity Issues with VM-based Subsystems

Did it fix the problem?
Is the setting BypassVirtualSubnetsOnlyV4 visible somewhere under %ProgramData%\Cisco\Cisco AnyConnect Secure Mobility Client\ so we can verify it?

@mashhood-ahmad
Copy link

Well, nothing else was working for me so I worked around this by converting WSL Ubuntu-18.04 to and from wsl2 a couple of times that eventually seems to fix everything for me at least.

Though I'm not sure about the real cause and not advocating this as a permanent solution. However, this seems to have worked for me and I can do coding within Windows again.

@prasadrajesh
Copy link

I had the same issue and resolved it with:

Get-NetIPInterface -InterfaceAlias "vEthernet (WSL)" | Set-NetIPInterface -InterfaceMetric 1
Get-NetAdapter | Where-Object {$_.InterfaceDescription -Match "Cisco AnyConnect"} | Set-NetIPInterface -InterfaceMetric 6000

after connecting my VPN and then restarting WSL. I also had to downgrade my Cisco AnyConnect to the version found here: https://olemiss.edu/helpdesk/vpn/windows.html

It's work for me. Thank you you saved my time.

@DJFormica
Copy link

After trying different things it does only work to me if I start WSL before activating AnyConnect VPN.

Thank you. I had to bang the rocks together for a while before I found this answer. I knew about the interface metric reset and setting the internal nameserver in resolv.conf, but somehow the order of execution had eluded me, probably because I'd done it randomly the first few times by chance.

I have additional issues in that my employer's implementation of the routing tables for AnyConnect is exceedingly strict, and includes all the IPv4 non-routable masks commonly used by the entire world for their home networks. (As frustrating as it is, they use them all, as we're a worldwide company with lots of internal networks. IPv6 plz.) I use RDP to connect to my work laptop in order to use my multi-monitor setup and custom ergonomic equipment, and after switching to a new work computer, there was a week of panic as I was having extreme difficulty connecting to the new machine. I would RDP in, establish the connection, and shortly thereafter, I'd get booted and the local UI would freeze up and require a hard power-down to re-establish access to the system. The prospect of having to program on a laptop keyboard was giving me RSI just thinking about it.

Should anyone else encounter something like this, I determined that I had to establish the VPN connection using the local UI, then establish the RDP connection, and everything would work fine. Adding WSL 2 to the mix, I have to log into the local UI using cached domain credentials, open Windows Terminal, launch WSL 2, open AnyConnect, establish the VPN connection, change the network interface metric (which, yes, I know I could probably now script using task scheduler and trigger on-event) and then RDP into the machine. But hey, at least Windows doesn't have to reboot all the time anymore 🤣

@toby-freemarket
Copy link

Here is a great blog post which deals with the 3 different VPN issues and why you should not change the Interface Metric to a high number.
https://janovesk.com/wsl/2022/01/24/wsl2-and-vpn-dns.html

@th0ger
Copy link

th0ger commented Mar 29, 2022

@tobyfreemarket very nice post. But this conclusion is close to wrong:

What’s not the fix?
I’ve seen people suggest a different “fix”. They set the InterfaceMetric of the VPN adapter to a very high value, like 6000. You can do that with a command like this Get-NetAdapter | Where-Object {$_.InterfaceDescription -Match "PANGP Virtual Ethernet Adapter"} | Set-NetIPInterface -InterfaceMetric 6000. This is not a good solution IMHO. A a high value of InterfaceMetric will prioritize every other network adapter over the VPN on the host machine. In effect, you are forcing ALL traffic to avoid the VPN, not just WSL 2! If that is an acceptable option, why not just turn the VPN off instead? Much easier (my emphasis).

  1. We use the InterfaceMetric workaround, which allows us to connect WSL2 to resources behind a VPN. Do I need to say, that if we close the VPN, we can't connect to the resources.
  2. The InterfaceMetric fix can be placed in a reuseable script. The method recommend in the blog requires several manual steps and is not trivially automated. not "much easier".

However, it would be interesting to automate the route-drop workaround, and place it in a scheduled task (as found earlier in this thread, specifically for Cisco VPN).

@dalgibbard
Copy link

dalgibbard commented Mar 29, 2022

Here is a great blog post which deals with the 3 different VPN issues and why you should not change the Interface Metric to a high number. https://janovesk.com/wsl/2022/01/24/wsl2-and-vpn-dns.html

That's great when you have one catch-all route; but in full-tunnel-default with some split-tunnel exclusions, or where the VPN client enforces/refreshes routes that are deleted, this isn't a suitable solution.

@th0ger
Copy link

th0ger commented Mar 29, 2022

We have activated the BypassVirtualSubnetsOnlyV4 fix of Cisco AnyConnect 4.10, but it did not appear to help. Anyone had success with that fix?

@dalgibbard
Copy link

We have activated the BypassVirtualSubnetsOnlyV4 fix of Cisco AnyConnect 4.10, but it did not appear to help. Anyone had success with that fix?

Probably one for Cisco support I guess, but based on this: https://www.cisco.com/c/en/us/td/docs/security/vpn_client/anyconnect/anyconnect410/administration/guide/b-anyconnect-admin-guide-4-10/troubleshoot-anyconnect.html#Cisco_Task_in_List_GUI.dita_3a9a8101-f034-4e9b-b24a-486ee47b5e9f
You have to create the custom attribute names, and assign them to a group policy; and then the version must be 4.10.01075 or newer.

@qinghuannn
Copy link

qinghuannn commented Aug 29, 2022

@marwatk here's the scripts I'm using. It's a bit of a mash right now, but should be simple to edit and setup your own.

works for me! Thank you!

@qwertycody
Copy link

I'm having this issue in December 2022 on an up to date version of Windows 10 with WSL 2 and the latest Ubuntu from Microsoft Store.

I agree with the others here that setting the metric high is not the answer - additionally these solutions do not consider developers that do not have administrative rights on their machine due to corporate policy.

Part of the issue I'm experiencing is the DNS issue where I have to manually set the DNS IP address in the /etc/resolv.conf

However, even after doing this sporadically I'll be unable to even ping an IP address directly leading me to believe that there still are some issues to be resolved with how the network adapters are handled by Windows.

Just wanted to share my experience.

P.S. - I wrote up an automatic powershell script to handle the DNS issue - but still have the other issue sporadically I described.

I guess I'll take 50% working over 0% working?

@th0ger
Copy link

th0ger commented Nov 30, 2022

We have activated the BypassVirtualSubnetsOnlyV4 fix of Cisco AnyConnect 4.10, but it did not appear to help. Anyone had success with that fix?

Probably one for Cisco support I guess, but based on this: https://www.cisco.com/c/en/us/td/docs/security/vpn_client/anyconnect/anyconnect410/administration/guide/b-anyconnect-admin-guide-4-10/troubleshoot-anyconnect.html#Cisco_Task_in_List_GUI.dita_3a9a8101-f034-4e9b-b24a-486ee47b5e9f You have to create the custom attribute names, and assign them to a group policy; and then the version must be 4.10.01075 or newer.

Finally the BypassVirtualSubnetsOnlyV4 fix from Cisco solved it for us! No more messing with interface metric workarounds.

@JorisNinja
Copy link

I had the same issue and resolved it with:

Get-NetIPInterface -InterfaceAlias "vEthernet (WSL)" | Set-NetIPInterface -InterfaceMetric 1
Get-NetAdapter | Where-Object {$_.InterfaceDescription -Match "Cisco AnyConnect"} | Set-NetIPInterface -InterfaceMetric 6000

after connecting my VPN and then restarting WSL. I also had to downgrade my Cisco AnyConnect to the version found here: https://olemiss.edu/helpdesk/vpn/windows.html

It's work for me. Thank you you saved my time.

This was actually my problem, Cisco VPN had connected status (even when disconnected). By chaging the priority of the Cisco one, the WSL2 variant started working again. Also changed my WiFi (primary interface) to priority of 1.

Thanks and you rock!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
network wsl2 Issue/feature applies to WSL 2
Projects
None yet
Development

No branches or pull requests