Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

While connected to VPN the curl command hangs up #4517

Closed
Darkside81 opened this issue Sep 17, 2019 · 35 comments
Closed

While connected to VPN the curl command hangs up #4517

Darkside81 opened this issue Sep 17, 2019 · 35 comments
Labels

Comments

@Darkside81
Copy link

Darkside81 commented Sep 17, 2019

OK: From our office and from home remotely, we can use the VPN connection, curl succeeds.
OK: While visiting our customer's office, without using the VPN connection, curl also succeeds.
FAIL: While visiting our customer's office, when using the VPN connection, curl hangs as follows:

$ curl ifconfig.co; curl -vvv https://testlink.limelightbox.com 
198.199.107.195 
* Rebuilt URL to: https://testlink.limelightbox.com/ 
* Trying 52.39.78.238... 
* TCP_NODELAY set 
* Connected to testlink.limelightbox.com (52.39.78.238) port 443 (#0) 
* ALPN, offering h2 
* ALPN, offering http/1.1 
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH 
* successfully set certificate verify locations: 
* CAfile: /etc/ssl/cert.pem 
CApath: none 
* TLSv1.2 (OUT), TLS handshake, Client hello (1):

Above simply hangs.

Can someone assist with advice

@craigloewen-msft
Copy link
Member

What build of Windows are you using? And are you using WSL 1 or WSL 2?

@therealkenc
Copy link
Collaborator

Also maybe try Windows curl.exe with the VPN up and see if that takes, just for the datapoint. Eliminates WSL if it doesn't work. Points finger at WSL if it does.

@ghost
Copy link

ghost commented Sep 25, 2019

This issue has been automatically marked as stale because it has marked as requiring author feedback but has not had any activity for 7 days. It will be closed if no further activity occurs within 7 days of this comment. Thank you for your contributions to WSL!

@zot
Copy link

zot commented Oct 1, 2019

I'm having a similar issue with our VPN.

Git and SSH both hang over the VPN from WSL but both work just fine from within Windows.

Windows version: Microsoft Windows [Version 10.0.18990.1]

The VPN client is Shrew Soft standard edition version 2.2.2.

By the way, I've been using this VPN client version with a VMware Linux VM for years just fine, NATing through windows.

@Darkside81
Copy link
Author

I'm using WSL 2

@craigloewen-msft
Copy link
Member

@Darkside81 and @zot could you please take networking logs of the issue that you're seeing? Instructions on how to do so are here. Please comment back on this thread with a link to your feedback item and we can take a look at what might be causing it! Thanks!

@zot
Copy link

zot commented Oct 7, 2019

OK, here's the link.

In the test, I was able to make a regular HTTP connection (we don't have HTTPS running internally) through our VPN but I could not SSH.

@zot
Copy link

zot commented Oct 10, 2019

I'm able to use services on our VPN by port-forwarding with SSH from within Windows-land. Maybe other people with this problem can use a similar technique. Here's my script (run it from git-bash/msys/cygwin):

#!/bin/sh
MAIL_HOST=YOUR-MAIL-HOST
SSH_USER=YOUR-USER-NAME
SSH_HOST=YOUR-VPN-SSH-HOST
ADDR=$(ipconfig|sed '/WSL/ {N;N;N;N;s/\n//g;s/^.*: \([^:]*\)$/\1/;p};d')
echo $ADDR
exec ssh -L $ADDR:11000:$MAIL_HOST:110 -L $ADDR:2500:$MAIL_HOST:25 $SSH_USER@$SSH_HOST

It just finds the ip addr of the WSL adapter, prints it out, and uses SSL to port-forward 11000->110 and 2500->25 on it.

Configure your Linux email client to use the printed IP address and port 11000 for pop, 2500 for outgoing.

Note that the WSL switch changes IP every time you boot: #4150

I have a second workaround for this, I created a NAT switch Hyper-V Manager and I actually use that instead of the WSL adapter (my script uses Ethernet NAT instead of WSL). It has a static IP, so it doesn't change every time I boot.

@timesnewmen
Copy link

I'm able to use services on our VPN by port-forwarding with SSH from within Windows-land. Maybe other people with this problem can use a similar technique. Here's my script (run it from git-bash/msys/cygwin):

#!/bin/sh
MAIL_HOST=YOUR-MAIL-HOST
SSH_USER=YOUR-USER-NAME
SSH_HOST=YOUR-VPN-SSH-HOST
ADDR=$(ipconfig|sed '/WSL/ {N;N;N;N;s/\n//g;s/^.*: \([^:]*\)$/\1/;p};d')
echo $ADDR
exec ssh -L $ADDR:11000:$MAIL_HOST:110 -L $ADDR:2500:$MAIL_HOST:25 $SSH_USER@$SSH_HOST

It just finds the ip addr of the WSL adapter, prints it out, and uses SSL to port-forward 11000->110 and 2500->25 on it.

Configure your Linux email client to use the printed IP address and port 11000 for pop, 2500 for outgoing.

Note that the WSL switch changes IP every time you boot: #4150

I have a second workaround for this, I created a NAT switch Hyper-V Manager and I actually use that instead of the WSL adapter (my script uses Ethernet NAT instead of WSL). It has a static IP, so it doesn't change every time I boot.

How can you set the WSL VM use the "NAT Switch" you have created?

@zot
Copy link

zot commented Oct 11, 2019

I'm able to use services on our VPN by port-forwarding with SSH from within Windows-land. Maybe other people with this problem can use a similar technique. Here's my script (run it from git-bash/msys/cygwin):

#!/bin/sh
MAIL_HOST=YOUR-MAIL-HOST
SSH_USER=YOUR-USER-NAME
SSH_HOST=YOUR-VPN-SSH-HOST
ADDR=$(ipconfig|sed '/WSL/ {N;N;N;N;s/\n//g;s/^.*: \([^:]*\)$/\1/;p};d')
echo $ADDR
exec ssh -L $ADDR:11000:$MAIL_HOST:110 -L $ADDR:2500:$MAIL_HOST:25 $SSH_USER@$SSH_HOST

It just finds the ip addr of the WSL adapter, prints it out, and uses SSL to port-forward 11000->110 and 2500->25 on it.
Configure your Linux email client to use the printed IP address and port 11000 for pop, 2500 for outgoing.
Note that the WSL switch changes IP every time you boot: #4150
I have a second workaround for this, I created a NAT switch Hyper-V Manager and I actually use that instead of the WSL adapter (my script uses Ethernet NAT instead of WSL). It has a static IP, so it doesn't change every time I boot.

How can you set the WSL VM use the "NAT Switch" you have created?

I'm not setting the WSL VM to use the NAT switch. I'm using a vanilla WSL 2 Ubuntu setup except that I'm limiting the memory usage (currently to 8G). The NAT switch just happens to work for forwarding because it has a static IP and it's part of my Windows setup (I actually used the NAT switch earlier with a Hyper-V VM that I don't use anymore).

@zot
Copy link

zot commented Oct 16, 2019

How can you set the WSL VM use the "NAT Switch" you have created?

Maybe my last comment wasn't so helpful. I just used a virtual switch I already had lying around. It's not actually connected to WSL 2 or any other VM. To make one, open the Hyper-V Manager and click "Virtual Switch Manager...". Then create a new virtual switch like this, hooking it into your ethernet or wifi network, whichever one you use:

image

Once you've created it, use ipconfig to see the IP address Windows has assigned to it. Mine uses a dynamic IP but Windows has given it the same IP every time I've booted, so far.

I tried using Default Switch, because I wanted it to work with both ethernet and wifi but that didn't work for me. Once you've made the switch, make your SSH tunnel listen on its address, like in my script, above.

@giggio
Copy link

giggio commented Apr 26, 2020

I'm having the exact same problem, I can't work when connected to my company network. The network does not seem to work in some scenarios.
DNS resolution works.
Curl does not work, either with http or https.

This is what I get on http:

❯ curl --verbose http://www.google.com
*   Trying 172.217.29.132:80...
* TCP_NODELAY set
* Connected to www.google.com (172.217.29.132) port 80 (#0)
> GET / HTTP/1.1
> Host: www.google.com
> User-Agent: curl/7.65.3
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

On https the log is the same as from this issue, but I'm on TLS 1.3:

❯ curl --verbose https://www.google.com
*   Trying 172.217.29.132:443...
* TCP_NODELAY set
* Connected to www.google.com (172.217.29.132) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.google.com:443
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.google.com:443

I'll report the logs as explained earlier, as well.

I'm on Windows Version 10.0.19041.207, WSL 2, Ubuntu 18.10 (but it also repros on Suse).

curl.exe works on Windows normally.

curl works when the vpn is not being used.

@kaskii
Copy link

kaskii commented Apr 28, 2020

I have same issue as @giggio except that my curl works for public addresses such as google.com. Any internal resource inside company private network I get the same behavior as @giggio. Windows side works fine. Check Point Mobile VPN client opened in Windows that to my knowledge should route ALL traffic via company network.

I made the feedback as @craigloewen-msft instructed and it can be found here. Hope it helps.

@FallenMax
Copy link

I'm having the same issue with WSL2+VPN:

  • ping company website domain - ok
  • curl/wget company website from wsl2 - hangs
  • curl/wget external website from wsl2 - ok
  • curl/wget company website from cmd.exe - ok

To get around this problem, I'm redirecting http(s)/ssh requests in wsl2 through a proxy set up in Windows-land, and curl/wget/git are working properly now. Hope it helps.

@giggio
Copy link

giggio commented Apr 30, 2020

@FallenMax how did you do that? Please share, that would be very useful to me right now.

@FallenMax
Copy link

FallenMax commented Apr 30, 2020

@giggio Here's my setup:

1. Setup a HTTP or SOCKS5 proxy in Windows-land

I'm using cow, as I already have it installed (you can use whatever proxy software works). My cow config rc.txt is as below:

listen = http://0.0.0.0:7777

It does nothing other than proxying traffic from port 7777 to Windows network.

2. Setup curl/wget/git in WSL2 to use the Windows proxy above

  • for curl/wget, I'm using proxychains-ng, a software that forces any linux program use specified proxy. But if you only need curl or wget, just setting up http_proxy and https_proxy environment variable should be enough. e.g. export http_proxy=http://${your windows local IP}:7777
  • for git (that uses ssh protocol), add these config to ~/.ssh/config:
Host git.yourcompany.com
User git
ProxyCommand nc -X connect -x 192.168.2.171:7777  %h %p

replace git.yourcompany.com with your company's Gitlab host, and replace 192.168.2.171 with your Windows machine's local IP (see ipconfig.exe) and it's done.

@giggio
Copy link

giggio commented May 1, 2020

Thanks, @FallenMax, this works perfectly.
Cow is a cool project, it's a pity I can't understand most of it because of the Chinese. But it will solve other issues for me, so thanks for that too.

I hope someone from the WSL team is able to investigate this in the meantime.

giggio added a commit to giggio/bashscripts that referenced this issue May 1, 2020
@FallenMax
Copy link

@giggio Thank you, glad it helps! English version of Cow's README.md and example config are available if needed.

@neseih
Copy link

neseih commented May 14, 2020

May this could be a fix for you:
Set MTU to the value of the VPN interface:
sudo ifconfig eth0 mtu 1350

@giggio
Copy link

giggio commented May 14, 2020

@neseih this actually solves the problem. What exactly is going on? How do I find out the 1350 number, is it on Windows somewhere? And thanks for that.

@neseih
Copy link

neseih commented May 14, 2020

You could type in cmd.exe netsh interface ipv4 show subinterfaces and look at the leftmost column MTU.

It seems like I found out in packet dumps the ssl handshake is fragmented and gets dropped for some reason.

@giggio
Copy link

giggio commented May 14, 2020

Cool. I'm trying now to set it permanently, but things seem a little different from a default Ubuntu installation. And as this needs to run as root, I can't add it to my init scripts.
I also found out that mtu 1400 also works for me. That is the value set in my VPN, my LAN works at 1500.

@giggio
Copy link

giggio commented May 14, 2020

I went the other way around, and found out that my VPN accepts 1500. I set it on Windows to 1500, but Ubuntu keeps failing, unless I set it to 1400. I have no idea why.
This article explains how to set the MTU for a VPN on Windows, and how to test for the correct MTU: https://support.purevpn.com/how-to-change-mtu-value-on-windows-xp-vista-7-and-8

@zot
Copy link

zot commented May 14, 2020

May this could be a fix for you:
Set MTU to the value of the VPN interface:
sudo ifconfig eth0 mtu 1350

Fantastic, works for me! Thanks!

@MoeRT09
Copy link

MoeRT09 commented May 27, 2020

Hello,

just jumping into the discussion as I'm experiencing the same problem while connecting via SSH over a VPN connection.

Setting the MTU manually can't be the solution to this issue. In the screenshot below you can see a traffic capture on the WSL interface as well as the VPN interface. Up until packet 29 everything lines up on both interfaces. At packet 30, fragmentation happens. This is exactly the point where my SSH connection froze (just launching htop). As you can see, fragmented packets don't get forwarded to the WSL interface which causes this issue.

Clipboard 1

A small addition, as @kaskii mentioned CheckPoint VPN. Im using CheckPoint Capsule VPN for Windows 10. I haven't yet experience the problem with an OpenVPN or Wireguard connection. What's interesting is, that, at least in my case, the don't fragment bit is set in the IPv4 header, nonetheless, the packet gets fragmented (according to RFC 791, in this case, the packet should be dropped, which finally happens and the sender should be notified via ICMP, which isn't happening). So maybe CheckPoint is doing something against the specification.

Second update:
Just traced the same situation over a Wireguard tunnel tunneled through an OpenVPN tunnel. And in this situation, the MTU gets correctly honored by WSL (no fragmentation happens, thus no problems). Same with an OpenVPN only tunnel.
I can spot the following differences:

  • Both, Wireguard (using TunSafe client) and OpenVPN use a "TAP-Windows Adapter" on the Windows side
  • CheckPoint uses a "WAN-Miniport (SSTP)" interface on the Windows side

The output of netsh.exe interface ipv4 show subinterface shows the correct MTU set for the CheckPoint Interface as well as the OpenVPN/Wireguard interfaces. So for some reason, the MTU for the OpenVPN/Wireguard is correctly honored by WSL whilst this is not the case for the CheckPoint interface.
Maybe @zot can point out what kind of interface the ShrewSoft client uses on Windows side. If it is a "WAN-Minport" interface as well, this might indicate that the problem can be tracked down to the MTU of those kind of interfaces not being correctly reported to WSL.

@zot
Copy link

zot commented May 31, 2020

I don't see an extra network interface, it looks to me like Shrew Soft installs a "Shrew Soft Lightweight Filter" into the current network adapter (properties show up in my network bridge). I don't know the command to list this so I'm using the network adapter GUI and it looks like this:

image

@erikmack
Copy link

erikmack commented Jul 13, 2020

WSL2 hangs indefinitely when I try to manually set MTU to 1400 with 'sudo ip link set dev eth0 mtu 1400' and a full reboot is necessary. This hang occurs whether or not I'm connected to our Pulse VPN.

+1 for this issue. When connected to our Pulse VPN many network connections hang, certainly due to the still-1500 MTU in the WSL2 environment.

Oddly I can sometimes work around the issue by exec'ing into a Debian docker container from the Debian WSL2 environment and doing my network operation there. I'm not sure why this works, since the interface in the container also maintains a 1500 MTU. I suppose the packets from Docker container don't actually traverse the interface from the WSL2 kernel.

@erikmack
Copy link

Since getting the latest kernel I'm now able to set MTU successfully.

@giggio
Copy link

giggio commented Jul 23, 2020

@erikmack what is your new kernel version? How did you get the new version?

@erikmack
Copy link

It's a manual .msi and download from here. I'm currently on

erikm@erikm-pc2:~$ uname -a
Linux erikm-pc2 4.19.104-microsoft-standard #1 SMP Wed Feb 19 06:37:35 UTC 2020 x86_64 GNU/Linux

@victorsferreira
Copy link

I still have this problem with Checkpoint VPN

@alex20465
Copy link

This has nothing to do with WSL I have exact the same issue with a native ubuntu installation:

CURL / SSH every IPv4 connection hangs:

*   Trying 142.250.185.228:443...
* TCP_NODELAY set
* Connected to www.google.com (142.250.185.228) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):

Other IPv6 supported connections are successfully communicating.

@gh0sthx
Copy link

gh0sthx commented Mar 31, 2022

Cool. I'm trying now to set it permanently, but things seem a little different from a default Ubuntu installation. And as this needs to run as root, I can't add it to my init scripts. I also found out that mtu 1400 also works for me. That is the value set in my VPN, my LAN works at 1500.

So how do you set it permanently?

@RioChndr
Copy link

May this could be a fix for you:
Set MTU to the value of the VPN interface:
sudo ifconfig eth0 mtu 1350

This works for me too.

my kernel

uname -a
Linux RioChndr-Lenovo-Legion 5.10.16.3-microsoft-standard-WSL2 #1 SMP Fri Apr 2 22:23:49 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

I make it permanent by add this line to /etc/wsl.conf

[boot]
command = sudo ifconfig eth0 mtu 1350

Copy link
Contributor

This issue has been automatically closed since it has not had any activity for the past year. If you're still experiencing this issue please re-file this as a new issue or feature request.

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests