Curl between vpncloud nodes getting RST? #206

nivekuil · 2021-06-28T05:34:06Z

nivekuil
Jun 28, 2021

Hi, I'm trying out vpncloud and noticed a strange behavior: curling an API works when the response is small but hangs and receives an RST when it's bigger. Both hosts are part of a docker swarm connected via vpncloud, connecting through an overlay network. The MTU on vpncloud0 is 1427. Not entirely sure vpncloud is to blame here as most other things seem to work, but I haven't seen this behavior before in any other circumstance.

bash-5.0# curl volume.lax-1:8080/ui/index.html -vvvvvv
*   Trying 10.0.2.56:8080...
* Connected to volume.lax-1 (10.0.2.56) port 8080 (#0)
> GET /ui/index.html HTTP/1.1
> Host: volume.lax-1:8080
> User-Agent: curl/7.73.0
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

server tcpdump

04:43:10.622309 IP 10.0.2.59.42528 > 10.0.2.56.8080: Flags [S], seq 888892677, win 28200, options [mss 1410,sackOK,TS val 223684749 ecr 0,nop,
wscale 7], length 0
04:43:10.622358 IP 10.0.2.56.8080 > 10.0.2.59.42528: Flags [S.], seq 2707244319, ack 888892678, win 27960, options [mss 1410,sackOK,TS val 376909812 ecr 223684749,nop,wscale 7], length 0
04:43:10.622643 IP 10.0.2.59.42528 > 10.0.2.56.8080: Flags [.], ack 1, win 221, options [nop,nop,TS val 223684750 ecr 376909812], length 0
04:43:10.622912 IP 10.0.2.59.42528 > 10.0.2.56.8080: Flags [P.], seq 1:95, ack 1, win 221, options [nop,nop,TS val 223684750 ecr 376909812], length 94: HTTP: GET /ui/index.html HTTP/1.1
04:43:10.622920 IP 10.0.2.56.8080 > 10.0.2.59.42528: Flags [.], ack 95, win 219, options [nop,nop,TS val 376909813 ecr 223684750], length 0
04:43:10.623260 IP 10.0.2.56.8080 > 10.0.2.59.42528: Flags [P.], seq 1:4097, ack 95, win 219, options [nop,nop,TS val 376909813 ecr 223684750], length 4096: HTTP: HTTP/1.1 200 OK
04:43:10.623503 IP 10.0.2.56.8080 > 10.0.2.59.42528: Flags [P.], seq 4097:4638, ack 95, win 219, options [nop,nop,TS val 376909814 ecr 223684750], length 541: HTTP
04:43:10.623733 IP 10.0.2.59.42528 > 10.0.2.56.8080: Flags [.], ack 1, win 229, options [nop,nop,TS val 223684751 ecr 376909813,nop,nop,sack 1 {4097:4638}], length 0
04:43:10.623751 IP 10.0.2.56.8080 > 10.0.2.59.42528: Flags [.], seq 1:1399, ack 95, win 219, options [nop,nop,TS val 376909814 ecr 223684751], length 1398: HTTP: HTTP/1.1 200 OK
04:43:10.825502 IP 10.0.2.56.8080 > 10.0.2.59.42528: Flags [.], seq 1:1399, ack 95, win 219, options [nop,nop,TS val 376910016 ecr 223684751], length 1398: HTTP: HTTP/1.1 200 OK
04:43:11.233498 IP 10.0.2.56.8080 > 10.0.2.59.42528: Flags [.], seq 1:1399, ack 95, win 219, options [nop,nop,TS val 376910424 ecr 223684751], length 1398: HTTP: HTTP/1.1 200 OK
04:43:12.073494 IP 10.0.2.56.8080 > 10.0.2.59.42528: Flags [.], seq 1:1399, ack 95, win 219, options [nop,nop,TS val 376911264 ecr 223684751], length 1398: HTTP: HTTP/1.1 200 OK
04:43:13.737519 IP 10.0.2.56.8080 > 10.0.2.59.42528: Flags [.], seq 1:1399, ack 95, win 219, options [nop,nop,TS val 376912928 ecr 223684751], length 1398: HTTP: HTTP/1.1 200 OK
04:43:17.001504 IP 10.0.2.56.8080 > 10.0.2.59.42528: Flags [.], seq 1:1399, ack 95, win 219, options [nop,nop,TS val 376916192 ecr 223684751], length 1398: HTTP: HTTP/1.1 200 OK
04:43:23.593517 IP 10.0.2.56.8080 > 10.0.2.59.42528: Flags [.], seq 1:1399, ack 95, win 219, options [nop,nop,TS val 376922784 ecr 223684751], length 1398: HTTP: HTTP/1.1 200 OK
04:43:36.905493 IP 10.0.2.56.8080 > 10.0.2.59.42528: Flags [.], seq 1:1399, ack 95, win 219, options [nop,nop,TS val 376936096 ecr 223684751], length 1398: HTTP: HTTP/1.1 200 OK
04:43:40.624716 IP 10.0.2.56.8080 > 10.0.2.59.42528: Flags [F.], seq 4638, ack 95, win 219, options [nop,nop,TS val 376939815 ecr 223684751], length 0
04:43:40.625235 IP 10.0.2.59.42528 > 10.0.2.56.8080: Flags [.], ack 1, win 229, options [nop,nop,TS val 223714753 ecr 376909813,nop,nop,sack 1 {4097:4639}], length 0
04:43:40.625273 IP 10.0.2.56.8080 > 10.0.2.59.42528: Flags [.], seq 1:1399, ack 95, win 219, options [nop,nop,TS val 376939815 ecr 223714753], length 1398: HTTP: HTTP/1.1 200 OK
04:43:40.833500 IP 10.0.2.56.8080 > 10.0.2.59.42528: Flags [.], seq 1:1399, ack 95, win 219, options [nop,nop,TS val 376940024 ecr 223714753], length 1398: HTTP: HTTP/1.1 200 OK
04:43:41.241504 IP 10.0.2.56.8080 > 10.0.2.59.42528: Flags [.], seq 1:1399, ack 95, win 219, options [nop,nop,TS val 376940432 ecr 223714753], length 1398: HTTP: HTTP/1.1 200 OK
04:43:42.089503 IP 10.0.2.56.8080 > 10.0.2.59.42528: Flags [.], seq 1:1399, ack 95, win 219, options [nop,nop,TS val 376941280 ecr 223714753], length 1398: HTTP: HTTP/1.1 200 OK
04:43:43.753502 IP 10.0.2.56.8080 > 10.0.2.59.42528: Flags [.], seq 1:1399, ack 95, win 219, options [nop,nop,TS val 376942944 ecr 223714753], length 1398: HTTP: HTTP/1.1 200 OK
04:43:47.017484 IP 10.0.2.56.8080 > 10.0.2.59.42528: Flags [.], seq 1:1399, ack 95, win 219, options [nop,nop,TS val 376946208 ecr 223714753], length 1398: HTTP: HTTP/1.1 200 OK
04:43:53.801494 IP 10.0.2.56.8080 > 10.0.2.59.42528: Flags [.], seq 1:1399, ack 95, win 219, options [nop,nop,TS val 376952992 ecr 223714753], length 1398: HTTP: HTTP/1.1 200 OK
04:44:07.113748 IP 10.0.2.56.8080 > 10.0.2.59.42528: Flags [.], seq 1:1399, ack 95, win 219, options [nop,nop,TS val 376966304 ecr 223714753], length 1398: HTTP: HTTP/1.1 200 OK
04:44:33.225579 IP 10.0.2.56.8080 > 10.0.2.59.42528: Flags [.], seq 1:1399, ack 95, win 219, options [nop,nop,TS val 376992416 ecr 223714753], length 1398: HTTP: HTTP/1.1 200 OK
04:44:40.683645 IP 10.0.2.59.42528 > 10.0.2.56.8080: Flags [.], ack 1, win 229, options [nop,nop,TS val 223774811 ecr 376909813,nop,nop,sack 1 {4097:4639}], length 0
04:44:40.683683 IP 10.0.2.56.8080 > 10.0.2.59.42528: Flags [.], ack 95, win 219, options [nop,nop,TS val 376999874 ecr 223714753], length 0
04:45:42.123587 IP 10.0.2.59.42528 > 10.0.2.56.8080: Flags [.], ack 1, win 229, options [nop,nop,TS val 223836251 ecr 376909813,nop,nop,sack 1 {4097:4639}], length 0
04:45:42.123634 IP 10.0.2.56.8080 > 10.0.2.59.42528: Flags [R], seq 2707244320, win 0, length 0

client tcpdump

04:43:10.621871 IP 10.0.2.59.42528 > 10.0.2.56.8080: Flags [S], seq 888892677, win 28200, options [mss 1410,sackOK,TS val 223684749 ecr 0,nop,wscale 7], length 0
04:43:10.622547 IP 10.0.2.56.8080 > 10.0.2.59.42528: Flags [S.], seq 2707244319, ack 888892678, win 27960, options [mss 1410,sackOK,TS val 376909812 ecr 223684749,nop,wscale 7], length 0
04:43:10.622570 IP 10.0.2.59.42528 > 10.0.2.56.8080: Flags [.], ack 1, win 221, options [nop,nop,TS val 223684750 ecr 376909812], length 0
04:43:10.622689 IP 10.0.2.59.42528 > 10.0.2.56.8080: Flags [P.], seq 1:95, ack 1, win 221, options [nop,nop,TS val 223684750 ecr 376909812], length 94: HTTP: GET /ui/index.html HTTP/1.1
04:43:10.623144 IP 10.0.2.56.8080 > 10.0.2.59.42528: Flags [.], ack 95, win 219, options [nop,nop,TS val 376909813 ecr 223684750], length 0
04:43:10.623649 IP 10.0.2.56.8080 > 10.0.2.59.42528: Flags [P.], seq 4097:4638, ack 95, win 219, options [nop,nop,TS val 376909814 ecr 223684750], length 541: HTTP
04:43:10.623659 IP 10.0.2.59.42528 > 10.0.2.56.8080: Flags [.], ack 1, win 229, options [nop,nop,TS val 223684751 ecr 376909813,nop,nop,sack 1 {4097:4638}], length 0
04:43:40.625044 IP 10.0.2.56.8080 > 10.0.2.59.42528: Flags [F.], seq 4638, ack 95, win 219, options [nop,nop,TS val 376939815 ecr 223684751], length 0
04:43:40.625087 IP 10.0.2.59.42528 > 10.0.2.56.8080: Flags [.], ack 1, win 229, options [nop,nop,TS val 223714753 ecr 376909813,nop,nop,sack 1 {4097:4639}], length 0
04:44:40.683143 IP 10.0.2.59.42528 > 10.0.2.56.8080: Flags [.], ack 1, win 229, options [nop,nop,TS val 223774811 ecr 376909813,nop,nop,sack 1 {4097:4639}], length 0
04:44:40.684751 IP 10.0.2.56.8080 > 10.0.2.59.42528: Flags [.], ack 95, win 219, options [nop,nop,TS val 376999874 ecr 223714753], length 0
04:45:42.123123 IP 10.0.2.59.42528 > 10.0.2.56.8080: Flags [.], ack 1, win 229, options [nop,nop,TS val 223836251 ecr 376909813,nop,nop,sack 1 {4097:4639}], length 0
04:45:42.123882 IP 10.0.2.56.8080 > 10.0.2.59.42528: Flags [R], seq 2707244320, win 0, length 0

Trying another API endpoint that returns a smaller response (tested with this 175 and a ~250 byte) works:

bash-5.0# curl volume.lax-1:8080/status -vvvvvv
*   Trying 10.0.2.56:8080...
* Connected to volume.lax-1 (10.0.2.56) port 8080 (#0)
> GET /status HTTP/1.1
> Host: volume.lax-1:8080
> User-Agent: curl/7.73.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: application/json
< Server: SeaweedFS Volume 30GB 2.55
< Date: Mon, 28 Jun 2021 04:49:27 GMT
< Content-Length: 175
<
* Connection #0 to host volume.lax-1 left intact
{"DiskStatuses":[{"dir":"/data","all":104565039104,"used":6857490432,"free":97707548672,"percent_free":93.44189,"percent_used":6.55811}],"Version":"30GB 2.55 ","Volumes":null}

server tcpdump

04:49:27.030872 IP 10.0.2.59.42694 > 10.0.2.56.8080: Flags [S], seq 1918348233, win 28200, options [mss 1410,sackOK,TS val 224061158 ecr 0,nop,wscale 7], length 0
04:49:27.030926 IP 10.0.2.56.8080 > 10.0.2.59.42694: Flags [S.], seq 3061453414, ack 1918348234, win 27960, options [mss 1410,sackOK,TS val 377286221 ecr 224061158,nop,wscale 7], length 0
04:49:27.031275 IP 10.0.2.59.42694 > 10.0.2.56.8080: Flags [.], ack 1, win 221, options [nop,nop,TS val 224061159 ecr 377286221], length 0
04:49:27.031318 IP 10.0.2.59.42694 > 10.0.2.56.8080: Flags [P.], seq 1:88, ack 1, win 221, options [nop,nop,TS val 224061159 ecr 377286221], length 87: HTTP: GET /status HTTP/1.1
04:49:27.031324 IP 10.0.2.56.8080 > 10.0.2.59.42694: Flags [.], ack 88, win 219, options [nop,nop,TS val 377286221 ecr 224061159], length 0
04:49:27.031777 IP 10.0.2.56.8080 > 10.0.2.59.42694: Flags [P.], seq 1:321, ack 88, win 219, options [nop,nop,TS val 377286222 ecr 224061159], length 320: HTTP: HTTP/1.1 200 OK
04:49:27.032046 IP 10.0.2.59.42694 > 10.0.2.56.8080: Flags [.], ack 321, win 229, options [nop,nop,TS val 224061159 ecr 377286222], length 0
04:49:27.032295 IP 10.0.2.59.42694 > 10.0.2.56.8080: Flags [F.], seq 88, ack 321, win 229, options [nop,nop,TS val 224061160 ecr 377286222], length 0
04:49:27.032360 IP 10.0.2.56.8080 > 10.0.2.59.42694: Flags [F.], seq 321, ack 89, win 219, options [nop,nop,TS val 377286222 ecr 224061160], length 0
04:49:27.032554 IP 10.0.2.59.42694 > 10.0.2.56.8080: Flags [.], ack 322, win 229, options [nop,nop,TS val 224061160 ecr 377286222], length 0

client tcpdump

04:49:27.030601 IP 10.0.2.59.42694 > 10.0.2.56.8080: Flags [S], seq 1918348233, win 28200, options [mss 1410,sackOK,TS val 224061158 ecr 0,nop,wscale 7], length 0
04:49:27.031227 IP 10.0.2.56.8080 > 10.0.2.59.42694: Flags [S.], seq 3061453414, ack 1918348234, win 27960, options [mss 1410,sackOK,TS val 377286221 ecr 224061158,nop,wscale 7], length 0
04:49:27.031272 IP 10.0.2.59.42694 > 10.0.2.56.8080: Flags [.], ack 1, win 221, options [nop,nop,TS val 224061159 ecr 377286221], length 0
04:49:27.031327 IP 10.0.2.59.42694 > 10.0.2.56.8080: Flags [P.], seq 1:88, ack 1, win 221, options [nop,nop,TS val 224061159 ecr 377286221], length 87: HTTP: GET /status HTTP/1.1
04:49:27.031477 IP 10.0.2.56.8080 > 10.0.2.59.42694: Flags [.], ack 88, win 219, options [nop,nop,TS val 377286221 ecr 224061159], length 0
04:49:27.032034 IP 10.0.2.56.8080 > 10.0.2.59.42694: Flags [P.], seq 1:321, ack 88, win 219, options [nop,nop,TS val 377286222 ecr 224061159], length 320: HTTP: HTTP/1.1 200 OK
04:49:27.032044 IP 10.0.2.59.42694 > 10.0.2.56.8080: Flags [.], ack 321, win 229, options [nop,nop,TS val 224061159 ecr 377286222], length 0
04:49:27.032288 IP 10.0.2.59.42694 > 10.0.2.56.8080: Flags [F.], seq 88, ack 321, win 229, options [nop,nop,TS val 224061160 ecr 377286222], length 0
04:49:27.032528 IP 10.0.2.56.8080 > 10.0.2.59.42694: Flags [F.], seq 321, ack 89, win 219, options [nop,nop,TS val 377286222 ecr 224061160], length 0
04:49:27.032537 IP 10.0.2.59.42694 > 10.0.2.56.8080: Flags [.], ack 322, win 229, options [nop,nop,TS val 224061160 ecr 377286222], length 0

Answered by dswd

Jun 28, 2021

Ok this looks like the UDP packets are getting too big and getting somehow dropped because of that. Can you lower the MTU on the vpncloud0 interface (on both ends) to maybe 1400 or even 1300?

Also it would be interesting to know the MTU on the other interfaces and whether you are using TUN or TAP mode in VpnCloud.

View full answer

dswd · 2021-06-28T06:21:26Z

dswd
Jun 28, 2021
Maintainer

Ok this looks like the UDP packets are getting too big and getting somehow dropped because of that. Can you lower the MTU on the vpncloud0 interface (on both ends) to maybe 1400 or even 1300?

Also it would be interesting to know the MTU on the other interfaces and whether you are using TUN or TAP mode in VpnCloud.

4 replies

nivekuil Jun 28, 2021
Author

Ah, I see. So vpncloud0's mtu is 1427 because it's 1500-73 (the overhead for TUN) and it looks like the mtu for a swarm overlay network is 1450. So I guess the right number would be 1500-73-50, and that would have to be manually set to account for this particular use case. What's the best way to apply such an offset? Thanks for the help.

dswd Jun 28, 2021
Maintainer

Well actually VpnCloud has some code to figure out the optimal MTU on its own. I don't know why it didn't work for you, hence the question for the other interfaces MTUs.

You can change the MTU manually via ip link set mtu 1400 dev vpncloud0. If that works you can use the ifup setting to execute this command on startup.

nivekuil Jun 28, 2021
Author

I think vpncloud correctly guessed it to be 1427 because the physical interfaces are 1500, but it can't know that I actually want to pass it through an even smaller interface of 1450, which are created after vpncloud is set up (overlay interfaces are only inside docker swarm tasks) Not sure if it would make sense to have an offset that takes into account the existing vpncloud logic in such a way rather than hard coding the mtu in ifup.

nivekuil Jul 2, 2021
Author

For reference I set it up properly by making sure all overlay networks are set to the mtu of vpncloud0, in this case 1427. Within containers the mtu will automatically be decreased to account for the 50 byte vxlan overhead. No vpncloud tweaking necessary

mulder999 · 2022-08-20T09:51:29Z

mulder999
Aug 20, 2022

Did you solve it ?

I have a similar issue with docker swarm running over a vpncloud underlay (irrelevant if tap or tun).
When going through ingress load balancing, I receive back only first packet for any endpoint not hosting the service himself.

In the attached tcpdump file captured at the client side in tun mode, we can see:

Packet 22: client request
Packet 24: first part of the answer (that reaches in curl)
Packet 26: second part of the answer (that never reaches curl)

When opening the file with wireshark, it reports some sequence issue at packet 26 probably leading to duplicate ack of packet 27.
MTU seems not the culprit here.

docker-swarm-ilb-answer-issue.zip

0 replies

mulder999 · 2022-08-20T18:52:40Z

mulder999
Aug 20, 2022

Well I was wrong, indeed the mtu was the culprit.
For me the mtu for vpncloud was 1427, so I had to change all my overlay network including ingress to have that mtu.

For reference here are the commands:

docker network rm ingress
docker network create --driver overlay --ingress \
  --subnet=10.0.0.0/16 \
  --gateway=10.0.0.1 \
  --opt com.docker.network.driver.mtu=1427 \
  ingress

And in all my yaml's service descriptions:

...
networks:
  some-network:
    driver_opts:
      com.docker.network.driver.mtu: 1427

Should probably also do something similar about local networks but have not seen the issue with them yet.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Curl between vpncloud nodes getting RST? #206

{{title}}

Replies: 3 comments 4 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Curl between vpncloud nodes getting RST? #206

nivekuil Jun 28, 2021

Replies: 3 comments · 4 replies

dswd Jun 28, 2021 Maintainer

nivekuil Jun 28, 2021 Author

dswd Jun 28, 2021 Maintainer

nivekuil Jun 28, 2021 Author

nivekuil Jul 2, 2021 Author

mulder999 Aug 20, 2022

mulder999 Aug 20, 2022

nivekuil
Jun 28, 2021

Replies: 3 comments 4 replies

dswd
Jun 28, 2021
Maintainer

nivekuil Jun 28, 2021
Author

dswd Jun 28, 2021
Maintainer

nivekuil Jun 28, 2021
Author

nivekuil Jul 2, 2021
Author

mulder999
Aug 20, 2022

mulder999
Aug 20, 2022