Path MTU Discovery is broken on pypi.python.org

[WGH-]

Consider your host is connected through a network with reduced MTU, e.g. HOST <--1500--> ROUTER_A <--1000--> ROUTER_B <--1500--> INTERNET.
Your host will connect advertising MSS as if MTU was 1500. However, when the remote hosts send large TCP segments, the router on the "MTU boundary" will reply with ICMP Destination Unreachable message with proper MTU attached, and the remote host will resend the TCP segments of proper size.

That's how Path MTU Discovery is supposed to work.

However, PMTUD seems completely broken on PyPI hosts, as they ignore the ICMP message. This results in connection hanging forever.

18:20:54.280730 IP 10.15.10.188.37612 > 151.101.84.223.443: Flags [S], seq 1008604264, win 29200, options [mss 1460,sackOK,TS val 3368973112 ecr 0,nop,wscale 7], length 0
18:20:54.302786 IP 151.101.84.223.443 > 10.15.10.188.37612: Flags [S.], seq 2542879512, ack 1008604265, win 26960, options [mss 1360,sackOK,TS val 355914490 ecr 3368973112,nop,wscale 9], length 0
18:20:54.303756 IP 10.15.10.188.37612 > 151.101.84.223.443: Flags [.], ack 1, win 229, options [nop,nop,TS val 3368973135 ecr 355914490], length 0
18:20:54.316799 IP 10.15.10.188.37612 > 151.101.84.223.443: Flags [P.], seq 1:518, ack 1, win 229, options [nop,nop,TS val 3368973148 ecr 355914490], length 517
18:20:54.338907 IP 151.101.84.223.443 > 10.15.10.188.37612: Flags [.], ack 518, win 55, options [nop,nop,TS val 355914499 ecr 3368973148], length 0
18:20:54.340397 IP 151.101.84.223.443 > 10.15.10.188.37612: Flags [P.], seq 1:4169, ack 518, win 55, options [nop,nop,TS val 355914500 ecr 3368973148], length 4168
18:20:54.340442 IP 10.15.10.188 > 151.101.84.223: ICMP 10.15.10.188 unreachable - need to frag (mtu 1354), length 556
18:20:54.428097 IP 151.101.84.223.443 > 10.15.10.188.37612: Flags [P.], seq 4045:4169, ack 518, win 55, options [nop,nop,TS val 355914522 ecr 3368973148], length 124
18:20:54.428974 IP 10.15.10.188.37612 > 151.101.84.223.443: Flags [.], ack 1, win 237, options [nop,nop,TS val 3368973261 ecr 355914499,nop,nop,sack 1 {4045:4169}], length 0
18:20:54.451106 IP 151.101.84.223.443 > 10.15.10.188.37612: Flags [.], seq 1:1349, ack 518, win 55, options [nop,nop,TS val 355914527 ecr 3368973261], length 1348
18:20:54.451145 IP 10.15.10.188 > 151.101.84.223: ICMP 10.15.10.188 unreachable - need to frag (mtu 1354), length 556
18:20:54.692492 IP 151.101.84.223.443 > 10.15.10.188.37612: Flags [.], seq 1:1349, ack 518, win 55, options [nop,nop,TS val 355914588 ecr 3368973261], length 1348
18:20:54.692529 IP 10.15.10.188 > 151.101.84.223: ICMP 10.15.10.188 unreachable - need to frag (mtu 1354), length 556
18:20:55.192128 IP 151.101.84.223.443 > 10.15.10.188.37612: Flags [.], seq 1:1349, ack 518, win 55, options [nop,nop,TS val 355914712 ecr 3368973261], length 1348
18:20:55.192191 IP 10.15.10.188 > 151.101.84.223: ICMP 10.15.10.188 unreachable - need to frag (mtu 1354), length 556
18:20:56.180178 IP 151.101.84.223.443 > 10.15.10.188.37612: Flags [.], seq 1:1349, ack 518, win 55, options [nop,nop,TS val 355914960 ecr 3368973261], length 1348
18:20:56.180236 IP 10.15.10.188 > 151.101.84.223: ICMP 10.15.10.188 unreachable - need to frag (mtu 1354), length 556

For example, this is how properly behaving server works:

18:12:59.499749 IP 10.15.10.188.44636 > 94.130.181.17.443: Flags [S], seq 3110462941, win 29200, options [mss 1460,sackOK,TS val 2863713539 ecr 0,nop,wscale 7], length 0
18:12:59.551404 IP 94.130.181.17.443 > 10.15.10.188.44636: Flags [S.], seq 2941235491, ack 3110462942, win 65160, options [mss 1460,sackOK,TS val 3537241376 ecr 2863713539,nop,wscale 7], length 0
18:12:59.552376 IP 10.15.10.188.44636 > 94.130.181.17.443: Flags [.], ack 1, win 229, options [nop,nop,TS val 2863713591 ecr 3537241376], length 0
18:12:59.567515 IP 10.15.10.188.44636 > 94.130.181.17.443: Flags [P.], seq 1:518, ack 1, win 229, options [nop,nop,TS val 2863713606 ecr 3537241376], length 517
18:12:59.619156 IP 94.130.181.17.443 > 10.15.10.188.44636: Flags [.], ack 518, win 506, options [nop,nop,TS val 3537241443 ecr 2863713606], length 0
18:12:59.622028 IP 94.130.181.17.443 > 10.15.10.188.44636: Flags [P.], seq 1:3049, ack 518, win 506, options [nop,nop,TS val 3537241446 ecr 2863713606], length 3048
18:12:59.622068 IP 10.15.10.188 > 94.130.181.17: ICMP 10.15.10.188 unreachable - need to frag (mtu 1354), length 556
18:12:59.672987 IP 94.130.181.17.443 > 10.15.10.188.44636: Flags [.], seq 1:1303, ack 518, win 506, options [nop,nop,TS val 3537241446 ecr 2863713606], length 1302
18:12:59.673157 IP 94.130.181.17.443 > 10.15.10.188.44636: Flags [P.], seq 1303:3049, ack 518, win 506, options [nop,nop,TS val 3537241446 ecr 2863713606], length 1746

A simple way to reproduce this problem is to start a Docker container, manually reduce MTU on docker0 bridge, and try to connect from inside the container. This is somewhat inaccurate way to reproduce the problem, as virtual interface ends up being plugged into bridge with smaller MTU, and as such outgoing packets may get dropped, but it illustrates the problem fine, as it's easy to find hosts that respond to ICMP Unreachable properly.

# docker run --rm -it ubuntu:18.04
root@cdb4f26389dc:/# apt update && apt install -y curl

# ip link set dev docker0 mtu 1000
# tcpdump -i enp0s1 -n 'tcp port 443 or icmp'

root@cdb4f26389dc:/# curl https://pypi.python.org

[WGH-]

It seems that broken PMTUD is relatively common on the global internet, and the workaround for the "criminally braindead ISPs or servers" (quoting man iptables-extensions) is to fix value MSS on router. The hack is known under names mssfix, -j TCPMSS --clamp-mss-to-pmtu.

[ewdurbin]

Thanks for your detailed report @WGH-, looks like this is something that would need to be addressed with our CDN.

Is this impacting your ability to use PyPI?

[WGH-]

Initially, it did. But after some research I figured out the workaround, and that this brokeness is unfortunately very common on the public Internet.

[ewdurbin]

I'm not sure this is actionable for us, the network issue is between the end user and the CDN edge servers. If we see further reports of MTU issues we can reopen and investigate with our CDN provider.