Clone this wiki locally
Forcing clients to validate connections through "tunneled" connections and verifying PMTUD works
Note that this is a required feature for all mirrors.
A new feature (Feb 2015) of falling-sky is to do a more aggressive "large packet" test. A new daemon (coupled with linux ip6tables rules) can create an artificial "Packet Too Big" message, to aggressive test any client who has anything larger than the minimum MTU size of 1280.
**This works on Linux, tested on Ubuntu 12 through 16, Centos 5 through 7. Later versions are expected to work as well, subject to crazy changes like "let's move everything to systemd" (I'm very much afraid of whatever comes next!).
To do this test, we need additional work to a base install of falling-sky:
- Additional IPv6 address allocated (as "deprecated",
- Installed on your primary host
- DNS name: mtu1280.test-ipv6.example.com (AAAA records only)
- Firewalls, if any, permit port 80 to this new IP address
- Host based firewall: ip6tables installed with custom NFQUEUE rule
- New daemon listening (mtu1280d) to the NFQUEUE rule.
The benefit to all of this is not having to run an artificial IPv6 tunnel just to drop the MTU size; and not relying on the tunnel provider to send enough ICMPv6 "Packet Too Big" messages to clients.
- Allocate IPv6 address for your VM's subnet
/etc/rc.localto configure the IPv6 address as a deprecated address. This will allow for the new address to be used for incoming connections; but stop from using it for outgoing connections.
ip -6 addr add 2001:db8:1:18::1280/64 dev eth0 preferred_lft 0
- Make sure the name is similar to your existing falling-sky hostnames, only starting with "mtu1280". For example, "mtu1280.test-ipv6.example.com" for people running as a subdomain off of example.com.
- Run the command above manually; or reboot to truly validate your /etc/rc.local changes
- Make sure this new address is ping6-able from another host.
[Thanks to John Mann for https://github.com/falling-sky/mtu1280d/issues/1 ]
- First check to see if http://mtu1280.test-ipv6.example.com works. If this works, you can skip the Apache configuration. Otherwise..
- Some people have their apache configuration
Listento explicit IP addresses. If this is the case for you, be sure to add the additional Listen statement.
- You'll need to make sure that there is a new "Listen" statement that listens to your new IPv6 address.
- You'll need to find your existing VirtualHost configuration; and add "ServerAlias 2001:db8:1:18::1280" (substitute your new IPv6 address)
- Restart apache
- Make sure that curl http://mtu1280.test-ipv6.example.com/ip/ works . If not, debug this before proceeding.
rsync -av email@example.com:stable/mtu1280d . cd mtu1280d less README.md # Check for prerequisites make mtu1280d sudo make install upstart
Depending on your platform, you may need to use
sudo make install init.d instead.
At this point you'll need to start the daemon. One of these should work:
sudo start mtu1280d(upstart)
- `sudo /etc/init.d/mtu1280d start" (init.d)
Make sure the daemon is listening. Use "ps". init.d users may see an additional "-d" flag in the command line.
guest% ps auxww |grep mtu1280d root 28932 0.0 0.0 10560 136 ? Ss 13:28 0:00 /usr/sbin/mtu1280d -q 1280 jfesler 28940 0.0 0.0 11744 884 pts/1 S+ 13:28 0:00 grep mtu1280d
This is something that is very site specific. You'll need to identify how your OS will persistently keep (and manage) iptables and ip6tables rules.
- Debian/Ubuntu: consider the "iptables-persistent" package, which runs
ip6tables-restore /etc/iptables/rules.v6on boot.
- RedHat/Centos: consider enabling the "iptables" and "ip6tables" services; these will on boot run
Example iptables rules
You're going to need to add 1 rule to the PREROUTING chain of your "mangle" table.
This will route all packets to your mtu1280 host IP to the mtu1280d daemon. This daemon will either ACCEPT or DROP packets based on the size; as well as generate the ICMPv6 Packet Too Big responses. ACCEPT at this point does not actually affect later filter rules, since this is still in PREROUTING.
ip6tables -t mangle -A PREROUTING -d 2001:db8:1:18::1280/128 -j NFQUEUE --queue-num 1280
Adjust the IPv6 address to match what you assigned to your host.
This is a complete (but overly simple) example:
# Generated by ip6tables-save v1.4.21 on Wed Feb 18 10:14:54 2015 *mangle :PREROUTING ACCEPT [0:0] :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] :POSTROUTING ACCEPT [0:0] -A PREROUTING -d 2001:470:1:18::1280/128 -j NFQUEUE --queue-num 1280 COMMIT # Completed on Wed Feb 18 10:14:54 2015 # Generated by ip6tables-save v1.4.21 on Wed Feb 18 10:14:54 2015 *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] :CHECK_ABUSE - [0:0] COMMIT # Completed on Wed Feb 18 10:14:54 2015
Confirmation requires a separate IPv6 enabled host, that can run "curl" and "tcpdump". Linux can work; I've also used my Mac.
The fastest way to test this is with a Linux version of ping6 that supports the "-M" option.
You need to do this from a *different Linux host *than your mirror.
% ping6 -c 3 -M dont -s 1400 mtu1280.test-ipv6.example.com PING mtu1280.test-ipv6.example.com(mtu1280.test-ipv6.example.com) 1400 data bytes From mtu1280.master.test-ipv6.com icmp_seq=1 Packet too big: mtu=1280 From mtu1280.master.test-ipv6.com icmp_seq=2 Packet too big: mtu=1280 From mtu1280.master.test-ipv6.com icmp_seq=3 Packet too big: mtu=1280 --- mtu1280.test-ipv6.example.com ping statistics --- 3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2009ms
Tested on OS X 10.11 - El Capitan. Must use
-v to see the Packet Too Big messages. Must use -m to avoid fragmenting. Must use sudo.
Jasons-MacBook:home jfesler$ sudo ping6 -s 1400 -v -m mtu1280.test-ipv6.com PING6(1448=40+8+1400 bytes) 2601:647:4900:78ae:25c9:d601:c352:a669 --> 2001:470:1:18::1281 new path MTU (1280) is notified 1240 bytes from 2001:470:1:18::1281: Packet too big mtu = 1280 Vr TC Flow Plen Nxt Hlim 6 00 af090 0580 3a 36 2601:647:4900:78ae:25c9:d601:c352:a669->2001:470:1:18::1281 ICMP6: type = 128, code = 0
You can use the online Validation tool to confirm all parts of your mirror operate the way they are expected to. This includes testing for a full MTU of 1500 on your primary address; and a lower MTU of 1280 on the PMTUD test address.
This is the most thorough way to do this. This will allow you to actually observe what happens when large packets come in; and how TCP responds.
Run this on your client host. This must be a different host than your mirror runs on. Adjust the IPv6 address to match your new mtu1280 IPv6 address.
sudo tcpdump -i eth0 -vvv 'host 2001:db8:1:18::1280 or (ip6 and icmp6 and ip6 = 2)'
On the same client host, run this curl command. Adjust the hostname to match your site.
You should see, the first time you run it,
laptop:~ jfesler$ sudo tcpdump 'host 2001:db8:1:18::1280 or (ip6 and icmp6 and ip6 = 2)' Password: tcpdump: data link type PKTAP tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on pktap, link-type PKTAP (Packet Tap), capture size 65535 bytes 09:03:57.697254 IP6 2001:db8:effd::1234.53649 > 2001:db8:1:18::1280.http: Flags [S], seq 2919600708, win 65535, options [mss 1440,nop,wscale 5,nop,nop,TS val 326886165 ecr 0,sackOK,eol], length 0 09:03:57.702436 IP6 2001:db8:1:18::1280.http > 2001:db8:effd::1234.53649: Flags [S.], seq 1207337852, ack 2919600709, win 28160, options [mss 1366,sackOK,TS val 325898144 ecr 326886165,nop,wscale 7], length 0 09:03:57.702469 IP6 2001:db8:effd::1234.53649 > 2001:db8:1:18::1280.http: Flags [.], ack 1, win 4104, options [nop,nop,TS val 326886170 ecr 325898144], length 0 09:03:57.702534 IP6 2001:db8:effd::1234.53649 > 2001:db8:1:18::1280.http: Flags [.], seq 1:1355, ack 1, win 4104, options [nop,nop,TS val 326886170 ecr 325898144], length 1354 09:03:57.702540 IP6 2001:db8:effd::1234.53649 > 2001:db8:1:18::1280.http: Flags [P.], seq 1355:1805, ack 1, win 4104, options [nop,nop,TS val 326886170 ecr 325898144], length 450 09:03:57.710307 IP6 2001:db8:1:18::1280 > 2001:db8:effd::1234: ICMP6, packet too big, mtu 1280, length 1240 09:03:57.710433 IP6 2001:db8:effd::1234.53649 > 2001:db8:1:18::1280.http: Flags [.], seq 1:1209, ack 1, win 4104, options [nop,nop,TS val 326886170 ecr 325898144], length 1208 09:03:57.710457 IP6 2001:db8:effd::1234.53649 > 2001:db8:1:18::1280.http: Flags [P.], seq 1209:1805, ack 1, win 4104, options [nop,nop,TS val 326886170 ecr 325898144], length 596 09:03:57.714993 IP6 2001:db8:1:18::1280.http > 2001:db8:effd::1234.53649: Flags [.], ack 1209, win 239, options [nop,nop,TS val 325898147 ecr 326886170], length 0 09:03:57.715904 IP6 2001:db8:1:18::1280.http > 2001:db8:effd::1234.53649: Flags [.], ack 1805, win 258, options [nop,nop,TS val 325898147 ecr 326886170], length 0 09:03:57.719508 IP6 2001:db8:1:18::1280.http > 2001:db8:effd::1234.53649: Flags [.], seq 1:1355, ack 1805, win 258, options [nop,nop,TS val 325898148 ecr 326886170], length 1354 09:03:57.720537 IP6 2001:db8:1:18::1280.http > 2001:db8:effd::1234.53649: Flags [P.], seq 1355:1871, ack 1805, win 258, options [nop,nop,TS val 325898148 ecr 326886170], length 516 09:03:57.720605 IP6 2001:db8:effd::1234.53649 > 2001:db8:1:18::1280.http: Flags [.], ack 1871, win 4079, options [nop,nop,TS val 326886187 ecr 325898148], length 0 09:03:57.721019 IP6 2001:db8:effd::1234.53649 > 2001:db8:1:18::1280.http: Flags [F.], seq 1805, ack 1871, win 4096, options [nop,nop,TS val 326886187 ecr 325898148], length 0 09:03:57.725160 IP6 2001:db8:1:18::1280.http > 2001:db8:effd::1234.53649: Flags [F.], seq 1871, ack 1806, win 258, options [nop,nop,TS val 325898150 ecr 326886187], length 0 09:03:57.725218 IP6 2001:db8:effd::1234.53649 > 2001:db8:1:18::1280.http: Flags [.], ack 1872, win 4096, options [nop,nop,TS val 326886191 ecr 325898150], length 0 ^C 16 packets captured 423 packets received by filter 0 packets dropped by kernel laptop:~ jfesler$
The "packet too big" will be in response to the first TCP packet to that host where the payload length is more than 1280.
The second time you run the curl command, you'll see no packet too big - but you'll also see no packets larger than 1280 bytes. This is due to your client host remembering the MTU. The length of time this is remembered varies from OS to OS.
Final Browser Test
- Point your browser at your mirror site.
- Shift-Reload the browser (to clear out any cached /site/config.js)
- Go to "Tests Run"
- Go to "Technical Info"
- Verify that "Test IPv6 large packet" shows "mtu1280.test-ipv6.domain.com" and success