Path MTU discovery

[dcarley]

Todos

Please try the following before submitting the issue:

• Upgrade Concourse if you are observing a failure in a test
  • N/A
• Use latest BOSH stemcell if the problem occurred in a BOSH VM
• Use Go 1.6 if the issue has to do with running unit tests locally
  • N/A

Description

We run Cloud Foundry and use SAP/ipsec-release to secure unencrypted traffic between routers and cells. After upgrading to garden-runc-release 1.4.0 we experienced an outage whereby requests to applications that generated HTTP responses larger than approximately 9k would hang after returning the response headers.

We eventually narrowed it down to the change in container MTUs from garden-runc-release 1.3.0 (af8d612 and cloudfoundry/garden-runc-release@a9b22fd). We deploy to AWS where some instances types get a host interface MTU of 9001, so with that release our containers MTUs changed from 1500 to 9001. This meant that they generated response packets that were too large after IPsec transport headers had been added.

Now, we understand that we're deviating a bit from the norm by running IPsec, and that it's not Guardian's responsibility to know about it. We can also workaround it by setting garden.network_mtu = 1500 but it feels fragile because we have to hardcode the value irrespective of what the host's actual MTU is.

However it feels like this should Just Work with path MTU discovery. I'm not clear why it doesn't - perhaps you can shed any light? There was a similar discussion in #51 previously.

Logging and/or test output

tcpdump on a cell shows ICMP unreachable packets being generated from the cell to itself. I'm unsure if anything is acting upon these though:

  14:11:22.727890 IP (tos 0xc0, ttl 64, id 19465, offset 0, flags [none], proto ICMP (1), length 576)
      cell-0.node.dc1.cf.internal > cell-0.node.dc1.cf.internal: ICMP router-0.node.dc1.cf.internal unreachable - need to frag (mtu 9001), length 556
          IP (tos 0x0, ttl 63, id 1347, offset 0, flags [DF], proto ESP (50), length 9048)
      cell-0.node.dc1.cf.internal > router-0.node.dc1.cf.internal: ESP(spi=REDACTED,seq=0x170), length 9028
  14:11:24.225646 IP (tos 0xc0, ttl 64, id 19581, offset 0, flags [none], proto ICMP (1), length 576)
      cell-0.node.dc1.cf.internal > cell-0.node.dc1.cf.internal: ICMP router-0.node.dc1.cf.internal unreachable - need to frag (mtu 9001), length 556
          IP (tos 0x0, ttl 63, id 22649, offset 0, flags [DF], proto ESP (50), length 9048)
      cell-0.node.dc1.cf.internal > router-0.node.dc1.cf.internal: ESP(spi=REDACTED,seq=0x174), length 9028
  14:11:24.227976 IP (tos 0xc0, ttl 64, id 19582, offset 0, flags [none], proto ICMP (1), length 576)
      cell-0.node.dc1.cf.internal > cell-0.node.dc1.cf.internal: ICMP router-0.node.dc1.cf.internal unreachable - need to frag (mtu 9001), length 556
          IP (tos 0x0, ttl 63, id 22652, offset 0, flags [DF], proto ESP (50), length 9048)
      cell-0.node.dc1.cf.internal > router-0.node.dc1.cf.internal: ESP(spi=REDACTED,seq=0x177), length 9028

Steps to reproduce

1. Deploy CF with:
   • the versions referenced below
   • SAP/ipsec-release
   • host MTUs set to 9001
2. Deploy the dora application
3. Request a response larger than the MTU size: curl <dora_url>/largetext/20

Please also provide the following information if applicable:

• Guardian release version: 1.4.0 (with diego-release 1.12.0 and cf-release 256)
• Linux kernel version: 4.4.0-72-generic (BOSH stemcell 3363.19 bosh-aws-xen-hvm-ubuntu-trusty-go_agent)
• Concourse version: N/A
• Go version: N/A, whatever is used by garden-runc-release

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/144453285

The labels on this github issue will be updated when the story is started.

[julz]

Hi @dcarley - apologies for this, we were convinced defaulting the MTU was "the right thing to do" because if it's higher than the external interface MTU that could never be correct (and this caused issues on some clouds where the default MTU is smaller than 1500 so our out-of-the-box config didn't work). I think what we missed is that defaulting in the way we did could also raise the container MTU, which might conflict with overlay or IPSec workflows.

It sounds like you have a decent workaround for this for now.. going forward if we were to change the defaulting behaviour so it picked the lowest of the external interface MTU or 1500 do you think that would work for you?

[sklevenz]

I am having the same problem and we are currently testing the workaround. Providing feedback as soon as I have results.

@julz What you are proposing would work for us. +1

[julz]

thanks all, I've pulled in a story to default to min(1500, externalMtu) rather than externalMtu. You can follow it here: https://www.pivotaltracker.com/story/show/145545479