Amazon ELBs - ICMP and Network Maximum Transmission Unit (MTU)

[ajohnstone]

The firewall rules created on Amazon ELBs do not include ICMP. This means that issues can occur with PMTU. Please see below for more information.

When a HTTP/TCP request exceeds a certain payload size, with ELBs not allowing "ICMP request/reply", the request stalls. This is due to the ELB trying to send big packets to the client but the client dropping them since the MTU was higher than expected.

To avoid this ICMP traffic should be allowed on the ELB so they can negotiate the MTU:

• https://en.wikipedia.org/wiki/Path_MTU_Discovery
• http://www.networkworld.com/article/2224654/cisco-subnet/mtu-size-issues.html
• http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_mtu.html

When enabling ICMP traffic the client will be able to communicate with the ELB normally.

An example of what occurs when ICMP is not enabled.

$ time curl -v -O > /dev/null http://a8222e4ee025111e6a25d0a91ac3ee4d-1379358768.eu-west-1.elb.amazonaws.com/docs-api/swaggerui/lib/jquery.slideto.min.js
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 52.49.165.25...
* Connected to a8222e4ee025111e6a25d0a91ac3ee4d-1379358768.eu-west-1.elb.amazonaws.com (52.49.165.25) port 80 (#0)
> GET /docs-api/swaggerui/lib/jquery.slideto.min.js HTTP/1.1
> Host: a8222e4ee025111e6a25d0a91ac3ee4d-1379358768.eu-west-1.elb.amazonaws.com
> User-Agent: curl/7.43.0
> Accept: */*
> 
  0     0    0     0    0     0      0      0 --:--:--  0:00:29 --:--:--     0< HTTP/1.1 200 OK
< content-length: 369
< content-type: application/javascript; charset=utf-8
< last-modified: Thu, 12 Feb 2015 14:06:22 GMT
< etag: "b849bafad46251cbced13d4b501694dd21bd7464"
< cache-control: no-cache
< accept-ranges: bytes
< Date: Thu, 14 Apr 2016 15:31:06 GMT
< Connection: keep-alive
< 
{ [369 bytes data]
100   369  100   369    0     0     12      0  0:00:30  0:00:29  0:00:01    85
* Connection #0 to host a8222e4ee025111e6a25d0a91ac3ee4d-1379358768.eu-west-1.elb.amazonaws.com left intact

real    0m29.547s
user    0m0.012s
sys 0m0.004s

Ideally security rules for ELBs should be allowed to be attached to an ELB via annotations and/or by default.

ICMP rules at a minimum to avoid this should cover:

• 0 - Echo Reply
• 8 - Echo Request

* Please note that this does not just affect Amazon ELBs, but in addition direct access to instances too.*

[lavalamp]

Not sure where this should go - one of @justinsb or @thockin should know.

[rata]

@ajohnstone Are you sure only echo request/reply is needed? Not Destination Unreachable?

[ajohnstone]

@rata
I recall adding ICMP echo request/reply and it working correctly after that and not adding anything else, when originally testing this. However looking at the wikipedia article https://en.wikipedia.org/wiki/Path_MTU_Discovery. It states ...

"However, in order for the Transmission Control Protocol (TCP) to operate most efficiently, ICMP Unreachable messages (type 3) should be permitted."

The full set of ICMP rules I typically allow are:

ICMP Code	Name	Reference
0	Echo Reply	RFC792
3	Destination Unreachable	RFC792
5	Redirect	RFC792
11	Time Exceeded	RFC792
12	Parameter Problem	RFC792

There maybe merit in considering other ICMP codes.

[rata]

Yes, and the AWS documentation says also Type 3. So, I really think icmp type 3 is needed for PMTU discovery.

[thockin]

@justinsb

This sounds plausible to me.

On Mon, May 9, 2016 at 3:52 PM, rata notifications@github.com wrote:

Yes, and the AWS documentation says also Type 3. So, I really think icmp
type 3 is needed for PMTU discovery.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#24254 (comment)

[goltermann]

@justinsb This one lacks a priority. Think it's P1 (needs to be fixed for 1.3) or P2/P3 (can be fixed in 1.4)?

[bprashanth]

Sounds like this should be easy enough to fix that we can try and get in 1.3, if someone more familiar with the AWS cloudprovider setup is willing to send a pr my way (or to @justinsb) that flips ICMP in the firewall rule