Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EC2 Instance networking goes down when running Docker and receiving fragmented IPv4 packets #672

Open
2 of 3 tasks
camdenfullmer opened this issue May 17, 2019 · 0 comments

Comments

@camdenfullmer
Copy link

camdenfullmer commented May 17, 2019

  • This is a bug report
  • This is a feature request
  • I searched existing issues before opening this one

Expected behavior

EC2 instance to continue running and stay connected to the network.

Actual behavior

EC2 instance CPU spikes and drops all networking causing it to no longer be reachable and fail instance status checks.

Steps to reproduce the behavior

  1. Run the attached iOS client app that sends UDP packets to the instance over IPv4. Note: It reliably happens when using T-Mobile cellular.
  2. Run the attached server app that receives the UDP packets. Note: This lock up happens when running inside or outside of Docker.
  3. Wait for the instance CPU to rise and networking to go out.

Output of docker version:

Client:
 Version:           18.09.5
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        e8ff056
 Built:             Thu Apr 11 04:43:57 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.5
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       e8ff056
  Built:            Thu Apr 11 04:10:53 2019
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 18.09.5
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-1032-aws
Operating System: Ubuntu 18.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 120GiB
Name: ip-172-31-42-22
ID: 7ZI2:YFJC:7H2E:6GB6:TBFY:CY3T:KIBJ:HRFY:S2MN:UENM:QV4N:6JT4
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.)

Base AWS AMI: ami-0a313d6098716f372
Instance Types: g3.4xlarge or c5.2xlarge
iOS app:
LockUpDemo.zip
Server app:
main.c.zip
tcpdump capture:
capture.pcap.zip
Wireshark Screenshot:
Screen Shot 2019-05-17 at 11 37 01 AM

The networking drop out does not happen unless Docker is installed on the machine. I think it has something to do with invalid fragmentation on the handoff from T-Mobile to AWS. Not sure why it would be spiking the CPU on the instance though (iptables can't handle the fragmentation?).

Also, ran a test using Cellular to my local network and I did not receive any of the fragmentation that is shown in the capture.

@camdenfullmer camdenfullmer changed the title EC2 Instance networking goes down when running Docker and receiving fragmented IP packets EC2 Instance networking goes down when running Docker and receiving fragmented IPv4 packets May 17, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant