Docker daemon unresponsive and gigabytes of memory when full BGP table added to system (650k routes)

[sysopcorner]

Description of problem:
We have a server which is connected to full BGP feed. It means that system has 650k routes in route table which in fact is quite normal. After starting docker daemon it allocates gigabytes of memory and never stops. All docker commands (eg. "docker ps") hangs. After removing routes everything is going back to normal.
Reproduced on docker vesions 1.6.2 and 1.7.1

docker version:
Client version: 1.7.1
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 786b29d
OS/Arch (client): linux/amd64
Server version: 1.7.1
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 786b29d
OS/Arch (server): linux/amd64

docker info:
Containers: 0
Images: 9
Storage Driver: devicemapper
 Pool Name: docker-9:1-1184122-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 493.9 MB
 Data Space Total: 107.4 GB
 Data Space Available: 12.15 GB
 Metadata Space Used: 1.102 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.146 GB
 Udev Sync Supported: false
 Deferred Removal Enabled: false
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.82-git (2013-10-04)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.18.11
Operating System: Debian GNU/Linux 7 (wheezy)
CPUs: 32
Total Memory: 125.9 GiB
Name: host
ID: DYIE:LMBJ:V5FV:FYTY:3WTQ:5ZLW:CPZI:RVHK:VFOO:XKXR:H2EP:PVUS
WARNING: No swap limit support

uname -a:
Linux host 3.18.11 #1 SMP Thu Jun 11 12:40:05 CEST 2015 x86_64 GNU/Linux

Environment details (AWS, VirtualBox, physical, etc.):
Physical server with full BGP route table and 128GB of ram. Can be reproduced on virtual servers too.

How reproducible:
Insert thousands of routes into routing table and restart docker daemon.

Steps to Reproduce:

1. add dummy interface:
   ip link add name dummy0 type dummy && ip link set up dummy0
2. add multiple routes into the system (it takes few minutes):

for a in {2..10}; do
  for b in {1..253}; do
   for c in {1..253};
    do ip route add 1.$a.$b.$c/32 dev dummy0;
   done ;
  done;
done

1. start docker daemon

docker -D -d
DEBU[0000] Registering OPTIONS,                         
DEBU[0000] Registering GET, /events                     
DEBU[0000] Registering GET, /images/json                
DEBU[0000] Registering GET, /containers/json            
DEBU[0000] Registering GET, /containers/{name:.*}/export 
DEBU[0000] Registering GET, /containers/{name:.*}/logs  
DEBU[0000] Registering GET, /containers/{name:.*}/stats 
DEBU[0000] Registering GET, /containers/{name:.*}/attach/ws 
DEBU[0000] Registering GET, /info                       
DEBU[0000] Registering GET, /version                    
DEBU[0000] Registering GET, /images/search              
DEBU[0000] Registering GET, /images/get                 
DEBU[0000] Registering GET, /images/{name:.*}/json      
DEBU[0000] Registering GET, /containers/{name:.*}/top   
DEBU[0000] Registering GET, /containers/ps              
DEBU[0000] Registering GET, /containers/{name:.*}/changes 
DEBU[0000] Registering GET, /exec/{id:.*}/json          
DEBU[0000] Registering GET, /_ping                      
DEBU[0000] Registering GET, /images/{name:.*}/get       
DEBU[0000] Registering GET, /images/{name:.*}/history   
DEBU[0000] Registering GET, /containers/{name:.*}/json  
DEBU[0000] Registering POST, /containers/{name:.*}/start 
DEBU[0000] Registering POST, /containers/{name:.*}/stop 
DEBU[0000] Registering POST, /exec/{name:.*}/start      
DEBU[0000] Registering POST, /images/create             
DEBU[0000] Registering POST, /images/load               
DEBU[0000] Registering POST, /containers/{name:.*}/kill 
DEBU[0000] Registering POST, /containers/{name:.*}/unpause 
DEBU[0000] Registering POST, /exec/{name:.*}/resize     
DEBU[0000] Registering POST, /containers/{name:.*}/rename 
DEBU[0000] Registering POST, /auth                      
DEBU[0000] Registering POST, /containers/create         
DEBU[0000] Registering POST, /containers/{name:.*}/wait 
DEBU[0000] Registering POST, /containers/{name:.*}/resize 
DEBU[0000] Registering POST, /containers/{name:.*}/exec 
DEBU[0000] Registering POST, /build                     
DEBU[0000] Registering POST, /images/{name:.*}/tag      
DEBU[0000] Registering POST, /containers/{name:.*}/restart 
DEBU[0000] Registering POST, /containers/{name:.*}/attach 
DEBU[0000] Registering POST, /commit                    
DEBU[0000] Registering POST, /images/{name:.*}/push     
DEBU[0000] Registering POST, /containers/{name:.*}/pause 
DEBU[0000] Registering POST, /containers/{name:.*}/copy 
DEBU[0000] Registering DELETE, /containers/{name:.*}    
DEBU[0000] Registering DELETE, /images/{name:.*}        
DEBU[0000] docker group found. gid: 999                 
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock) 

Actual Results:

ip route show | wc -l
576088

ps aux --sort=-vsz,-rss | grep docker
root     17145  100  8.8 11867512 11623548 pts/2 Sl+ 14:17   1:21 docker -D -d

Expected Results:
Docker daemon can start and be responsive with full BGP route table.

[unclejack]

I'll look into this because I've investigate some related code with similar problems in the past.

[sanmai-NL]

ping