-
Notifications
You must be signed in to change notification settings - Fork 18.9k
Description
I have an Ubuntu server, running Docker without any issues. It has several projects managed with docker-compose. Suddenly a docker-compose up of one of those projects caused the server to become unresponsive on the network.
This server uses the ip range 192.168.64.0/24 and has the IP 192.168.64.2 with 192.168.64.1 being the router that servers as gateway for this server.
This router routes traffic to and from the office network at 192.168.16.0/20.
After examining the server by logging in physically, it becomes clear that Docker or Docker Compose had suddenly created a bridge interface with ip 192.168.16.1 in the 192.168.16.0/20 range which made it impossible for it to connect to our office network.
Steps to reproduce the issue:
Unfortunately I'm unable to reproduce this issue reliably.
Describe the results you received:
A bridge interface with ip 192.168.16.1 was created in the 192.168.16.0/20 range.
Describe the results you expected:
A bridge interface to be created in a 172.*.0.0/16 range.
Additional information you deem important (e.g. issue happens only occasionally):
Output of docker version:
Client:
Version: 18.06.1-ce
API version: 1.38
Go version: go1.10.3
Git commit: e68fc7a
Built: Tue Aug 21 17:24:51 2018
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 18.06.1-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: e68fc7a
Built: Tue Aug 21 17:23:15 2018
OS/Arch: linux/amd64
Experimental: false
Output of docker info:
Containers: 21
Running: 0
Paused: 0
Stopped: 21
Images: 67
Server Version: 18.06.1-ce
Storage Driver: btrfs
Build Version: Btrfs v4.15.1
Library Version: 102
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-33-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 20
Total Memory: 94.38GiB
Name: bumblebee
ID: 5Q5F:LBIJ:OM5P:7EAI:PJU3:7GRH:ESXL:ZLWQ:V2SK:P6L7:ZEHA:CHDU
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
Additional environment details (AWS, VirtualBox, physical, etc.):
physical
Output of docker network inspect:
[
{
"Name": "john_assets3",
"Id": "a319f31dc146dd7a2541cb935b3a66998d9d0ff2476565fda2cdecc213363456",
"Created": "2018-09-11T08:38:38.403766297Z",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "192.168.16.0/20",
"Gateway": "192.168.16.1"
}
]
},
"Internal": false,
"Attachable": true,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {},
"Options": {},
"Labels": {
"com.docker.compose.network": "assets3",
"com.docker.compose.project": "john",
"com.docker.compose.version": "1.21.2"
}
}
]
Anonymised contents of docker-compose.yml:
version: "3.2"
services:
api:
image: registry.example.net/project/api:latest
tty: true
depends_on:
- "postgres"
- "redis"
- "elasticsearch-145"
- "mailcatcher"
- "api-uploads"
- "memcached"
- "elasticsearch"
networks:
default:
aliases:
- "api-alias"
- "api-alias1"
- "proxy1"
- "proxy2"
- "proxy3"
- "proxy4"
- "proxy5"
- "proxy6"
- "proxy7"
- "proxy8"
- "proxy9"
volumes:
- "labels:/var/www/html/labels"
- "logos:/var/www/html/logos"
- "invoices:/var/www/html/invoices"
- "sepa:/var/www/html/sepa"
- "storage:/var/www/html/storage"
- "uploads:/var/www/html/uploads"
- "direct_debit_forms:/var/www/html/direct_debit_forms"
- "bank_transactions:/var/www/html/bank_transactions"
- "rsa-keys:/var/www/html/config/rsa-keys"
environment:
- ENVIRONMENT=prod
- PLASTIC_HOST=elasticsearch
- MEMCACHED_HOST=memcached
- NEW_RELIC_LICENSE=foo
- BLACKFIRE_SERVER_ID=bar
- BLACKFIRE_SERVER_TOKEN=baz
api-proxy:
image: registry.example.net/project/api-proxy:latest
tty: true
depends_on:
- "api"
environment:
- ENVIRONMENT=prod
api-uploads:
image: nginx:alpine
volumes:
- "uploads:/usr/share/nginx/html"
ui:
image: registry.example.net/project/ui:latest
depends_on:
- "api"
- "frontoffice"
- "assets1"
networks:
default:
assets3:
environment:
- FOO=false
admin:
image: registry.example.net/project/admin:latest
depends_on:
- "api"
environment:
- FOO=false
assets2:
image: registry.example.net/project/assets2:latest
networks:
default:
assets2:
aliases:
- "assets"
assets3:
image: registry.example.net/project/assets3:latest
networks:
default:
assets3:
aliases:
- "assets"
assets1:
image: registry.example.net/project/assets1:latest
networks:
default:
assets1:
aliases:
- "assets"
meteor2:
image: registry.example.net/project/meteor2:latest
tty: true
depends_on:
- "mongodb"
- "api"
- "assets2"
networks:
default:
assets2:
environment:
- ENVIRONMENT=prod
- NPM_CONFIG_LOGLEVEL=verbose
- MONGO_URL=mongodb://mongodb/meteor2
- MAIL_URL=smtp://mailcatcher:1025/
- ROOT_URL=https://meteor2.example.com
- PORT=3000
- NODE_ENV=${ENVIRONMENT}
meteor3:
image: registry.example.net/project/meteor3:latest
tty: true
depends_on:
- "mongodb"
- "api"
- "assets3"
networks:
default:
assets3:
environment:
- ENVIRONMENT=prod
- NPM_CONFIG_LOGLEVEL=verbose
- MONGO_URL=mongodb://mongodb/meteor3
- MAIL_URL=smtp://mailcatcher:1025/
- ROOT_URL=https://meteor3.example.com
- PORT=3000
- NODE_ENV=${ENVIRONMENT}
frontoffice1:
image: registry.example.net/project/frontoffice1:latest
tty: true
depends_on:
- "api"
networks:
default:
aliases:
- "frontoffice"
registration:
image: registry.example.net/project/registration:latest
tty: true
depends_on:
- "mongodb"
- "api"
environment:
- ENVIRONMENT=prod
- NPM_CONFIG_LOGLEVEL=verbose
- MONGO_URL=mongodb://mongodb/registration
- MAIL_URL=smtp://mailcatcher:1025/
- ROOT_URL=https://registration.example.com
- PORT=3000
- NODE_ENV=${ENVIRONMENT}
corporate:
image: registry.example.net/project/corporate:latest
depends_on:
- "api-proxy"
environment:
- FOO=false
- TZ=CET-1CEST,M3.5.0,M10.5.0/3
corporate2:
image: registry.example.net/project/corporate2:latest
environment:
- TZ=CET-1CEST,M3.5.0,M10.5.0/3
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:6.2.1
networks:
default:
aliases:
- "elasticsearch"
- "es6"
- "es61"
volumes:
- "elasticsearch-data:/usr/share/elasticsearch/data"
environment:
- TZ=CET-1CEST,M3.5.0,M10.5.0/3
# kibana:
# image: docker.elastic.co/kibana/kibana:6.2.1
# depends_on:
# - "elasticsearch"
elasticsearch-145:
image: airdock/elasticsearch:1.4
networks:
default:
aliases:
- "es"
- "es1"
volumes:
- "elasticsearch-145-data:/usr/share/elasticsearch/data"
environment:
- TZ=CET-1CEST,M3.5.0,M10.5.0/3
mailcatcher:
image: registry.example.net/project/mailcatcher
memcached:
image: memcached:1.5
mongodb:
image: mongo:3.4
networks:
default:
aliases:
- "mongodb"
- "md"
- "md1"
volumes:
- "mongodb-data:/data/db"
environment:
- TZ=CET-1CEST,M3.5.0,M10.5.0/3
postgres:
image: postgres:9.6
networks:
default:
aliases:
- "postgres"
- "pg"
- "pg1"
volumes:
- "postgres-data:/var/lib/postgresql/data"
environment:
- POSTGRES_USER=foo
- POSTGRES_PASSWORD=
- POSTGRES_DB=project
redis:
image: redis:2.8
networks:
default:
aliases:
- "redis"
- "rd"
- "rd1"
volumes:
- "redis-data:/data"
environment:
- TZ=CET-1CEST,M3.5.0,M10.5.0/3
swagger:
image: swaggerapi/swagger-ui
environment:
- API_URL=http://api.${ENVIRONMENT}.example.com/swagger.json
volumes:
postgres-data:
elasticsearch-data:
elasticsearch-145-data:
redis-data:
mongodb-data:
labels:
logos:
invoices:
sepa:
storage:
uploads:
direct_debit_forms:
bank_transactions:
rsa-keys:
networks:
default:
assets2:
assets3:
assets1:
Anonymised contents of docker-compose.override.yml:
version: '3.2'
services:
api:
image: 'registry.example2.net:5000/project/api:latest'
environment:
- ENVIRONMENT=stage
- PLASTIC_HOST=elasticsearch
- MEMCACHED_HOST=memcached
- NEW_RELIC_LICENSE=foo
- BLACKFIRE_SERVER_ID=bar
- BLACKFIRE_SERVER_TOKEN=baz
- 'APP_URL=http://api.john.staging.example.com'
- APP_DEBUG=1
- DB_DATABASE=staging
- DB_USERNAME=foo
- NEW_RELIC_APP_NAME=api.john.staging.example.com
labels:
traefik.enable: 'true'
traefik.port: '8000'
traefik.frontend.rule: 'Host:api.john.staging.example.com'
traefik.docker.network: john_default
deploy:
resources:
limits:
cpus: '4'
memory: 1024M
labels:
traefik.enable: 'true'
traefik.port: '8000'
traefik.frontend.rule: 'Host:api.john.staging.example.com'
traefik.docker.network: john_default
api-proxy:
image: 'registry.example2.net:5000/project/api-proxy:latest'
environment:
- ENVIRONMENT=stage
labels:
traefik.enable: 'true'
traefik.port: '80'
traefik.frontend.rule: 'Host:proxy.john.staging.example.com'
traefik.docker.network: john_default
deploy:
resources:
limits:
cpus: '1'
memory: 128M
labels:
traefik.enable: 'true'
traefik.port: '80'
traefik.frontend.rule: 'Host:proxy.john.staging.example.com'
traefik.docker.network: john_default
api-uploads:
labels:
traefik.enable: 'true'
traefik.port: '80'
traefik.frontend.rule: 'Host:uploads.john.staging.example.com'
traefik.docker.network: john_default
deploy:
resources:
limits:
cpus: '1'
memory: 128M
labels:
traefik.enable: 'true'
traefik.port: '80'
traefik.frontend.rule: 'Host:uploads.john.staging.example.com'
traefik.docker.network: john_default
ui:
image: 'registry.example2.net:5000/project/ui:latest'
environment:
- FOO=false
labels:
traefik.enable: 'true'
traefik.port: '80'
traefik.frontend.rule: 'Host:ui.john.staging.example.com'
traefik.docker.network: john_default
deploy:
resources:
limits:
cpus: '1'
memory: 128M
labels:
traefik.enable: 'true'
traefik.port: '80'
traefik.frontend.rule: 'Host: ui.john.staging.example.com'
traefik.docker.network: john_default
admin:
image: 'registry.example2.net:5000/project/admin:latest'
environment:
- FOO=false
labels:
traefik.enable: 'true'
traefik.port: '80'
traefik.frontend.rule: 'Host:admin.john.staging.example.com'
traefik.docker.network: john_default
deploy:
resources:
limits:
cpus: '1'
memory: 128M
labels:
traefik.enable: 'true'
traefik.port: '80'
traefik.frontend.rule: 'Host: admin.john.staging.example.com'
traefik.docker.network: john_default
assets2:
image: 'registry.example2.net:5000/project/assets2:latest'
labels:
traefik.enable: 'true'
traefik.port: '80'
traefik.frontend.rule: 'Host:assets.john.staging.example2.com'
traefik.docker.network: john_default
deploy:
resources:
limits:
cpus: '1'
memory: 128M
labels:
traefik.enable: 'true'
traefik.port: '80'
traefik.frontend.rule: 'Host:assets.john.staging.example2.com'
traefik.docker.network: john_default
assets3:
image: 'registry.example2.net:5000/project/assets3:latest'
labels:
traefik.enable: 'true'
traefik.port: '80'
traefik.frontend.rule: 'Host:assets.john.staging.example3.com'
traefik.docker.network: john_default
deploy:
resources:
limits:
cpus: '1'
memory: 128M
labels:
traefik.enable: 'true'
traefik.port: '80'
traefik.frontend.rule: 'Host:assets.john.staging.example3.com'
traefik.docker.network: john_default
assets1:
image: 'registry.example2.net:5000/project/assets1:latest'
labels:
traefik.enable: 'true'
traefik.port: '80'
traefik.frontend.rule: 'Host:assets.john.staging.example.com'
traefik.docker.network: john_default
deploy:
resources:
limits:
cpus: '1'
memory: 512M
labels:
traefik.enable: 'true'
traefik.port: '80'
traefik.frontend.rule: 'Host:assets.john.staging.example.com'
traefik.docker.network: john_default
meteor2:
image: 'registry.example2.net:5000/project/meteor2:latest'
environment:
- ENVIRONMENT=stage
- NPM_CONFIG_LOGLEVEL=verbose
- 'MONGO_URL=mongodb://mongodb/meteor2'
- 'MAIL_URL=smtp://mailcatcher:1025/'
- 'ROOT_URL=https://meteor.john.staging.example2.com'
- PORT=3000
- NODE_ENV=stage
labels:
traefik.enable: 'true'
traefik.port: '3000'
traefik.frontend.rule: 'Host:meteor.john.staging.example2.com'
traefik.docker.network: john_default
deploy:
resources:
limits:
cpus: '2'
memory: 512M
labels:
traefik.enable: 'true'
traefik.port: '3000'
traefik.frontend.rule: 'Host:meteor.john.staging.example2.com'
traefik.docker.network: john_default
meteor3:
image: 'registry.example2.net:5000/project/meteor3:latest'
environment:
- ENVIRONMENT=stage
- NPM_CONFIG_LOGLEVEL=verbose
- 'MONGO_URL=mongodb://mongodb/meteor3'
- 'MAIL_URL=smtp://mailcatcher:1025/'
- 'ROOT_URL=https://meteor.john.staging.example3.com'
- PORT=3000
- NODE_ENV=stage
labels:
traefik.enable: 'true'
traefik.port: '3000'
traefik.frontend.rule: 'Host:meteor.john.staging.example3.com'
traefik.docker.network: john_default
deploy:
resources:
limits:
cpus: '2'
memory: 512M
labels:
traefik.enable: 'true'
traefik.port: '3000'
traefik.frontend.rule: 'Host:meteor.john.staging.example3.com'
traefik.docker.network: john_default
frontoffice1:
image: 'registry.example2.net:5000/project/frontoffice1:latest'
registration:
image: 'registry.example2.net:5000/project/registration:latest'
environment:
- ENVIRONMENT=stage
- NPM_CONFIG_LOGLEVEL=verbose
- 'MONGO_URL=mongodb://mongodb/registration'
- 'MAIL_URL=smtp://mailcatcher:1025/'
- 'ROOT_URL=https://registration.john.staging.example.com'
- PORT=3000
- NODE_ENV=stage
labels:
traefik.enable: 'true'
traefik.port: '3000'
traefik.frontend.rule: 'Host:registration.john.staging.example.com'
traefik.docker.network: john_default
deploy:
resources:
limits:
cpus: '1'
memory: 512M
labels:
traefik.enable: 'true'
traefik.port: '3000'
traefik.frontend.rule: 'Host:registration.john.staging.example.com'
traefik.docker.network: john_default
corporate:
image: 'registry.example2.net:5000/project/corporate:latest'
environment:
- FOO=false
labels:
traefik.enable: 'true'
traefik.port: '80'
traefik.frontend.rule: 'Host:www.john.staging.example.com'
traefik.docker.network: john_default
deploy:
resources:
limits:
cpus: '1'
memory: 128M
labels:
traefik.enable: 'true'
traefik.port: '80'
traefik.frontend.rule: 'Host:www.john.staging.example.com'
traefik.docker.network: john_default
corporate2:
image: 'registry.example2.net:5000/project/corporate2:latest'
labels:
traefik.enable: 'true'
traefik.port: '80'
traefik.frontend.rule: 'Host:www.john.staging.example2.com'
traefik.docker.network: john_default
deploy:
resources:
limits:
cpus: '1'
memory: 128M
labels:
traefik.enable: 'true'
traefik.port: '80'
traefik.frontend.rule: 'Host:www.john.staging.example2.com'
traefik.docker.network: john_default
postgres:
environment:
- POSTGRES_DB=staging
- POSTGRES_USER=foo
- POSTGRES_PASSWORD=
deploy:
resources:
limits:
cpus: '2'
memory: 1024M
elasticsearch:
environment:
- 'TZ=CET-1CEST,M3.5.0,M10.5.0/3'
- VIRTUAL_PORT=9200
- VIRTUAL_HOST=es6.john.staging.example.com
- discovery.type=single-node
labels:
traefik.enable: 'true'
traefik.port: '9200'
traefik.frontend.rule: 'Host:es6.john.staging.example.com'
traefik.docker.network: john_default
deploy:
resources:
limits:
cpus: '1'
memory: 2048M
labels:
traefik.enable: 'true'
traefik.port: '9200'
traefik.frontend.rule: 'Host:es6.john.staging.example.com'
traefik.docker.network: john_default
elasticsearch-145:
environment:
- 'TZ=CET-1CEST,M3.5.0,M10.5.0/3'
- 'ES_JAVA_OPTS=-Xms1m -Xmx512m'
labels:
traefik.enable: 'true'
traefik.port: '9200'
traefik.frontend.rule: 'Host:es.john.staging.example.com'
traefik.docker.network: john_default
deploy:
resources:
limits:
cpus: '1'
memory: 1024M
labels:
traefik.enable: 'true'
traefik.port: '9200'
traefik.frontend.rule: 'Host:es.john.staging.example.com'
traefik.docker.network: john_default
mailcatcher:
image: 'registry.example2.net:5000/project/mailcatcher:latest'
labels:
traefik.enable: 'true'
traefik.port: '1080'
traefik.frontend.rule: 'Host:mailcatcher.john.staging.example.com'
traefik.docker.network: john_default
deploy:
resources:
limits:
cpus: '1'
memory: 128M
labels:
traefik.enable: 'true'
traefik.port: '1080'
traefik.frontend.rule: 'Host:mailcatcher.john.staging.example.com'
traefik.docker.network: john_default
networks:
default:
driver: bridge
attachable: true
assets2:
driver: bridge
attachable: true
assets3:
driver: bridge
attachable: true
assets1:
driver: bridge
attachable: true