Skip to content

Docker (compose?) suddenly creates bridge using 192.168.16.0/20 range #37823

@AlwinGarside

Description

@AlwinGarside

I have an Ubuntu server, running Docker without any issues. It has several projects managed with docker-compose. Suddenly a docker-compose up of one of those projects caused the server to become unresponsive on the network.

This server uses the ip range 192.168.64.0/24 and has the IP 192.168.64.2 with 192.168.64.1 being the router that servers as gateway for this server.

This router routes traffic to and from the office network at 192.168.16.0/20.

After examining the server by logging in physically, it becomes clear that Docker or Docker Compose had suddenly created a bridge interface with ip 192.168.16.1 in the 192.168.16.0/20 range which made it impossible for it to connect to our office network.

Steps to reproduce the issue:
Unfortunately I'm unable to reproduce this issue reliably.

Describe the results you received:
A bridge interface with ip 192.168.16.1 was created in the 192.168.16.0/20 range.

Describe the results you expected:
A bridge interface to be created in a 172.*.0.0/16 range.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:24:51 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:23:15 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker info:

Containers: 21
 Running: 0
 Paused: 0
 Stopped: 21
Images: 67
Server Version: 18.06.1-ce
Storage Driver: btrfs
 Build Version: Btrfs v4.15.1
 Library Version: 102
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-33-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 20
Total Memory: 94.38GiB
Name: bumblebee
ID: 5Q5F:LBIJ:OM5P:7EAI:PJU3:7GRH:ESXL:ZLWQ:V2SK:P6L7:ZEHA:CHDU
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):
physical

Output of docker network inspect:

[
    {
        "Name": "john_assets3",
        "Id": "a319f31dc146dd7a2541cb935b3a66998d9d0ff2476565fda2cdecc213363456",
        "Created": "2018-09-11T08:38:38.403766297Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "192.168.16.0/20",
                    "Gateway": "192.168.16.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "assets3",
            "com.docker.compose.project": "john",
            "com.docker.compose.version": "1.21.2"
        }
    }
]

Anonymised contents of docker-compose.yml:

version: "3.2"
services:

  api:
    image: registry.example.net/project/api:latest
    tty: true
    depends_on:
      - "postgres"
      - "redis"
      - "elasticsearch-145"
      - "mailcatcher"
      - "api-uploads"
      - "memcached"
      - "elasticsearch"
    networks:
      default:
        aliases:
          - "api-alias"
          - "api-alias1"
          - "proxy1"
          - "proxy2"
          - "proxy3"
          - "proxy4"
          - "proxy5"
          - "proxy6"
          - "proxy7"
          - "proxy8"
          - "proxy9"
    volumes:
      - "labels:/var/www/html/labels"
      - "logos:/var/www/html/logos"
      - "invoices:/var/www/html/invoices"
      - "sepa:/var/www/html/sepa"
      - "storage:/var/www/html/storage"
      - "uploads:/var/www/html/uploads"
      - "direct_debit_forms:/var/www/html/direct_debit_forms"
      - "bank_transactions:/var/www/html/bank_transactions"
      - "rsa-keys:/var/www/html/config/rsa-keys"
    environment:
      - ENVIRONMENT=prod
      - PLASTIC_HOST=elasticsearch
      - MEMCACHED_HOST=memcached
      - NEW_RELIC_LICENSE=foo
      - BLACKFIRE_SERVER_ID=bar
      - BLACKFIRE_SERVER_TOKEN=baz

  api-proxy:
    image: registry.example.net/project/api-proxy:latest
    tty: true
    depends_on:
      - "api"
    environment:
      - ENVIRONMENT=prod

  api-uploads:
    image: nginx:alpine
    volumes:
      - "uploads:/usr/share/nginx/html"

  ui:
    image: registry.example.net/project/ui:latest
    depends_on:
      - "api"
      - "frontoffice"
      - "assets1"
    networks:
      default:
      assets3:
    environment:
      - FOO=false

  admin:
    image: registry.example.net/project/admin:latest
    depends_on:
      - "api"
    environment:
      - FOO=false

  assets2:
    image: registry.example.net/project/assets2:latest
    networks:
      default:
      assets2:
        aliases:
          - "assets"

  assets3:
    image: registry.example.net/project/assets3:latest
    networks:
      default:
      assets3:
        aliases:
          - "assets"

  assets1:
    image: registry.example.net/project/assets1:latest
    networks:
      default:
      assets1:
        aliases:
          - "assets"

  meteor2:
    image: registry.example.net/project/meteor2:latest
    tty: true
    depends_on:
      - "mongodb"
      - "api"
      - "assets2"
    networks:
      default:
      assets2:
    environment:
      - ENVIRONMENT=prod
      - NPM_CONFIG_LOGLEVEL=verbose
      - MONGO_URL=mongodb://mongodb/meteor2
      - MAIL_URL=smtp://mailcatcher:1025/
      - ROOT_URL=https://meteor2.example.com
      - PORT=3000
      - NODE_ENV=${ENVIRONMENT}

  meteor3:
    image: registry.example.net/project/meteor3:latest
    tty: true
    depends_on:
      - "mongodb"
      - "api"
      - "assets3"
    networks:
      default:
      assets3:
    environment:
      - ENVIRONMENT=prod
      - NPM_CONFIG_LOGLEVEL=verbose
      - MONGO_URL=mongodb://mongodb/meteor3
      - MAIL_URL=smtp://mailcatcher:1025/
      - ROOT_URL=https://meteor3.example.com
      - PORT=3000
      - NODE_ENV=${ENVIRONMENT}

  frontoffice1:
    image: registry.example.net/project/frontoffice1:latest
    tty: true
    depends_on:
      - "api"
    networks:
      default:
        aliases:
          - "frontoffice"

  registration:
    image: registry.example.net/project/registration:latest
    tty: true
    depends_on:
      - "mongodb"
      - "api"
    environment:
      - ENVIRONMENT=prod
      - NPM_CONFIG_LOGLEVEL=verbose
      - MONGO_URL=mongodb://mongodb/registration
      - MAIL_URL=smtp://mailcatcher:1025/
      - ROOT_URL=https://registration.example.com
      - PORT=3000
      - NODE_ENV=${ENVIRONMENT}

  corporate:
    image: registry.example.net/project/corporate:latest
    depends_on:
      - "api-proxy"
    environment:
      - FOO=false
      - TZ=CET-1CEST,M3.5.0,M10.5.0/3

  corporate2:
    image: registry.example.net/project/corporate2:latest
    environment:
      - TZ=CET-1CEST,M3.5.0,M10.5.0/3

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.2.1
    networks:
      default:
        aliases:
          - "elasticsearch"
          - "es6"
          - "es61"
    volumes:
      - "elasticsearch-data:/usr/share/elasticsearch/data"
    environment:
      - TZ=CET-1CEST,M3.5.0,M10.5.0/3

#  kibana:
#    image: docker.elastic.co/kibana/kibana:6.2.1
#    depends_on:
#      - "elasticsearch"

  elasticsearch-145:
    image: airdock/elasticsearch:1.4
    networks:
      default:
        aliases:
        - "es"
        - "es1"
    volumes:
      - "elasticsearch-145-data:/usr/share/elasticsearch/data"
    environment:
      - TZ=CET-1CEST,M3.5.0,M10.5.0/3

  mailcatcher:
    image: registry.example.net/project/mailcatcher

  memcached:
    image: memcached:1.5

  mongodb:
    image: mongo:3.4
    networks:
      default:
        aliases:
          - "mongodb"
          - "md"
          - "md1"
    volumes:
      - "mongodb-data:/data/db"
    environment:
      - TZ=CET-1CEST,M3.5.0,M10.5.0/3

  postgres:
    image: postgres:9.6
    networks:
      default:
        aliases:
          - "postgres"
          - "pg"
          - "pg1"
    volumes:
      - "postgres-data:/var/lib/postgresql/data"
    environment:
      - POSTGRES_USER=foo
      - POSTGRES_PASSWORD=
      - POSTGRES_DB=project

  redis:
    image: redis:2.8
    networks:
      default:
        aliases:
          - "redis"
          - "rd"
          - "rd1"
    volumes:
      - "redis-data:/data"
    environment:
      - TZ=CET-1CEST,M3.5.0,M10.5.0/3

  swagger:
    image: swaggerapi/swagger-ui
    environment:
      - API_URL=http://api.${ENVIRONMENT}.example.com/swagger.json

volumes:
  postgres-data:
  elasticsearch-data:
  elasticsearch-145-data:
  redis-data:
  mongodb-data:
  labels:
  logos:
  invoices:
  sepa:
  storage:
  uploads:
  direct_debit_forms:
  bank_transactions:
  rsa-keys:

networks:
  default:
  assets2:
  assets3:
  assets1:

Anonymised contents of docker-compose.override.yml:

version: '3.2'
services:
  api:
    image: 'registry.example2.net:5000/project/api:latest'
    environment:
      - ENVIRONMENT=stage
      - PLASTIC_HOST=elasticsearch
      - MEMCACHED_HOST=memcached
      - NEW_RELIC_LICENSE=foo
      - BLACKFIRE_SERVER_ID=bar
      - BLACKFIRE_SERVER_TOKEN=baz
      - 'APP_URL=http://api.john.staging.example.com'
      - APP_DEBUG=1
      - DB_DATABASE=staging
      - DB_USERNAME=foo
      - NEW_RELIC_APP_NAME=api.john.staging.example.com
    labels:
      traefik.enable: 'true'
      traefik.port: '8000'
      traefik.frontend.rule: 'Host:api.john.staging.example.com'
      traefik.docker.network: john_default
    deploy:
      resources:
        limits:
          cpus: '4'
          memory: 1024M
      labels:
        traefik.enable: 'true'
        traefik.port: '8000'
        traefik.frontend.rule: 'Host:api.john.staging.example.com'
        traefik.docker.network: john_default

  api-proxy:
    image: 'registry.example2.net:5000/project/api-proxy:latest'
    environment:
      - ENVIRONMENT=stage
    labels:
      traefik.enable: 'true'
      traefik.port: '80'
      traefik.frontend.rule: 'Host:proxy.john.staging.example.com'
      traefik.docker.network: john_default
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 128M
      labels:
        traefik.enable: 'true'
        traefik.port: '80'
        traefik.frontend.rule: 'Host:proxy.john.staging.example.com'
        traefik.docker.network: john_default

  api-uploads:
    labels:
      traefik.enable: 'true'
      traefik.port: '80'
      traefik.frontend.rule: 'Host:uploads.john.staging.example.com'
      traefik.docker.network: john_default
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 128M
      labels:
        traefik.enable: 'true'
        traefik.port: '80'
        traefik.frontend.rule: 'Host:uploads.john.staging.example.com'
        traefik.docker.network: john_default

  ui:
    image: 'registry.example2.net:5000/project/ui:latest'
    environment:
      - FOO=false
    labels:
      traefik.enable: 'true'
      traefik.port: '80'
      traefik.frontend.rule: 'Host:ui.john.staging.example.com'
      traefik.docker.network: john_default
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 128M
      labels:
        traefik.enable: 'true'
        traefik.port: '80'
        traefik.frontend.rule: 'Host: ui.john.staging.example.com'
        traefik.docker.network: john_default

  admin:
    image: 'registry.example2.net:5000/project/admin:latest'
    environment:
      - FOO=false
    labels:
      traefik.enable: 'true'
      traefik.port: '80'
      traefik.frontend.rule: 'Host:admin.john.staging.example.com'
      traefik.docker.network: john_default
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 128M
      labels:
        traefik.enable: 'true'
        traefik.port: '80'
        traefik.frontend.rule: 'Host: admin.john.staging.example.com'
        traefik.docker.network: john_default

  assets2:
    image: 'registry.example2.net:5000/project/assets2:latest'
    labels:
      traefik.enable: 'true'
      traefik.port: '80'
      traefik.frontend.rule: 'Host:assets.john.staging.example2.com'
      traefik.docker.network: john_default
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 128M
      labels:
        traefik.enable: 'true'
        traefik.port: '80'
        traefik.frontend.rule: 'Host:assets.john.staging.example2.com'
        traefik.docker.network: john_default

  assets3:
    image: 'registry.example2.net:5000/project/assets3:latest'
    labels:
      traefik.enable: 'true'
      traefik.port: '80'
      traefik.frontend.rule: 'Host:assets.john.staging.example3.com'
      traefik.docker.network: john_default
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 128M
      labels:
        traefik.enable: 'true'
        traefik.port: '80'
        traefik.frontend.rule: 'Host:assets.john.staging.example3.com'
        traefik.docker.network: john_default

  assets1:
    image: 'registry.example2.net:5000/project/assets1:latest'
    labels:
      traefik.enable: 'true'
      traefik.port: '80'
      traefik.frontend.rule: 'Host:assets.john.staging.example.com'
      traefik.docker.network: john_default
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 512M
      labels:
        traefik.enable: 'true'
        traefik.port: '80'
        traefik.frontend.rule: 'Host:assets.john.staging.example.com'
        traefik.docker.network: john_default

  meteor2:
    image: 'registry.example2.net:5000/project/meteor2:latest'
    environment:
      - ENVIRONMENT=stage
      - NPM_CONFIG_LOGLEVEL=verbose
      - 'MONGO_URL=mongodb://mongodb/meteor2'
      - 'MAIL_URL=smtp://mailcatcher:1025/'
      - 'ROOT_URL=https://meteor.john.staging.example2.com'
      - PORT=3000
      - NODE_ENV=stage
    labels:
      traefik.enable: 'true'
      traefik.port: '3000'
      traefik.frontend.rule: 'Host:meteor.john.staging.example2.com'
      traefik.docker.network: john_default
    deploy:
      resources:
        limits:
          cpus: '2'
          memory: 512M
      labels:
        traefik.enable: 'true'
        traefik.port: '3000'
        traefik.frontend.rule: 'Host:meteor.john.staging.example2.com'
        traefik.docker.network: john_default

  meteor3:
    image: 'registry.example2.net:5000/project/meteor3:latest'
    environment:
      - ENVIRONMENT=stage
      - NPM_CONFIG_LOGLEVEL=verbose
      - 'MONGO_URL=mongodb://mongodb/meteor3'
      - 'MAIL_URL=smtp://mailcatcher:1025/'
      - 'ROOT_URL=https://meteor.john.staging.example3.com'
      - PORT=3000
      - NODE_ENV=stage
    labels:
      traefik.enable: 'true'
      traefik.port: '3000'
      traefik.frontend.rule: 'Host:meteor.john.staging.example3.com'
      traefik.docker.network: john_default
    deploy:
      resources:
        limits:
          cpus: '2'
          memory: 512M
      labels:
        traefik.enable: 'true'
        traefik.port: '3000'
        traefik.frontend.rule: 'Host:meteor.john.staging.example3.com'
        traefik.docker.network: john_default

  frontoffice1:
    image: 'registry.example2.net:5000/project/frontoffice1:latest'

  registration:
    image: 'registry.example2.net:5000/project/registration:latest'
    environment:
      - ENVIRONMENT=stage
      - NPM_CONFIG_LOGLEVEL=verbose
      - 'MONGO_URL=mongodb://mongodb/registration'
      - 'MAIL_URL=smtp://mailcatcher:1025/'
      - 'ROOT_URL=https://registration.john.staging.example.com'
      - PORT=3000
      - NODE_ENV=stage
    labels:
      traefik.enable: 'true'
      traefik.port: '3000'
      traefik.frontend.rule: 'Host:registration.john.staging.example.com'
      traefik.docker.network: john_default
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 512M
      labels:
        traefik.enable: 'true'
        traefik.port: '3000'
        traefik.frontend.rule: 'Host:registration.john.staging.example.com'
        traefik.docker.network: john_default

  corporate:
    image: 'registry.example2.net:5000/project/corporate:latest'
    environment:
      - FOO=false
    labels:
      traefik.enable: 'true'
      traefik.port: '80'
      traefik.frontend.rule: 'Host:www.john.staging.example.com'
      traefik.docker.network: john_default
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 128M
      labels:
        traefik.enable: 'true'
        traefik.port: '80'
        traefik.frontend.rule: 'Host:www.john.staging.example.com'
        traefik.docker.network: john_default

  corporate2:
    image: 'registry.example2.net:5000/project/corporate2:latest'
    labels:
      traefik.enable: 'true'
      traefik.port: '80'
      traefik.frontend.rule: 'Host:www.john.staging.example2.com'
      traefik.docker.network: john_default
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 128M
      labels:
        traefik.enable: 'true'
        traefik.port: '80'
        traefik.frontend.rule: 'Host:www.john.staging.example2.com'
        traefik.docker.network: john_default

  postgres:
    environment:
      - POSTGRES_DB=staging
      - POSTGRES_USER=foo
      - POSTGRES_PASSWORD=
    deploy:
      resources:
        limits:
          cpus: '2'
          memory: 1024M

  elasticsearch:
    environment:
      - 'TZ=CET-1CEST,M3.5.0,M10.5.0/3'
      - VIRTUAL_PORT=9200
      - VIRTUAL_HOST=es6.john.staging.example.com
      - discovery.type=single-node
    labels:
      traefik.enable: 'true'
      traefik.port: '9200'
      traefik.frontend.rule: 'Host:es6.john.staging.example.com'
      traefik.docker.network: john_default
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 2048M
      labels:
        traefik.enable: 'true'
        traefik.port: '9200'
        traefik.frontend.rule: 'Host:es6.john.staging.example.com'
        traefik.docker.network: john_default

  elasticsearch-145:
    environment:
      - 'TZ=CET-1CEST,M3.5.0,M10.5.0/3'
      - 'ES_JAVA_OPTS=-Xms1m -Xmx512m'
    labels:
      traefik.enable: 'true'
      traefik.port: '9200'
      traefik.frontend.rule: 'Host:es.john.staging.example.com'
      traefik.docker.network: john_default
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 1024M
      labels:
        traefik.enable: 'true'
        traefik.port: '9200'
        traefik.frontend.rule: 'Host:es.john.staging.example.com'
        traefik.docker.network: john_default

  mailcatcher:
    image: 'registry.example2.net:5000/project/mailcatcher:latest'
    labels:
      traefik.enable: 'true'
      traefik.port: '1080'
      traefik.frontend.rule: 'Host:mailcatcher.john.staging.example.com'
      traefik.docker.network: john_default
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 128M
      labels:
        traefik.enable: 'true'
        traefik.port: '1080'
        traefik.frontend.rule: 'Host:mailcatcher.john.staging.example.com'
        traefik.docker.network: john_default

networks:
  default:
    driver: bridge
    attachable: true
  assets2:
    driver: bridge
    attachable: true
  assets3:
    driver: bridge
    attachable: true
  assets1:
    driver: bridge
    attachable: true

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions