WITH DOCKER is not correctly releasing network resources (and most likely, the network resource is not isolated to a container)

[alexcb]

What went wrong?

Consider a docker-compose.yml:

name: composure

networks:
  frontend:
    driver: bridge
    ipam:
      config:
        - subnet: 192.168.200.0/24

  backend:
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 10.0.0.0/16

services:
  server:
    image: nginx:latest
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 10s
    networks:
      - frontend
      - backend

  client:
    image: client:latest
    command: sleep 999
    depends_on:
      server:
        condition: service_healthy
    networks:
      - frontend

and Earthfile:

VERSION 0.7

client:
  FROM ubuntu:latest
  RUN apt update && apt install -y iproute2 curl iputils-ping

test:
    FROM earthly/dind:ubuntu
    COPY ./docker-compose.yml ./
    ARG CACHE_BUSTER
    RUN echo "running test $CACHE_BUSTER"
    WITH DOCKER \
        --pull nginx:latest \
        --load client:latest=+client
        RUN docker compose up --detach --wait --wait-timeout 15 && docker ps && docker compose exec client /bin/sh -c "ip address && ip route && ping -W 10 -c 3 server && echo YWxsIGdvb2QK | base64 -d"; export code=$? && (docker ps -q | xargs -t -n 1 docker logs) && exit $code
    END

When running many instances of earthly +test --CACHE_BUSTER=$RANDOM (e.g. 30 or more), some will fail with:

            +breakit *failed* | --- server ping statistics ---
            +breakit *failed* | 3 packets transmitted, 0 received, 100% packet loss, time 2044ms

It's interesting to note, when the networks section is removed from the docker-compose.yml file (i.e. all services are only on the same default network), this reproduction-case no longer occurs.

Update: Parallelism isn't required here, simply running earthly -P +test --CACHE_BUSTER=$RANDOM && earthly -P +test --CACHE_BUSTER=$RANDOM will reproduce this problem -- the first run succeeds, and the second consistently fails.

Potential work-around

This problem does not occur when creating an internal network via docker network create --internal ... or by setting the internal: true option in the docker-compose file. This work-around will not work for cases where you expect a network that's externally accessible.

[alexcb]

Here's a similar reproduction of this error that does not require docker compose:

VERSION 0.7

client:
  FROM ubuntu:latest
  RUN apt update && apt install -y iproute2 curl iputils-ping

test:
  FROM earthly/dind:ubuntu-23.04-docker-24.0.5-1
  ARG CACHE_BUSTER
  RUN echo "running test $CACHE_BUSTER"
  WITH DOCKER --load myclient:latest=+client
    RUN \
        docker network create --subnet 192.168.200.0/24 foo && \
        docker run --network foo --rm --name host1 -d myclient:latest /bin/sh -c 'sleep 999' && \
        docker run --network foo --rm --name host2 -d myclient:latest /bin/sh -c 'sleep 999' && \
        docker exec host1 /bin/sh -c 'ip address && ip route' && \
        docker exec host2 /bin/sh -c 'ip address && ip route' && \
        docker exec host1 /bin/sh -c 'ping -W 1 -c 3 host2' && \
        docker exec host2 /bin/sh -c 'ping -W 1 -c 3 host1'
  END

A notable discovery, is this does not require high levels of parallelism, it can also be reproduced with a sequential loop:

for i in $(seq 1 100); do echo "test $i" && earthly -P +test --CACHE_BUSTER=$RANDOM || break; done

and it seems to be consistently erroring on the second iteration (even after killing the earthly-buildkit container, and even restarting docker).

[alexcb]

work-around: clean up after yourself

test:
  FROM earthly/dind:ubuntu-23.04-docker-24.0.5-1
  ARG CACHE_BUSTER
  RUN echo "running test $CACHE_BUSTER"
  WITH DOCKER --load myclient:latest=+client
    RUN \
        docker network create --subnet 192.168.200.0/24 foo && \
        docker run --network foo --rm --name host1 -d myclient:latest /bin/sh -c 'sleep 999' && \
        docker run --network foo --rm --name host2 -d myclient:latest /bin/sh -c 'sleep 999' && \
        docker exec host1 /bin/sh -c 'ip address && ip route' && \
        docker exec host2 /bin/sh -c 'ip address && ip route' && \
        docker exec host1 /bin/sh -c 'ping -W 1 -c 3 host2' && \
        docker exec host2 /bin/sh -c 'ping -W 1 -c 3 host1' && \
        docker rm -f host1 && \
        docker rm -f host2 && \
        docker network rm foo  # <--   After adding this command, this error no longer occurs
  END

[alexcb]

I expanded my test to have a simple python-based udp server/client which would send messages in a loop, and was able to start two instances in parallel, both of which started and continued working.

I then shut down the first instance, but left the second instance running, and it continued to work.

At that point I started a third instance (leaving the second instance up and running), and it's only that third instance that failed -- the second instance continued to run.

The code:

# server.py
import socket
import sys

host = sys.argv[1]
port = int(sys.argv[2])
msg = sys.argv[3]

print(f'creating server on {host}:{port} that will print {msg}', flush=True)
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind((host, port))

while True:
    data, address = sock.recvfrom(4096)
    print('received %s bytes from %s' % (data.decode('utf8'), address), flush=True)
    if data:
        sent = sock.sendto(msg.encode('utf8'), address)

# client.py
import socket
import sys

host = sys.argv[1]
port = int(sys.argv[2])
msg = sys.argv[3]

print(f'creating udp socket to send to {host}:{port}', flush=True)
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
try:
    print(f'sending {msg} to {host}:{port}', flush=True)
    sent = sock.sendto(msg.encode('utf8'), (host, port))
    print(f'waiting for response', flush=True)
    data, server = sock.recvfrom(4096)
    print('received "%s"' % data)
finally:
    sock.close()

# Earthfile
VERSION 0.7

client:
  FROM ubuntu:latest
  COPY server.py client.py .
  RUN apt update && apt install -y iproute2 curl iputils-ping python3

test:
  FROM earthly/dind:ubuntu-23.04-docker-24.0.5-1
  ARG CACHE_BUSTER
  RUN echo "running test $CACHE_BUSTER"
  WITH DOCKER --load myclient:latest=+client
    RUN \
        docker network create --subnet 192.168.200.0/24 foo && \
        docker run --network foo --rm --name host1 myclient:latest /bin/sh -c "python3 server.py 0.0.0.0 1234 server$CACHE_BUSTER" & \
        docker run --network foo --rm --name host2 -d myclient:latest /bin/sh -c 'sleep 999' && \
        docker exec host1 /bin/sh -c 'ip address && ip route' && \
        docker exec host2 /bin/sh -c 'ip address && ip route' && \
        docker ps -a && \
        echo "sending udp" && \
        docker exec host2 /bin/sh -c "while true; do python3 client.py host1 1234 client$CACHE_BUSTER; sleep 10; done"
  END

[alexcb]

Here's another reproduction case:

VERSION 0.7

test:
  FROM docker:dind
  COPY test-inside-earthly.sh .
  RUN --no-cache --privileged dockerd-entrypoint.sh & ./test-inside-earthly.sh

and

#!/bin/sh
set -e
while ! [ -S /var/run/docker.sock ]; do echo "waiting for dind daemone to start" && sleep 1; done
docker ps -a
docker network create --subnet 192.168.200.0/24 foo
docker network ls
docker run -d --network foo --name host1 alpine /bin/sh -c "sleep 999"
docker run -d --network foo --name host2 alpine /bin/sh -c "sleep 999"
docker exec host2 /bin/sh -c "ping -W 1 -c 3 host1"
echo done

The first time this works, and on the second time we get 100% packet loss.

If however you run docker:dind directly (without earthly), it is re-entrant. So the ultimate question is what is the outer docker run doing that runc is not.

[alexcb]

This issue does not occur when using the --internal network option, which "[restricts] external access to the network".

Here's a full example:

VERSION 0.7

test:
  FROM earthly/dind:alpine
  ARG CACHE_BUSTER
  RUN echo "running test $CACHE_BUSTER"
  WITH DOCKER --pull alpine:latest
    RUN \
        docker network create --subnet 192.168.200.0/24 --internal foo && \   # Note the --internal flag
        docker run --network foo --rm --name host1 -d alpine:latest /bin/sh -c 'sleep 999' && \
        docker run --network foo --rm --name host2 -d alpine:latest /bin/sh -c 'sleep 999' && \
        docker exec host1 /bin/sh -c 'ifconfig' && \
        docker exec host2 /bin/sh -c 'ifconfig' && \
        docker exec host1 /bin/sh -c 'ping -W 1 -c 3 host2'
  END

which can be run multiple times: earthly -P +test --CACHE_BUSTER=$RANDOM && earthly -P +test --CACHE_BUSTER=$RANDOM

[alexcb]

Note that the --internal work-around posted above, also works with docker-compose networks by using internal: true, e.g.

networks:
  frontend:
    driver: bridge
    internal: true     # example of setting a network to "internal" 
    ipam:
      config:
        - subnet: 192.168.200.0/24

Note that this work-around requires changing the internal value. Be careful not to confuse this with the unrelated external setting.

[kp-mariappan-ramasamy]

Amazing. Thanks for the find @alexcb.
I am trying this config. Will update the results after testing.

[alexcb]

We have documented that internal networks should be used whenever possible under #3515

It's not clear what's needed to support the external networks -- if anyone runs into a specific use-case for external networks, please share it with us. Until then this issue will be deprioritized.

[kp-mariappan-ramasamy]

Thanks @alexcb
Yes, with our testing, internal network does not have this issue and works perfectly in parallel test execution.