Skip to content

e2e: fix docker network pool exhaustion#2834

Merged
snormore merged 1 commit intomainfrom
e2e-docker-network-cleanup
Feb 5, 2026
Merged

e2e: fix docker network pool exhaustion#2834
snormore merged 1 commit intomainfrom
e2e-docker-network-cleanup

Conversation

@snormore
Copy link
Contributor

@snormore snormore commented Feb 5, 2026

Summary

Fixes Docker network address pool exhaustion errors in CI:

failed to create default network: failed to create network: create network: Error response from daemon: all predefined address pools have been fully subnetted

Problem

The e2e tests create two Docker networks per test:

  1. Default network - for general container communication
  2. CYOA network - for client/device traffic

The default network was not specifying an IPAM config, causing Docker to allocate from its built-in address pool. Docker's default pool (172.17.0.0/12) was allocating /16 subnets, which only allows ~15 networks total. With parallel=12 tests and accumulated orphaned networks from previous runs, this pool was being exhausted.

Evidence from the CI runner:

172.17.0.0/16
172.18.0.0/16
...
172.28.0.0/16   # 10 /16 networks = nearly exhausted

Solution

  1. Use separate IP ranges for each network type:

    • Default network: 10.0.0.0/8 range with /24 subnets (65,536 available)
    • CYOA network: 9.128.0.0/9 range with /24 subnets (32,768 available)
  2. CI cleanup: Add pre-test step to prune orphaned networks from previous runs

The separation also fixes a test failure in TestE2E_DeviceMaxusersRollover where tests detecting interfaces by IP range (grep 'inet 9.') would incorrectly match both networks when they shared the same range.

Before vs After

Network Before After
Default Docker pool (172.x/16) ~15 max 10.x.x.x/24 ~65k max
CYOA 9.x.x.x/24 ~32k max 9.x.x.x/24 ~32k max (unchanged)

Testing Verification

  • go build ./e2e/... compiles successfully
  • CI e2e tests pass without network exhaustion errors
  • Tests that grep for inet 9. correctly identify only CYOA interfaces

@snormore snormore force-pushed the e2e-docker-network-cleanup branch from 12243d7 to 073a084 Compare February 5, 2026 21:07
The default network creation was not specifying an IPAM config, causing
Docker to allocate from its built-in address pool. Docker's default pool
only supports ~15 /16 networks, which was being exhausted when running
parallel tests on CI.

Now the default network uses explicit /24 subnets from 10.0.0.0/8, while
the CYOA network continues using 9.128.0.0/9. This separation also fixes
a test failure where tests detecting interfaces by IP range (grep 'inet 9.')
would incorrectly match both networks.

Also adds a pre-test cleanup step to remove orphaned networks from previous
runs.
@snormore snormore force-pushed the e2e-docker-network-cleanup branch from 073a084 to 143f0f1 Compare February 5, 2026 22:53
@snormore snormore marked this pull request as ready for review February 5, 2026 23:22
@snormore snormore merged commit 0542cb6 into main Feb 5, 2026
24 checks passed
@snormore snormore deleted the e2e-docker-network-cleanup branch February 5, 2026 23:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants