[NotBug]: Container network becomes unreachable after system restarts

[adiled]

I have done the following

• I have searched the existing issues
• If possible, I've reproduced the issue using the 'main' branch of this project

Steps to reproduce

1. Create a network and start a container with --publish port forwarding:

   container network create mynet --subnet 192.168.100.0/24
   container create --name mynginx --net mynet --publish 127.0.0.1:8000:8000 docker.io/library/nginx:alpine
   container start mynginx
   

2. Verify the container is reachable from the host via the gateway IP (192.168.100.1) and via the published port (127.0.0.1:8000). Both work.
3. Restart the Mac (full system restart, not just sleep/wake).
4. After login, verify the container daemon is running (container system status).
5. Attempt to reach the container via the gateway IP or published port.

Current behavior

After a system restart, the container reports as "running" via container list, but:

• The gateway IP (e.g. 192.168.100.1) is no longer reachable from inside the container. Requests from the container to host services through the gateway hang indefinitely.
• Published ports on the host still accept TCP connections (the NIO socket forwarder binds successfully), but requests hang because the forwarder cannot route traffic to the container's vmnet IP.
• container stop hangs indefinitely (related to [Bug]: Unable to stop a container when it's frozen. #576). The graceful shutdown path sends SIGTERM, waits, sends SIGKILL, then calls lc.stop(), but the communication with the VM appears to be broken, so none of these complete.
• container exec also hangs.
• The only recovery is to kill the container's underlying process via kill -9 <pid>, then container rm, then recreate.

This is consistently reproducible across system restarts. It does not require VPN changes (#1307), though that issue may share the same root cause (vmnet interface state not surviving system state changes).

Expected behavior

Containers should either remain network-reachable after a system restart, or the runtime should detect the broken state and either recover automatically or transition the container to a stopped/error state so that container stop and container rm work without hanging.

Environment

- OS: macOS 26.3.2 (Tahoe)
- Hardware: Apple Silicon (M-series)
- Container: container CLI version 0.10.0

Analysis

Looking at the source code, the vmnet network is created once during ReservedVmnetNetwork.start() via vmnet_network_create() and the resulting vmnet_network_ref is stored in a Mutex<State>. There is no mechanism to detect that the underlying vmnet interface has become invalid after a system restart, and no reconnection or re-creation logic.

The SandboxService.stop() path calls gracefulStopContainer() which relies on communicating with the VM via the container agent. When the VM's network is in a broken state, lc.wait() and lc.stop() appear to block indefinitely, making the container unrecoverable through normal commands.

A potential approach could be to detect stale container/network state on daemon startup after a system restart and either re-establish vmnet interfaces or clean up containers that are in an unrecoverable state.

Relevant log output

# Container appears running but is unreachable
$ container list
ID           IMAGE                              STATUS    NETWORKS
mynginx      docker.io/library/nginx:alpine     running   mynet

# Attempting to stop hangs (must be killed with timeout)
$ timeout 10 container stop mynginx
# (no output, killed by timeout)

# Gateway IP unreachable from container context
$ container exec mynginx -- wget -q -O - --timeout=5 http://192.168.100.1:9090/
# (hangs indefinitely)

Code of Conduct

• I agree to follow this project's Code of Conduct

[jglogan]

Looking at the source code, the vmnet network is created once during ReservedVmnetNetwork.start() via vmnet_network_create() and the resulting vmnet_network_ref is stored in a Mutex. There is no mechanism to detect that the underlying vmnet interface has become invalid after a system restart, and no reconnection or re-creation logic.

The vmnet_network_ref is ephemeral; its lifetime is that of the container-linux-runtime XPC helper that holds the reference to it.

It doesn't survive across a reboot (or a container system stop ; container system start), so the statement "There is no mechanism to detect that the underlying vmnet interface has become invalid after a system restart, and no reconnection or re-creation logic." is invalid.

Let's focus on your "current behavior" points which spells out what you're seeing really well.

Could you tell me how you know the first bullet "The gateway IP (e.g. 192.168.100.1) is no longer reachable from inside the container. Requests from the container to host services through the gateway hang indefinitely." is true? The other bullets make it look like you only have a stuck container (as you pointed out, #576) and no other networking issues.

With the system in the failed state, what happens if you run a request in a different container?

container run --rm curlimages/curl https://google.com

For your stuck container, can you provide the output of:

container logs --boot container-id > container-boot.log
container logs container-id > container.log

Another thing I'm not sure I understand is steps 3-5 in your procedure. What are you actually running after rebooting the system and logging in? I would expect:

container system start
container start nginx

and then: "Attempt to reach the container via the gateway IP or published port."

Are you doing this differently?

[jglogan]

Also this issue title seems to be incorrect - the description explicitly says "full restart, not sleep/wake". Consider updating the title to better reflect the description.

[adiled]

Thanks for the thorough questions, they led me to the actual root cause, which is not a networking bug.

What was actually happening:

My setup uses launchd plists (LaunchAgents) to manage container lifecycle. Each container has a plist with KeepAlive: true that runs container start --attach <name>. After a reboot, these plists fire at login, but the apiserver hasn't been started yet. All container start calls fail with XPC connection error: Connection invalid.

I initially misattributed this to broken vmnet networking because:

1. com.apple.containermanagerd shows as running after reboot (it's an on-demand system service), which I mistook for the container daemon being up
2. The apiserver plist at ~/Library/Application Support/com.apple.container/apiserver/apiserver.plist has RunAtLoad: true, but since it's not in ~/Library/LaunchAgents/, launchd doesn't auto-discover it after reboot. It only gets loaded when container system start explicitly registers it
3. In earlier releases I had a dedicated plist that ran container system start on login. When I upgraded to 0.10.0, the networking rework (multiple network plugins, new container-network-vmnet launchd services) and the presence of containermanagerd led my bot to believe the daemon was now managed natively, so I removed it

Once I run container system start and then start containers, everything works. Host-to-container, container-to-gateway, container-to-internet networking all function correctly. I was never actually in a state where containers were running with broken networking.

Your suggested test (container run --rm curlimages/curl https://google.com): this timed out on first attempt post-reboot, but that was due to image pull latency, not networking. Subsequent container operations all succeeded.

No stuck containers observed in this reproduction. After reboot + daemon restart, previously-running containers came back as stopped and started cleanly.

Apologies for the noise, this was a startup ordering issue on my end, not a container runtime bug. Closing this.