[Bug]: `container system start` hangs indefinitely at `Verifying apiserver is running...`

[n8henrie]

I have done the following

• I have searched the existing issues
• If possible, I've reproduced the issue using the 'main' branch of this project

Steps to reproduce

I am building from current main and now can't even get container to start. Have rebooted multiple times and cleaned the build directory, the issue persists.

$ git rev-parse HEAD
9fd15f01bf0f7caf757b66e17ac3f8e1be432bde
$ sudo BUILD_CONFIGURATION=release make all install
Building container binaries...
swift-driver version: 1.127.15 Apple Swift version 6.2.4 (swiftlang-6.2.4.1.4 clang-1700.6.4.2)
Target: arm64-apple-macosx26.0
[1/1] Planning build
Building for production...
[10/10] Compiling SwiftProtobuf AnyMessageStorage.swift
Build complete! (23.59s)
make[1]: Entering directory '/Users/n8henrie/git/container'
Installing container binaries from /Users/n8henrie/git/container/.build/arm64-apple-macosx/release into bin/release/staging/...
Install update script
Install uninstaller script
Signing container binaries...
bin/release/staging/bin/container: replacing existing signature
bin/release/staging/bin/container-apiserver: replacing existing signature
bin/release/staging/libexec/container/plugins/container-core-images/bin/container-core-images: replacing existing signature
bin/release/staging/libexec/container/plugins/container-runtime-linux/bin/container-runtime-linux: replacing existing signature
bin/release/staging/libexec/container/plugins/container-network-vmnet/bin/container-network-vmnet: replacing existing signature
Creating application installer
pkgbuild: Inferring bundle components from contents of bin/release/staging/
pkgbuild: Wrote package to bin/release/container-installer-unsigned.pkg
Installing container installer package
make[1]: Leaving directory '/Users/n8henrie/git/container'
Building initfs if containerization is in edit mode
Installing container binaries from /Users/n8henrie/git/container/.build/arm64-apple-macosx/release into bin/release/staging/...
Install update script
Install uninstaller script
Signing container binaries...
bin/release/staging/bin/container: replacing existing signature
bin/release/staging/bin/container-apiserver: replacing existing signature
bin/release/staging/libexec/container/plugins/container-core-images/bin/container-core-images: replacing existing signature
bin/release/staging/libexec/container/plugins/container-runtime-linux/bin/container-runtime-linux: replacing existing signature
bin/release/staging/libexec/container/plugins/container-network-vmnet/bin/container-network-vmnet: replacing existing signature
Creating application installer
pkgbuild: Inferring bundle components from contents of bin/release/staging/
pkgbuild: Wrote package to bin/release/container-installer-unsigned.pkg
Installing container installer package
installer: Package name is container-installer-unsigned
installer: Installing at base path /
installer: The install was successful.
$ /usr/local/bin/container system start
Registering API server with launchd...
Verifying apiserver is running...

Current behavior

container hangs indefinitely

Expected behavior

system should start

Environment

$ sw_vers
ProductName:            macOS
ProductVersion:         26.3.2
BuildVersion:           25D2140
$ xcodebuild -version
Xcode 26.3
Build version 17C529

Relevant log output

N/A

Code of Conduct

• I agree to follow this project's Code of Conduct

[jglogan]

Hi @n8henrie What does the sw_vers command output on your system?

Could you stop your container system start, and then run container system stop?

Then in another terminal run a command to stream the relevant OS log messages:

log stream --debug --info --predicate 'subsystem = "com.apple.container" or subsystem = "com.apple.NetworkSharing"'

Then, try container system start and let it hang.

Finally, copy the OS log messages you captured and paste them into a code block (or upload a log.txt file) here.

[manojmahapatra]

Looks like a dupe of #1319, likely similar issues.

[n8henrie]

@jglogan

Hi @n8henrie What does the sw_vers command output on your system?

Thanks for your time -- this information is in my initial post above, formatted in a codeblock. Please search for sw_vers.

container system stop also hangs indefinitely (for at least the last few minutes):

$ container system stop
2026-03-19T16:31:41-0600 info com.apple.container.cli: [ContainerCommands] checking if APIServer is alive

The logs repeat with cannot find any plugins with type network:

$ log stream --debug --info --predicate 'subsystem = "com.apple.container" or subsystem = "com.apple.NetworkSharing"'
Filtering the log data using "subsystem ==[cd] "com.apple.container" OR subsystem ==[cd] "com.apple.NetworkSharing""
Timestamp                       Thread     Type        Activity             PID    TTL
2026-03-19 16:34:06.096568-0600 0x2692bc   Info        0x0                  21761  0    container-apiserver: [com.apple.container:APIServer] starting helper [name=container-apiserver]
2026-03-19 16:34:06.096921-0600 0x2692bc   Info        0x0                  21761  0    container-apiserver: [com.apple.container:APIServer] configuring XPC server
2026-03-19 16:34:06.096938-0600 0x2692bc   Info        0x0                  21761  0    container-apiserver: [com.apple.container:APIServer] initializing plugin loader [installRoot=/Users/n8henrie/git/container/.build/]
2026-03-19 16:34:06.097014-0600 0x2692bc   Info        0x0                  21761  0    container-apiserver: [com.apple.container:APIServer] detecting user plugins directory [path=/Users/n8henrie/git/container/.build/libexec/container-plugins]
2026-03-19 16:34:06.097896-0600 0x2692bc   Info        0x0                  21761  0    container-apiserver: [com.apple.container:APIServer] discovered plugin directory [path=/Users/n8henrie/git/container/.build/arm64-apple-macosx/release/plugins]
2026-03-19 16:34:06.097918-0600 0x2692bc   Info        0x0                  21761  0    container-apiserver: [com.apple.container:APIServer] discovered plugin directory [path=/Users/n8henrie/git/container/.build/libexec/container/plugins]
2026-03-19 16:34:06.098007-0600 0x2692bc   Info        0x0                  21761  0    container-apiserver: [com.apple.container:APIServer] initializing plugins
2026-03-19 16:34:06.098124-0600 0x2692bc   Info        0x0                  21761  0    container-apiserver: [com.apple.container:APIServer] initializing containers service
2026-03-19 16:34:06.098986-0600 0x2692bc   Info        0x0                  21761  0    container-apiserver: [com.apple.container:APIServer] initializing networks service
2026-03-19 16:34:06.099425-0600 0x2692bc   Error       0x0                  21761  0    container-apiserver: [com.apple.container:APIServer] helper failed [name=container-apiserver] [error=internalError: "cannot find any plugins with type network"]
2026-03-19 16:34:16.152196-0600 0x269377   Info        0x0                  21772  0    container-apiserver: [com.apple.container:APIServer] starting helper [name=container-apiserver]
2026-03-19 16:34:16.152667-0600 0x269377   Info        0x0                  21772  0    container-apiserver: [com.apple.container:APIServer] configuring XPC server
2026-03-19 16:34:16.152684-0600 0x269377   Info        0x0                  21772  0    container-apiserver: [com.apple.container:APIServer] initializing plugin loader [installRoot=/Users/n8henrie/git/container/.build/]
2026-03-19 16:34:16.152782-0600 0x269377   Info        0x0                  21772  0    container-apiserver: [com.apple.container:APIServer] detecting user plugins directory [path=/Users/n8henrie/git/container/.build/libexec/container-plugins]
2026-03-19 16:34:16.153932-0600 0x269377   Info        0x0                  21772  0    container-apiserver: [com.apple.container:APIServer] discovered plugin directory [path=/Users/n8henrie/git/container/.build/arm64-apple-macosx/release/plugins]
2026-03-19 16:34:16.153957-0600 0x269377   Info        0x0                  21772  0    container-apiserver: [com.apple.container:APIServer] discovered plugin directory [path=/Users/n8henrie/git/container/.build/libexec/container/plugins]
2026-03-19 16:34:16.154072-0600 0x269377   Info        0x0                  21772  0    container-apiserver: [com.apple.container:APIServer] initializing plugins
2026-03-19 16:34:16.154203-0600 0x269377   Info        0x0                  21772  0    container-apiserver: [com.apple.container:APIServer] initializing containers service
2026-03-19 16:34:16.155285-0600 0x269377   Info        0x0                  21772  0    container-apiserver: [com.apple.container:APIServer] initializing networks service
2026-03-19 16:34:16.155855-0600 0x269377   Error       0x0                  21772  0    container-apiserver: [com.apple.container:APIServer] helper failed [error=internalError: "cannot find any plugins with type network"] [name=container-apiserver]
2026-03-19 16:34:26.203420-0600 0x2693f3   Info        0x0                  21775  0    container-apiserver: [com.apple.container:APIServer] starting helper [name=container-apiserver]
2026-03-19 16:34:26.203722-0600 0x2693f3   Info        0x0                  21775  0    container-apiserver: [com.apple.container:APIServer] configuring XPC server
2026-03-19 16:34:26.203732-0600 0x2693f3   Info        0x0                  21775  0    container-apiserver: [com.apple.container:APIServer] initializing plugin loader [installRoot=/Users/n8henrie/git/container/.build/]
2026-03-19 16:34:26.203778-0600 0x2693f3   Info        0x0                  21775  0    container-apiserver: [com.apple.container:APIServer] detecting user plugins directory [path=/Users/n8henrie/git/container/.build/libexec/container-plugins]
2026-03-19 16:34:26.204643-0600 0x2693f3   Info        0x0                  21775  0    container-apiserver: [com.apple.container:APIServer] discovered plugin directory [path=/Users/n8henrie/git/container/.build/arm64-apple-macosx/release/plugins]
2026-03-19 16:34:26.204666-0600 0x2693f3   Info        0x0                  21775  0    container-apiserver: [com.apple.container:APIServer] discovered plugin directory [path=/Users/n8henrie/git/container/.build/libexec/container/plugins]
2026-03-19 16:34:26.204769-0600 0x2693f3   Info        0x0                  21775  0    container-apiserver: [com.apple.container:APIServer] initializing plugins
2026-03-19 16:34:26.204894-0600 0x2693f3   Info        0x0                  21775  0    container-apiserver: [com.apple.container:APIServer] initializing containers service
2026-03-19 16:34:26.205778-0600 0x2693f3   Info        0x0                  21775  0    container-apiserver: [com.apple.container:APIServer] initializing networks service
2026-03-19 16:34:26.206164-0600 0x2693f3   Error       0x0                  21775  0    container-apiserver: [com.apple.container:APIServer] helper failed [name=container-apiserver] [error=internalError: "cannot find any plugins with type network"]

@manojmahapatra perhaps a duplicate, but I'm not yet sure -- for example I'm not seeing references to vmnet as noted in that issue.

[manojmahapatra]

@n8henrie Thanks for providing the logs. This does not look like the same issue as #1319. can you please try a few things,

1/ force boot-out any stale services:

launchctl bootout gui/$(id -u)/com.apple.container.apiserver || true
launchctl bootout user/$(id -u)/com.apple.container.apiserver || true

2/ start with explicit install root:

/usr/local/bin/container system start --install-root /usr/local

3/ share the outputs of:

launchctl print gui/$(id -u)/com.apple.container.apiserver
launchctl print user/$(id -u)/com.apple.container.apiserver

and confirm the CONTAINER_INSTALL_ROOT value.

[katiewasnothere]

@n8henrie Could you try running launchctl bootout com.apple.container.apiserver?

Then do:

1. make clean cleancontent
2. sudo BUILD_CONFIGURATION=release make all install

In the second set of logs you provided, it's interesting that the apiserver is looking for plugins in /Users/n8henrie/git/container/.build/libexec/container/plugins. Did you previously try to run container from the .build directory directly? If so, that would not work since the binaries in that directory have not been signed (see makefile)

edit: oops @manojmahapatra beat me to posting this :D

[n8henrie]

Really appreciate the help here.

@katiewasnothere -- yes I had previously tried to just swift build -c release and run from .build (generally try not to make install or elevate privileges until I'm done exploring a new project and decided whether or not to keep it around). But I had subsequently cleaned (but not cleancontented) and assumed all traces of the prior build would have been removed. ¯\_(ツ)_/¯

Had already bootouted everything but will do again.

...

Well after all that things seems to be working now, many thanks for such prompt help!

$ /usr/local/bin/container system start
Registering API server with launchd...
Verifying apiserver is running...
$ /usr/local/bin/container system status
FIELD              VALUE
status             running
appRoot            /Users/n8henrie/Library/Application Support/com.apple.container/
installRoot        /usr/local/
logRoot
apiserver.version  container-apiserver version 0.10.0-27-g9fd15f0 (build: release, commit: 9fd15f0)
apiserver.commit   9fd15f01bf0f7caf757b66e17ac3f8e1be432bde
apiserver.build    release
apiserver.appName  container-apiserver

For future readers, the steps I took to get things working again:

• container system stop
• launchctl bootout gui/$UID/com.apple.container.apiserver
• /usr/local/bin/uninstall-container.sh -k
• make clean cleancontent
• sudo BUILD_CONFIGURATION=release make all install