Local Talos on macOS via Apple container (not Docker, not QEMU): working provider + findings

[BinHsu]

TL;DR — I built a working pkg/provision provisioner for Apple's container runtime, so
talosctl cluster create apple-container brings up a local Talos cluster as one micro-VM per
node on macOS — no Docker socket, no QEMU. It's verified end to end (incl. MetalLB + Gateway
API). Before I polish a PR I'd like to know: is this something you'd want in-tree?

Repo (provider + a from-zero runbook + verification log):
https://github.com/BinHsu/aegis-talos-apple-container-provisioner

Why

Local Talos on macOS today is either the docker provisioner (needs a Linux Docker socket —
Docker Desktop / OrbStack / Colima) or the qemu provisioner (shipped, but with open macOS
issues, e.g. #11865, #13579, and the recurring "not supported on this platform" reports like
discussion #11438). Apple's container (1.0.0, macOS 26, Apple Silicon) gives a Kata-derived
micro-VM per node — a higher-fidelity on-prem analog than shared-kernel containers, with no
Docker API in the path.

What works (verified by hand, receipts in the repo)

• talosctl cluster create apple-container → 2-node cluster, all nodes Ready, clean
  cluster destroy (no orphans). Also a 3-node etcd quorum.
• The provider compiles against the real pkg/provision interface (it's an additive package +
  a factory case + a cluster create subcommand — no framework changes).
• Networking: MetalLB L2 LoadBalancer is host-reachable — the gratuitous ARP that the
  qemu-vmnet path drops ([macOS] Support socket_vmnet / vmnet-bridged networking for QEMU provisioner #12834) is forwarded here. L7 verified with both ingress-nginx v1.15.1
  and Gateway API (Envoy Gateway), host-routed.

The one interesting design point

apple/container assigns node IPs via vmnet DHCP (no static --ip). The docker provider's
model (static IP + machine config injected as USERDATA at create) can't be honored as-is. So
the provider owns Create and does a post-launch reconciliation: launch nodes bare into
maintenance, discover the DHCP IPs, patch cluster.controlPlane.endpoint, then apply config over
the maintenance API. This keeps the whole DHCP workaround inside the provider — pkg/provision
is untouched. (It's also why a Docker-API shim / socktainer route doesn't work: there's no seam
to reconcile the IP.)

Honest scope / limitations

• This is a weekend spike by one person, not production-hardened.
• Nodes are ephemeral: a host/daemon restart loses the cluster (tmpfs + DHCP → recreate). Talos
  blocks in-place reboot in container mode anyway. A static-IP option in apple/container would
  remove this.
• macOS 26 + Apple Silicon + container 1.0.0 only.

Ask

Would you accept this as a third macOS provisioner? I noticed a "remote provisioner" is being
added (#13576), so new provisioners seem welcome — if there's interest I'll open a PR (the delta
is ready: pkg/provision/providers/apple/, the factory case, and a cluster create apple-container
subcommand) and align it to your conventions.

[smira]

The issues cited are old/outdated and not relevant mostly. Talos/QEMU works on MacOS without any major issues.

I would not rather have a provisioner which is a "snowflake" for a specific platform (MacOS) and has no real benefit over QEMU.
Plus we don't have tests for MacOS.

QEMU provider might be improved for better networking when needed, and this is the only provider which does full Talos experience, any container mode is more of a workaround.

[BinHsu]

Thanks Andrey — that's fair, and you're the right person to make the call.

Conceding outright: no macOS CI is a real blocker, a platform-specific provisioner is a maintenance snowflake, and you're right that container mode is a workaround — apple/container runs Talos in
container mode, so it doesn't give the full disk-based boot/upgrade/reboot experience QEMU does. I also shouldn't have leaned on the older networking issues; if QEMU networking on macOS is solid now,
that removes most of my "why not QEMU" argument.

The one thing I'd reframe: I see this less as an alternative to QEMU and more as an alternative to the docker provisioner — a no-Docker, per-node-kernel option for fast ephemeral dev on Apple Silicon.
But I hear that that's not enough benefit to justify upstream maintenance.

I'll keep it as a standalone provider for anyone who wants it, and leave QEMU as the supported path. If there's appetite for QEMU networking improvements on macOS, I'd be glad to help there instead —
sounds like that's where the real demand is.

[smira]

I would be interested to hear if we could support "external" provisioners better - whether we need to expose some of the Talos code to make it easer to reuse common components?

[BinHsu]

Yes, happy to dig into this, and I think it's a more valuable direction than any single provider.

From building the apple/container provider fully out-of-tree, the split was clean:

Already reusable, zero friction: pkg/machinery (config generate / bundle / configpatcher) and pkg/provision (the Provisioner / Cluster interfaces and request types) imported and compiled against the real module first try. That layer is a solid public surface as-is.

What forced a fork was the cluster-create orchestration, all under cmd/talosctl/cmd/mgmt/cluster/create/:

• The clusterops glue that turns a Provisioner into a working talosctl cluster create — config-bundle assembly, the create → bootstrap → health sequence, options wiring — lives in the cmd tree, not a
  reusable pkg/.
• The config makers under clusterops/configmaker/internal/makers sit behind internal/, so out-of-tree code can't import them at all — adding a platform's maker means forking, even though the maker is the
  most provider-specific, swappable piece.
• --provisioner is a hardcoded switch in the cmd package — no seam for an external provisioner to register and be selected.

Concrete proposal — promote that orchestration into a reusable package (something like pkg/cluster/create) so the CLI becomes a thin wrapper over it:

1. A ClusterMaker interface lifted out of internal/makers, so a provider supplies its own config maker without forking.
2. A lifecycle entrypoint that, given a provision.Provisioner + a maker + options, runs gen-config → Create → bootstrap → health-wait (and the destroy path) — the logic that's in clusterops today.
3. A small provisioner registry (Register(name, factory)) so --provisioner resolves in-tree and out-of-tree providers.
4. Normal Go-module versioning on that surface, so external consumers can pin and upgrade deliberately.

The payoff: external providers become third-party integrations maintained by their authors, not the Talos team — which directly addresses the macOS-CI and "snowflake" concerns from the original thread. The
in-tree docker/qemu providers become just two consumers of the same library. Anyone could then ship a Proxmox / Incus / Firecracker / cloud provider out-of-tree against a supported surface.

Two scopes, if it helps frame it:

• Library mode (smaller): external authors import the lifecycle lib and ship their own binary. My cmd/aegis driver is basically this already.
• Plugin mode (fuller): talosctl discovers and dispatches to external provisioners, so --provisioner works in the canonical CLI.

One real design wrinkle I hit that's worth baking in: the current create flow assumes static-IP-at-create. apple/container only has DHCP, so I had to launch bare, read the address, then patch + apply — a reusable lifecycle that treats "discover address after boot" as a first-class path would help more providers than just mine.

Happy to prototype the library-mode extraction with apple/container as the guinea-pig external consumer, or open a focused issue to scope it — whichever's more useful.