Gate cni-sidecar sandbox scheduling on verified per-node CNI readiness

[russellb]

Problem Statement

The cni-sidecar supervisor topology moves pod-network egress enforcement from an in-pod openshell-network-init container (which required NET_ADMIN) to a node-level privileged CNI DaemonSet. The DaemonSet's openshell-cni plugin programs the bypass-prevention firewall during CNI ADD, so no container in the sandbox pod needs NET_ADMIN.

Because the in-pod fail-closed backstop is deliberately removed, enforcement now depends on the node's CNI DaemonSet pod being Ready before any sandbox pod is scheduled there. A cold-start scheduling race remains: a sandbox pod scheduled on a node before that node's CNI DaemonSet pod has completed its first conflist patch is admitted with unrestricted egress. This affects newly joined nodes, nodes reimaged/rebooted, and the initial rollout window.

The reconcile loop added alongside this issue re-verifies and re-patches the chained plugin each tick, which closes the "CNI config rewrite" and "DaemonSet restart" windows (bounded to one interval). It does not close the cold-start scheduling race, because that requires coordinating pod scheduling with per-node CNI readiness — outside what an in-pod reconcile loop can do.

Proposed Design

Gate sandbox scheduling on verified per-node CNI readiness using node labels:

1. CNI installer marks node ready. After the first successful conflist/chain patch, the installer labels its own node openshell.ai/cni-ready=true. Add a small openshell-cni subcommand (e.g. openshell-cni mark-node-ready) that reads NODE_NAME from the downward API and PATCHes the node label via the Kubernetes API (the plugin already speaks to the API for pod annotations via reqwest, so this reuses the existing client path). On preStop, remove the label so a node losing the DaemonSet stops receiving new sandbox pods.

2. RBAC. Extend the dedicated CNI ServiceAccount with a cluster-scoped ClusterRole granting nodes: [get, patch] (label mutation only). This is an escalation of a privileged, hostNetwork SA to cluster-wide node mutation, so it must be gated behind the same cni.enabled/topology conditions and documented as such.

3. Driver scheduling constraint. In cni-sidecar topology only, the Kubernetes driver adds a nodeAffinity requiring openshell.ai/cni-ready=true to every sandbox pod. Pods then only land on nodes with confirmed enforcement; if no node is ready, the pod stays Pending (fail-closed) rather than running unprotected.

User-facing behavior: no new configuration in the common case — the affinity and label are managed automatically when the topology is enabled. Operators see sandbox pods stay Pending until at least one node's CNI is Ready, instead of running without egress enforcement.

Alternatives Considered

• In-pod fail-closed backstop (re-add network-init). Reintroduces NET_ADMIN into the sandbox pod — the exact privilege the cni-sidecar topology exists to remove — so it undercuts the topology's purpose. Rejected.
• Document-only (current state). The cold-start window is documented in architecture/compute-runtimes.md with an operational mitigation (ensure the CNI DaemonSet is Ready on all schedulable nodes before enabling the topology in production). Adequate as an interim measure but leaves a real fail-open in dynamic-node environments (autoscaling, reboots). This issue tracks the durable fix.
• Scheduler plugin / admission webhook keyed on node CNI readiness. More general but far heavier to build and operate than node-label affinity; not justified for this single readiness signal.

Agent Investigation

• The cni-sidecar topology and its removal of the in-pod backstop live in crates/openshell-driver-kubernetes/src/driver.rs (apply_supervisor_sidecar_topology(..., install_network_init=false) for SupervisorTopology::CniSidecar).
• The node CNI installer is deploy/helm/openshell/templates/cni-daemonset.yaml; the reconcile-tick re-verify (ensure_conflist / write_chain_conf) closes the config-rewrite and restart windows but cannot gate scheduling.
• The CNI ServiceAccount currently has only a namespaced Role with pods: [get] (deploy/helm/openshell/templates/cni-rbac.yaml); node labeling would require a new cluster-scoped ClusterRole.
• The openshell-cni binary already has a Kubernetes API client (reqwest, kubeconfig-driven) in crates/openshell-cni/src/lib.rs, so a node-label subcommand can reuse it.

Checklist

• I've reviewed existing issues and the architecture docs
• This is a design proposal, not a "please build this" request

[russellb]

This was my agent filing something based on a branch i'm working on. it doesn't need to be an open issue here.