KubernetesEnvironmentProvider: bring-your-own EKS + managed namespaces for K8s-native customer stacks

[khaliqgant]

Part of Epic: Factory Verification & Environments (#141), Track C. Implements the K8s provider behind the EnvironmentProvider port (#142). Depends conceptually on #142 (port), #143 (descriptor), #145 (gate).

Why

Our tier-5/6 substrate is Cloudflare-first, but that's a fidelity mismatch for a customer whose product runs on EKS/Kubernetes: their stack is Helm/kustomize, and it needs a real cluster (their k8s version, CNI, IRSA/IAM, ingress, node types) to be verified faithfully. Verifying such a feature on Cloudflare Workers would prove the wrong thing. The EnvironmentProvider port already anticipates this — this issue makes the K8s path real.

Two modes (per-customer config)

1. Bring-your-own-cluster (BYOC) — default, highest fidelity

Deploy the feature's stack into an ephemeral, isolated namespace (or vcluster) inside the customer's own EKS, via a customer-granted scoped kubeconfig / IRSA role. Verifies against their real environment; their infra, their cost. This is #129's "integrate an existing provider" applied to K8s.

2. Managed fallback

When a customer won't grant cluster access, provision ephemeral namespaces in a shared verification EKS / vcluster we run (in factory-test-infra), deploying their Helm chart to approximate their env — with an explicit, documented fidelity caveat (our addons/versions/IAM ≠ theirs).

Consider integrating an existing K8s preview-env tool (vcluster/Loft, Argo CD ApplicationSets, Uffizzi, Bunnyshell, Okteto) rather than hand-rolling namespace orchestration — evaluate 2–3 against the requirements below, consistent with #129's "integration, not a new sandbox runtime."

What

1. KubernetesEnvironmentProvider (src/environments/kubernetes-provider.ts) implementing [factory] EnvironmentProvider port + Cloudflare ephemeral-environment provider (Workers for Platforms + Containers); K8s/Daytona optional #142's EnvironmentProvider port: provision (create ephemeral namespace/vcluster + scoped RBAC + quota + NetworkPolicy), status, endpoints (ingress/port-forward URL), destroy, TTL reaper.
2. Connection registry — resolve which cluster for a given customer/repo: BYOC (kubeconfig/IRSA reference from the customer's env config, stored as a secret reference — never inline) or managed (the shared verification cluster). Lives in factory-test-infra alongside the Cloudflare config (#146).
3. Descriptor support (#143) — deployKind: kubernetes (Helm/kustomize/manifests) + target: byoc | managed. Already listed as an escape-hatch kind; this fills it in.
4. Safety guardrails (the crux — required for BYOC):
   • Namespace with a generated name + ownership/TTL labels; never deploy into an existing/prod namespace.
   • Least-privilege ServiceAccount/Role — namespace-scoped, never cluster-admin; cluster-scoped resources only if the chart requires AND the customer opts in.
   • ResourceQuota + LimitRange per environment.
   • NetworkPolicies denying egress to the customer's prod namespaces / metadata endpoints.
   • Prefer a dedicated node pool / taints (or hard resource caps) so verification load can't starve prod.
   • TTL reaper deletes the namespace (mirror src/orchestrator/reaper.ts); secrets pulled by reference from the customer's secret manager.

Acceptance

• KubernetesEnvironmentProvider implements the [factory] EnvironmentProvider port + Cloudflare ephemeral-environment provider (Workers for Platforms + Containers); K8s/Daytona optional #142 port; unit tests with a faked K8s client.
• BYOC: provisions a scoped, quota-capped, NetworkPolicy-isolated namespace in a target cluster from a kubeconfig/IRSA reference; deploys a Helm/kustomize stack; resolves endpoints; reaps.
• Managed: provisions in the shared verification cluster with the same guardrails; fidelity caveat documented.
• Descriptor deployKind: kubernetes + target honored by the deployer/gate with no gate changes (substrate-agnostic).
• Provider evaluation (BYOC integration tool vs hand-rolled) documented.

✅ End-to-end verification — Factory success criteria (REQUIRED)

"Compiles" is not done. Deploy a real Helm stack into a real cluster as a stand-in customer, prove the isolation guardrails bind, run load, and reap. CI uses kind/k3d as the "customer cluster"; a real EKS for the periodic full run.

Run it

npm run build && npm test                  # unit (faked client)

kind create cluster --name customer-eks-sim
KUBECONFIG=... npx tsx test/e2e/kubernetes-provider.e2e.ts   # BYOC path against the sim cluster
kind delete cluster --name customer-eks-sim

The E2E must (against the real sim cluster):

1. Provision + deploy: create an ephemeral namespace from a kubeconfig reference; deploy a fixture Helm chart (e.g. an app + Postgres); assert readiness + endpoints() returns a reachable URL (curl 200).
2. RBAC bind: the scoped SA can manage its namespace but is denied a cluster-admin action (assert the denial).
3. Isolation bind: a pod in the verification namespace cannot reach a service in a separate "prod" namespace (NetworkPolicy enforced — assert the connection fails).
4. Quota bind: a workload exceeding the ResourceQuota is rejected.
5. Load: run k6 (Load-simulation harness: in-cluster load generation with SLO gates and evidence #144) against the deployed endpoint; assert the SLO gate evaluates.
6. Reaper: the namespace is deleted on teardown AND an orphaned one (past TTL) is swept — assert both; prod namespace untouched.

Deliverable

• test/e2e/kubernetes-provider.e2e.ts running provision→deploy→reachable→(RBAC/isolation/quota asserted)→load→reap against a kind cluster, one command in CI.
• Each guardrail assertion fails loudly if the guardrail is removed (no false green); the "prod" namespace is provably never touched.

Anchor files

• src/ports/environment.ts (from [factory] EnvironmentProvider port + Cloudflare ephemeral-environment provider (Workers for Platforms + Containers); K8s/Daytona optional #142) → new src/environments/kubernetes-provider.ts
• src/environments/stack-descriptor.ts / stack-deployer.ts (Verification-stack descriptor contract + deployer: repos declare their full stack, Factory brings it up #143 — deployKind: kubernetes + target)
• src/orchestrator/reaper.ts (identity-checked reaper), src/config/schema.ts, src/index.ts
• factory-test-infra ([factory] Stand up factory-test-infra: Cloudflare-based ephemeral verification environments (Workers for Platforms + Containers) + factory provider wiring #146): customer→cluster connection registry, shared managed verification EKS/vcluster, RBAC/Quota/NetworkPolicy base charts