RFC: K3s-backed linked environments via external provisioner (no new scheduler)

[kvnloo]

Problem

T3 already lets one client control multiple machines, supports Tailscale/SSH/T3 Connect routes, optional new-thread load balancing, and (behind Linked environments beta) persistent peer links and delegate_task to another environment. I run self-hosted K3s workloads over Tailscale and want to use that existing T3 control surface with on-demand, isolated K3s-backed agent environments—without running a second agent scheduler or relying on a managed cloud service.

Today I can connect to a T3 server that already exists. The missing piece is: provision a worker environment, make its identity/reachability stable, obtain the user's permission to link it, then resume/stop/dispose it safely. A K3s Pod/Job is not itself a T3 environment, and the client-side environment picker does not provision or grant server-to-server access.

Proposed seam: an optional, external environment-provisioning broker

Start outside T3 core: a narrow K3s adapter/broker manages Kubernetes resources, then returns a normal T3 environment endpoint and identity. T3 continues to own threads, approvals, tools, remote delegation and durable orchestration.

T3 owner environment / existing agent + UI
       | approved provision request (stable task/lease ID)
       v
opt-in external K3s broker
       | create/reuse isolated pod + PVC/workspace + service
       v
headless T3 server in K3s, reachable through private Tailscale HTTPS
       | user-approved peer pairing (existing /mcp + PeerLinks)
       v
existing linked-environment tools / delegate_task
       | result / status / artifacts; verified by owner
       v
broker stop / resume / cleanup (explicit retention policy)

The concrete question: Is the right extension boundary a standalone broker and existing t3 environment link / PeerLinks + remote delegation, with no new core API at first? If a small native integration is eventually justified, would maintainers prefer (a) a provisioning hook that returns an endpoint and stable environment ID or (b) an adapter behind an existing environment-management service? I am not asking to implement a new UI, scheduler or Kubernetes controller in T3 core now.

Scope and guardrails

• No extra orchestration inside T3: K3s schedules pods; T3 owns task semantics. Do not migrate a live thread, distribute SQLite, share a writable checkout, or silently turn the client environment catalog into agent permission.
• Separate identity, route and authority: stable T3 environment ID across pod restarts; stable private URL via Tailscale Kubernetes Operator/Ingress or equivalent; pairing/linked access remains explicit, revocable and no broader than the originating agent's mode. Do not treat raw 100.x IP as inherently secure.
• Safe lifecycle: idempotent provision key; persisted repo/ref/image digest, namespace/pod/PVC identities and lease; bounded CPU/RAM/GPU and concurrency; owner/worker restart recovery; cancel/retry without duplicate accepted work; stop versus delete with user-controlled retention; scoped secrets and service-account RBAC.
• No background cost when disabled. A first version is local-only, one namespace, one worker, no public ingress and no cloud GPU.
• No premature auto-balancing: a pod is not automatically made eligible for new threads just because it is reachable; use manually selected approved links first.

Smallest acceptance proof

1. Launch one headless T3 server in a local K3s namespace with persistent identity/data and a private HTTPS route. After a pod restart, a connected T3 client and peer link see the same environment.
2. From a separate, persistent owner environment, explicitly approve/link that worker, launch a bounded task on a pinned repository revision, read status and result through existing linked-environment operations, and cancel it.
3. Simulate duplicate provision calls, a lost link/launch response, owner restart, pod death, link revocation and cleanup; confirm no duplicate accepted result, cross-task write, credential spill, or data deletion without policy.
4. Report exact commits, K3s/version, client type, command/test results, cold-start latency, observed limits and NOT RUN cases. No performance or isolation claims from manifests alone.

Relationship to existing work (not duplicates)

• Remote architecture, remote access, and apps/server/src/peer/{PeerLinks,RemoteDelegation}.ts: environment ID / routing / authorization / delegation already have their own owners.
• #9895 merged: opt-in balancing among connected machines, not machine provisioning.
• #16719 merged into a peer feature stack: delegate to a linked environment; not automatic creation of one. Check current main and beta state before implementation.
• #15350 open: Docker worktree sandboxes inside one owning host, with provider/terminal isolation and port relay. K3s external environments are different: no shared host paths, and no need to modify this active work.
• #14793 closed: proposed general external sandbox broker. This asks specifically whether existing, now-implemented peer-link / environment lifecycle is sufficient for an out-of-tree K3s proof; if it is the same request, please point to the preferred existing thread/seam.
• #14620 closed not planned: managed disposable cloud workers. This proposal expressly keeps infrastructure and cost control user-owned.
• #5284, #5287 closed unmerged: broad Docker packaging + provider-auth UI. Avoid bundling unrelated auth features.
• #13582 includes a pod-IP/ingress pairing report; #11849 covers container host-identity changes; #17051 covers remote-launch durability. These are concrete integration constraints, not claims they are solved.

I can keep the proof and implementation on my downstream fork issue until there is an agreed small upstream seam. No upstream PR or core change is requested by this issue. I understand feature proposals ordinarily belong in Ideas discussions; if that's the preferred venue, I can move the design discussion there instead of treating the issue as implementation approval.