🚀 Announcing Agent Sandbox v0.5.4!
We're excited to announce the release of Agent Sandbox v0.5.4! This release brings significant improvements in core stability, performance, and introduces powerful new features for the Python SDK and router. Key areas of focus include hardening sandbox lifecycle management, optimizing API server interactions, and expanding integration capabilities.
⚠️ Breaking Changes / Action Required
- Sandbox
Suspendedcondition changed to comply with Kubernetes convention (#1150):- The
Suspendedstatus condition is now always present onSandboxresources after initial reconciliation. Clients should inspect its booleanstatusvalue (e.g.,status: "False") rather than checking for its presence or absence. - The legacy
PodNotTerminatedreason string is deprecated in favor ofPodTerminating. Update any client-side logic that explicitly checks forPodNotTerminated.
- The
Key Highlights
Core Stability & Lifecycle Management
- Persistent Suspended Conditions for Sandbox (#1150): The Sandbox
Suspendedstatus condition is now consistently present and transitions its status (TrueorFalse) instead of appearing or disappearing, enabling standardkubectl waitcommands and preserving history. - Robust Adoption Assignment & Optimistic Locks (#1277): Prevents assignment flips, orphaned sandboxes, and duplicate adoptions under load by applying optimistic locks on adoption patches and resolving conflicts against authoritative reads. Benign adoption contention now surfaces as an
AdoptionConflictReady reason. - Optimistic-Locked Status Writes (#1256): Eliminates duplicate startup-latency histogram observations and stale status overwrites by using optimistic-locked status patches, ensuring metrics are recorded exactly once and status reflects the authoritative state.
- Reduced Redundant Reconciles & Status Patching (#1254): Improves controller efficiency by using non-optimistic merge patches for Sandbox status writes (eliminating 409 conflicts) and adding predicates to only reconcile owning
SandboxClaimon relevantSandboxfield changes. - Warm Pool Over-Creation Mitigations (#1266): Prevents
SandboxWarmPoolfrom over-creating replicas due to informer-cache lag. Sandbox creations are now gated by a ReplicaSet-style expectations tracker, and terminating sandboxes count against the target. IntroducesWarmPoolNotProgressingevents for capacity shortfalls. - Efficient Pod Cache Transformation (#1272): Strips
Podfinalizers from the informer cache, reducing memory usage and event decoding costs as they are not used by Agent Sandbox controllers. - Semantic Comparison for K8s API Objects (#1278): Replaces
reflect.DeepEqualwithapiequality.Semantic.DeepEqualfor Kubernetes API object comparisons, preventing unnecessary status updates and reconciliation due to non-semantic differences. - Stricter API Group Verification (#945): Enhances warm-pool and sandbox ownership/identity validation by consistently verifying API Group in addition to Kind for controller and owner references.
- Improved RBAC for Events (#1080): Grants core API group (
"") events permission for the leader election event recorder, fixingevents is forbiddenerrors. - Exposed Sandbox Service Ports (#1258): Generated headless
SandboxServices now automatically include ports derived from declared container ports, improving compatibility with service meshes like Istio. - Accurate Startup Latency Metrics (#1087): Fixes
SandboxClaimcontroller startup latency histograms to record exactly once per claim lifetime, preventing overcounts due to readiness probe flapping. - Go Client Handle Registry Race Fix (#998): Prevents leaking orphaned sandbox handles in the Go client by making
GetSandbox/CreateSandboxrace-safe and ensuring proper disconnection of redundant handles. - Enhanced Adoption Conflict Resolution (#1304): Improves adoption handling for optimistic-lock contention and refines user-facing conflict messaging to avoid exposing low-level internal errors.
Performance & Scalability
- API Connection Sharding (#1240): Introduces opt-in controller flags (
--separate-watch-connectionand--api-connections) to use dedicated HTTP/2 connections for informer watches and shard non-watch traffic, significantly improving API concurrency and reducing watch starvation. - Optimized Metadata Writes (#1250): Implements direct merge patches for hot-path metadata writes on
SandboxClaimcontrollers, reducing CPU and memory allocations by building targeted patch payloads instead of full-object diffs. - Stress Test Configuration Improvements (#1283): Increases Kubelet API QPS limits and caps Kubelet event spam during stress tests, allowing for higher churn rates and reducing API server load.
- KWOK Scalability Presubmit Test (#1269): Adds a fast, lightweight presubmit test using KWOK (Kubernetes WithOut Kubelet) to automatically benchmark performance and catch scalability regressions on every Pull Request.
- Optimized Pod Counting for Circuit Breaker (#1232): The circuit breaker now efficiently counts pods using
limit=1andremainingItemCountto reduce API server and memory load at large scales. - Infrastructure Tuning for Stress Tests (#1275): Configures benchmark worker node root volumes on
pd-ssdand control plane nodes onc3-standard-22, addressing disk I/O and control plane CPU bottlenecks for improved stress test performance. - Beta API Serving Optimization (#1234): Stress tests now serve only the beta Sandbox APIs to reduce conversion traffic and improve API server efficiency.
Router Enhancements
- Scoped-Token Authorizer (#1243): Introduces a new
--authz-mode=scoped-tokenforsandbox-router, allowing local verification of signed tokens bound to a single (namespace, name) without requiring a kube-apiserver round-trip. - Improved Warm-Pool Routing (#1239):
sandbox-routernow routes requests carryingX-Sandbox-Idvia the Pod-IP cache's namespace/name index, fixing 502 (NXDOMAIN) errors for warm-pool sandboxes without a dedicated Service.
SDK & Integrations
- Agent Sandbox Recycling (Python SDK) (#1232): Introduces sandbox recycling (
reuse_git_restore_sandbox) in theagent-sandbox-rlSDK, allowing reuse of claimed sandboxes across tasks (e.g., RL rollouts) to reduce claim latency and API load. Includes contamination guards, persistent exec sessions, and warm-pool over-creation mitigations. - Optimized Python SDK
wait_for_claim_ready(#1241): The Python SDK'swait_for_claim_ready()now uses a single, watch-based approach on the claim itself, significantly reducing latency compared to the previous two-sequential-watch method. Also, faster dev port-forward polling. - AsyncSandbox Functionality Alignment (#999): Aligns
AsyncSandboxfunctionality withSandboxin the Python SDK, ensuring consistent behavior and expanding API capabilities. - Langchain-Deepagents Integration (#1144): Adds a new Python package for integrating with Langchain-Deepagents, enabling sandbox lifecycle management, command execution, and file transfer.
- MCP Server Introduction (#1141): Adds an Agent Sandbox MCP (Multi-Cluster Proxy) server with basic implementation for listing sandboxes and tools for create/delete, command execution, and file operations.
Documentation & Examples
- Expanded Security Threat Model (#1299): The security threat model documentation has been expanded to include architectural overview, trust boundaries, threat analysis with mitigations, and
SandboxTemplateenforcement capabilities. - Clarified Repository Scope (#1298): The
README.mdnow explicitly clarifies that Agent Sandbox is a sandbox orchestrator, delegating low-level container isolation to secure runtimes like gVisor or Kata Containers. - Hermes Agents-as-a-Service Example (#1271): Adds a comprehensive example demonstrating the multi-user "agents as a service" platform pattern, featuring warm-pool claims, suspend/resume for cost management, PVC state survival, and a minimal gateway.
- Firecracker Sandbox Example (#1238): Introduces an example demonstrating how to run Agent Sandboxes on Kata Containers with the Firecracker VMM, providing microVM isolation and fast boot times.
- APF Insulation Overlay Example (#1270): Adds an opt-in API Priority and Fairness (APF) insulation overlay and operator guide to prioritize critical controller API traffic and isolate bulk workloads.
- Runtime-Class-Aware Benchmarks (#1262): Adds e2e tests and benchmarks for warm pool subsystem across runc, gVisor, and Kata runtimes, measuring cold start latency, warm pool claim speed, and burst recovery.
- Enhanced Stress Testing Phases (#1287): Refactors tests to include a phase for testing with large numbers of pods (e.g., up to 80% capacity) and adds new capacity-related test assertions.
- Improved Stress Test Reports (#1284): Enhances stress test reports with a "Limiter Regime by Component" table to identify client-go rate limiter behavior (queueing vs. pacing) and estimate implied QPS limits.
Installation
Standard Install (Core + Extensions)
Recommended for most users and GitOps engines (Argo CD, Config Sync, kustomize):
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/v0.5.4/sandbox-with-extensions.yamlSelective Install
Install components separately:
# Core only:
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/v0.5.4/sandbox.yaml
# Extensions (opt-in):
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/v0.5.4/extensions.yamlPython SDK
pip install k8s-agent-sandbox==0.5.4Contributors
We extend our sincere thanks to all contributors to this release:
@ArthurKamalov, @ChristianZaccaria, @Geun-Oh, @YahiaBadr, @aditya-shantanu, @aegeiger, @dongjiang1989, @gruebel, @hsinhoyeh, @janetkuo, @justinsb, @matiasinsaurralde, @ronaknnathani, @shrutiyam-glitch, @tomergee, @vvoronko
New Contributors
- @matiasinsaurralde made their first contribution in #998
- @ronaknnathani made their first contribution in #1258
- @YahiaBadr made their first contribution in #1269
- @aegeiger made their first contribution in #1087
- @gruebel made their first contribution in #999
- @vvoronko made their first contribution in #1262
- @ChristianZaccaria made their first contribution in #1080
- @Geun-Oh made their first contribution in #1239
Full Changelog: v0.5.3...v0.5.4