Repository navigation
Now witness the power of this fully armed and operational ... system for managing agents and agent-like workloads securely at scale.
Thank you to our MANY contributors who helped get us here! We couldn't have done it without you. ❤️
What's Changed
⚠️ Breaking Changes
- Envoy moves to v1.39.5, micro-VM assets to kata 4.3.1, kind and CSI images pinned by digest by Davanum Srinivas (@dims) in #3181
- WorkerPools with similar long names no longer share a NetworkPolicy by Yufan Su (@yufan-su) in #3130
- MicroVM guests get their whole memory limit, and placement books each actor's runtime overhead by Benjamin Elder (@BenTheElder) in #3104
- Rename --cordon-control-plane to --cordon-system-infra-components and its node pool label to ate-system-infra by haiyanmeng in #3210
- A microVM actor using more host memory than it booked no longer takes down its worker by Benjamin Elder (@BenTheElder) in #3429
- Remove the unused public MintActorCertificate API by Eitan Yarmush (@EItanya) in #3427
- [release-1.0] Cherry-pick #2392: Run containers as the image's USER and WORKDIR, with layer file owners kept by Benjamin Elder (@BenTheElder) in #3473
- [release-1.0] Cherry-pick #3487: Unknown Linux capability names in an ActorTemplate are rejected at admission by Benjamin Elder (@BenTheElder) in #3514
- [release-1.0] Cherry-pick 23 main changes through 4488a66 by Eitan Yarmush (@EItanya) in #3536
- [release-1.0] Cherry-pick 19 main changes through 5a5b192 by Eitan Yarmush (@EItanya) in #3571
- [release-1.0] Cherry-pick 21 main changes through fc931c4 by Eitan Yarmush (@EItanya) in #3596
- [release-1.0] Cherry-pick #3580: Restrict actor capabilities.add and make mounts nodev by Dmitry Berkovich (@dberkov) in #3611
- [release-1.0] Cherry-pick 5 main changes through 2289ade by Dmitry Berkovich (@dberkov) in #3645
Bug Fixes
- atelet: detach Checkpoint from caller cancellation once underway by Chenyi Wang (@chw120) in #2602
- Actors can reach the network while they boot, before their wakeup probe passes by Quentin Bisson (@QuentinBisson) in #2117
- atenet: classify a 403 route outcome as denied, not resume_error by Timo Derstappen (@teemow) in #2534
- A gVisor actor going over its memory limit no longer takes down every actor on its worker by Benjamin Elder (@BenTheElder) in #3106
- benchmarking/nighthawk-egress: create namespaces in both deploy scripts by Yuan Gao (@ygao-g) in #3315
- microvm: forward sysctls and lift the guest's privileged-port limit to match gVisor by Davanum Srinivas (@dims) in #1904
- Releases publish the envoy-dataplane image by Grant McCloskey (@MushuEE) in #3252
- atelet, ateworker: finish a checkpoint retry after atelet restarts by Chenyi Wang (@chw120) in #3218
- ateworker: reuse a completed checkpoint only on the same boot by Chenyi Wang (@chw120) in #3213
- atelet: keep the worker's Unavailable when a checkpoint fails by Aditya Shantanu (@aditya-shantanu) in #3123
- fix atelet deviceplugin wedge by Benjamin Elder (@BenTheElder) in #3323
- [release-1.0] Cherry-pick #3476: leave page cache out of the memory guard's overshoot check by Benjamin Elder (@BenTheElder) in #3513
Features: Networking and Egress
- e2e: use a port-forward per router CONNECT tunnel by Benjamin Elder (@BenTheElder) in #3051
- envoy: cross compile instead of compiling under slow emulation by Benjamin Elder (@BenTheElder) in #3152
- actornet: pin the sysctls a sandbox namespace inherits from the node by Benjamin Elder (@BenTheElder) in #3310
- Add kubectl ate proxy actor command. by Julian Gutierrez Oschmann (@juli4n) in #3191
- Allow the egress gateway to use more than one credential provider by Max Thompson (@thompsonmax) in #3409
Features: Security and Identity
- manifests: Run control plane containers as non-root by Mike Dickey (@mikedickey) in #3028
- Enable runsc core tagging when supported by the kernel by Fabricio Voznika (@fvoznika) in #2887
Features: Workers and Actors
- require workers to run in their own cgroup namespace by Benjamin Elder (@BenTheElder) in #3110
- Update gVisor to release 20261026.0 by nybidari in #3364
- Update a deleting tag returns FAILED_PRECONDITION by Luiz Oliveira (@laoj2) in #3340
- Suspending actors no longer OOM-kills tight worker pods by Benjamin Elder (@BenTheElder) in #3353
- ateapi: carry the golden actor's crash message into the template status by Davanum Srinivas (@dims) in #2644
- [release-1.0] Cherry-pick #3456: microvm: skip software bounce buffer (SWIOTLB) by default by jiayimeow in #3515
Features: Install and Operations
- ci: automate release artifact build and publish by Steven Shriver (@stshrive) in #2112
- On size10 clusters, the bundled Postgres accepts 500 connections instead of 300 by haiyanmeng in #3141
- benchmarking: add the egress-tunnels workload by Yuan Gao (@ygao-g) in #2716
- Add documentation for SLOs and alerting by Adrian Ludwin (@adrianludwin) in #2476
- [release-1.0] Cherry-pick #3446: ate-setup: retry each ko release image with backoff by Anna Pendleton (@annapendleton) in #3459
Documentation
- document that PRs are squash-merged, only HEAD needs to build, optimize for reviews by Benjamin Elder (@BenTheElder) in #3109
- correct the minimum kernel for the image cache: Linux 6.7, or 6.6.23 by Benjamin Elder (@BenTheElder) in #3128
- docs: use kvm-arm.vgic_v3_group1_trap=1 for Lima microVM setup on arm64 by Benjamin Elder (@BenTheElder) in #3149
- microvm consistency: standardize docs and agent instructions by Benjamin Elder (@BenTheElder) in #3105
- atelet: say snapshot cache hits fall back to copies across mounts by Haowei Cai (Roy) (@roycaihw) in #3064
- docs: require DCO sign-off instead of the Google CLA by Benjamin Elder (@BenTheElder) in #3237
- docs: fix the README note on enabling beta APIs by Haven Xia (@HavenXia) in #3266
- [bugbash] hack/teardown.sh: add --cluster-scoped to tear down one cluster by Haven Xia (@HavenXia) in #3296
- benchmarking: what to set and watch before a dense run by Davanum Srinivas (@dims) in #2499
- docs: add extensibility guide for gateways by Greg Castle (@destijl) in #3317
- AGENTS.md: add the version skew rules by Haven Xia (@HavenXia) in #3151
- Document running the ate-api-server cold resume tests from a workstation by haiyanmeng in #3394
- Add Hugo Docsite by Mofi Rahman (@moficodes) in #1974
Other Changes
- Prevent storing actors/tags with malformed durable snapshot URIs by Luiz Oliveira (@laoj2) in #3082
- Remove outdated roadmap and update security vulnerability reporting guidelines by Greg Castle (@destijl) in #3122
- glutton: run the RunScript timing tests on a synctest bubble by Max Smythe (@maxsmythe) in #3090
- atecontroller: Limit the Secrets read to the egress MITM CA pool by Mike Dickey (@mikedickey) in #3036
- manifests: Give podcertificate-controller its own ServiceAccount by Mike Dickey (@mikedickey) in #3059
- atelet: return Canceled when Run or Restore fails to reset actor dirs by Zoe Zhao (@zoez7) in #3063
- Resume restores an uploaded snapshot when its node is gone by igooch in #3094
- Move public APIs under
api/<api-name>/v1. by Julian Gutierrez Oschmann (@juli4n) in #2996 - Fix statusz port in k8s credential provider by Yufan Su (@yufan-su) in #3067
- benchmarking/locust: cap trial_summary actor density by live actors by Chenyi Wang (@chw120) in #3140
- ateapi: squash PostgreSQL migrations and lock them once merged by Jeremy Alvis (@iplay88keys) in #2991
- e2e: fix the TestKillMatching flake, wait for the child's command line by Davanum Srinivas (@dims) in #3155
- ateapi accepts snapshot report from atelet for actors in CRASHED state by Zoe Zhao (@zoez7) in #3148
- ateworker: reuse completed checkpoint when workload is already stopped by Chenyi Wang (@chw120) in #3015
- gVisor sandboxes move to release 20261019.0 from the 2026-10-17 nightly by Davanum Srinivas (@dims) in #3177
- Reject WorkerPool names longer than 63 characters by haiyanmeng in #3136
- atelet: verify microvm SandboxConfig assets before writing sandbox-runtimes.json by Shrutiya Mohan (@shrutiyam-glitch) in #3143
- benchmarking: dial atenet-router over SPIFFE mTLS on port 443 by Sairaj Pokale (@sairajp-rewind) in #3089
- Base images drop the distrusted eMudhra roots and the kind Jaeger moves to v2.22.0 by Davanum Srinivas (@dims) in #3196
- UpdateActor fails with Aborted while another operation on the actor is in progress by Zoe Zhao (@zoez7) in #3131
- e2e: lower kind atelet snapshot upload delay and update revert tests by Michelle Au (@msau42) in #3182
- atelet: emit a Warning Event on worker pods when syncing sandbox runtimes fails by Shrutiya Mohan (@shrutiyam-glitch) in #3146
- Generate the actor identity CA with ECDSA P-256 so external relying parties accept it by Greg Castle (@destijl) in #3251
- Releases publish the Kubernetes Secrets credential provider image by Grant McCloskey (@MushuEE) in #3220
- Fix flake: retry adding the kill container on a conflict by Haven Xia (@HavenXia) in #3183
- ateapi: deny streaming RPCs that have no authorization rule by Shruti Nair (@SHRUTI6991) in #3262
- Rename
SnapshotStorage.statustostateby Luiz Oliveira (@laoj2) in #3215 - An actor's new egress connections no longer fail with 403 an hour after it starts by Chenyi Wang (@chw120) in #3144
- Update agentgateway version to fix drift and CI by Keith Mattix II (@keithmattix) in #2962
- Document ReportActorCrashed in the WorkerService authz table by Shruti Nair (@SHRUTI6991) in #3324
- ate-setup: separate e2e kustomize component from kind overlay by Michelle Au (@msau42) in #3318
- e2e: test RevertActor across multiple durable snapshot generations by nybidari in #3338
- Assert HTTP/2 in egress CONNECT access log and reduce /mux concurrency by Bo-Cheng Chu (@bochengchu) in #3339
- The GCP Secret Manager credential provider moved to agent-substrate-plugins/credprovider-gcp-secret-manager by Benjamin Elder (@BenTheElder) in #3365
- Add full name of sfunkenhauser to MAINTAINERS.md by Scott Funkenhauser (@sfunkenhauser) in #3384
- UpdateActor rejects Actors marked for deletion by Luiz Oliveira (@laoj2) in #3341
- CreateActor fails if [golden] tag is marked for deletion by Luiz Oliveira (@laoj2) in #3382
- doc: add details on how a DNS denial from egress policy looks inside the actor by Yufan Su (@yufan-su) in #3363
- The template reconciler skips templates marked for deletion by Luiz Oliveira (@laoj2) in #3342
- benchmarking: remove default ATE_ENV_IMAGE and stop deploying ate-env template by default by Sairaj Pokale (@sairajp-rewind) in #3366
- Delete hack/install-ate.sh and hack/install-ate-kind.sh by Haven Xia (@HavenXia) in #3354
- atelet: clear node-local caches after a node reboot by igooch in #3376
- Reject snapshotConfig.storageLocation without its preview gate by Sneha-at (@Sneha-at) in #3357
- ate-api-server's --egress-gateway-spiffe-id is repeatable, and ate-setup sets it from ATE_EGRESS_GATEWAY_EXTRA_SPIFFE_IDS by Max Thompson (@thompsonmax) in #3360
- site: align footer with CNCF website guidelines and update navbar GitHub button by Mofi Rahman (@moficodes) in #3426
- objectstore-plugin: move the credential exchange's actor checks into actorauthz by Anish Ramasekar (@aramase) in #3138
- docs: say "no escape" from the sandbox on the ate.dev home page and security overview by Greg Castle (@destijl) in #3440
- [release-1.0 Cherry-pick #3507: docs: clone the repo before quickstarting by Ricardo Pchevuzinske Katz (@rikatz) in https://github.com//pull/3512
- [release-1.0] Cherry-pick #3442: docs: fix outdated statements in the egress guides by Yufan Su (@yufan-su) in #3516
- Revert "[release-1.0] Cherry-pick 19 main changes through 5a5b192 (#3571)" by Dmitry Berkovich (@dberkov) in #3593
- [release-1.0] Cherry-pick #2605 and #2911: Document actor JWT verification, federation, and key rotation by Max Thompson (@thompsonmax) in #3605
- [release-1.0] Cherry-pick #3644: Update Copyright for CNCF. All Code was donated, accept only "The Agent Substrate Authors" going forward by Benjamin Elder (@BenTheElder) in #3661
- [release-1.0] cherry pick: combine collaborating.md and contributing.md into the standard CONTRI… by Benjamin Elder (@BenTheElder) in #3662
- Extend CONTRIBUTING.md to discuss compatibility by Tim Hockin (@thockin) in #3697
- [release-1.0] Cherry-pick #3642: Point the README GKE quickstart at the GKE installation guide by Benjamin Elder (@BenTheElder) in #3699
New Contributors
- Mike Dickey (@mikedickey) made their first contribution in #3036
- Timo Derstappen (@teemow) made their first contribution in #2534
- Adrian Ludwin (@adrianludwin) made their first contribution in #2476
Full Changelog: v0.6.0...v1.0.0
Images
Container images for this release are published as ghcr.io/agent-substrate/substrate/<image>:v1.0.0 for linux/amd64 and linux/arm64. The images a release publishes are listed in Release images.
Install
From a checkout of v1.0.0, with Go, install the release images into the current kubectl context. On GKE, the default --sandbox-classes=gvisor,microvm downloads microVM guest assets from GitHub and uploads them with gcloud to gs://<bucket>-assets (created with atelet read access by tools/setup-gcp create bucket and create iam in https://github.com/ai-on-gke/substrate-gke). Add --kind for a kind cluster, or --sandbox-classes=gvisor to skip microVM staging.
git clone --depth 1 --branch v1.0.0 https://github.com/agent-substrate/substrate.git
cd substrate
go run ./cmd/ate-setup deploy ate-system \
--image-repo ghcr.io/agent-substrate/substrate --image-tag v1.0.0 \
--storage-bucket-name=<bucket> \
--credential-provider='{"name":"k8s.io"}' \
--ingress-allowed-spiffe-ids=<SPIFFE IDs of your ingress clients>See Installing pre-built images for the flags.
kubectl-ate
kubectl-ate is attached for Linux and macOS on amd64 and arm64. Download the binary for your platform and its .sha256 file, check it, and put it on your PATH as kubectl-ate:
shasum -a 256 -c kubectl-ate-linux-amd64.sha256
install -m 0755 kubectl-ate-linux-amd64 ~/.local/bin/kubectl-ateCommitters in this release
- Adam Wathieu (@adamwathieu)
- Aditya Shantanu (@aditya-shantanu)
- Adrian Ludwin (@adrianludwin)
- Anna Pendleton (@annapendleton)
- Anish Ramasekar (@aramase)
- Benjamin Elder (@BenTheElder)
- Bo-Cheng Chu (@bochengchu)
- Chuang Wang (@chuangw6)
- Chenyi Wang (@chw120)
- Luomu (@cloudonly)
- Dmitry Berkovich (@dberkov)
- Greg Castle (@destijl)
- Davanum Srinivas (@dims)
- Eitan Yarmush (@EItanya)
- Eric Curtin (@ericcurtin)
- Fabricio Voznika (@fvoznika)
- haiyanmeng
- Haven Xia (@HavenXia)
- Huy Pham (@hdp617)
- igooch
- Jeremy Alvis (@iplay88keys)
- jiayimeow
- Julian Gutierrez Oschmann (@juli4n)
- Keith Mattix II (@keithmattix)
- Luiz Oliveira (@laoj2)
- Lior Lieberman (@LiorLieberman)
- Max Smythe (@maxsmythe)
- Mike Dickey (@mikedickey)
- Mofi Rahman (@moficodes)
- Michelle Au (@msau42)
- Grant McCloskey (@MushuEE)
- noahlwest
- nybidari
- Quentin Bisson (@QuentinBisson)
- Ricardo Pchevuzinske Katz (@rikatz)
- Haowei Cai (Roy) (@roycaihw)
- Sairaj Pokale (@sairajp-rewind)
- Scott Funkenhauser (@sfunkenhauser)
- Shruti Nair (@SHRUTI6991)
- Shrutiya Mohan (@shrutiyam-glitch)
- Sneha-at (@Sneha-at)
- Steven Shriver (@stshrive)
- Timo Derstappen (@teemow)
- Tim Hockin (@thockin)
- Max Thompson (@thompsonmax)
- Tom O'Rourke (@tjorourke)
- Yuan Gao (@ygao-g)
- Yufan Su (@yufan-su)
- Zoe Zhao (@zoez7)