Skip to content

v1.0.0-rc.2

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 04 Nov 02:35
· 54 commits to main since this release
v1.0.0-rc.2
e59281c

v1.0.0-rc.2 is the second release candidate for Substrate 1.0. It brings release-1.0 up to main as of November 3 and hardens the sandboxes before GA: actors run as the image's USER, capabilities.add is limited to a short allowlist, /tmp and /dev/shm are tmpfs, and mounts are nodev on both runtimes. It also fixes a worker outage from microVM memory overshoot, a 15 s stall on every atenet-router start, an actor and worker assignment that could drift apart, and the device plugin wedging after an atelet restart.

Upgrading from rc.1 is possible but breaks gVisor snapshots and needs an image cache reset. See Upgrading from v1.0.0-rc.1 at the end. Upgrading from v0.6.0 is not supported; reinstall.

What's Changed

⚠️ Breaking Changes

Settings to change before upgrading:

  • Snapshot location path segments are validated (#2784 by Anish Ramasekar (@aramase)). Each segment of --snapshot-location (SNAPSHOT_LOCATION) must match ^[A-Za-z0-9_-]+$ and cannot equal atespaces in any case. Fix the path before upgrading or ate-api-server refuses to start.
  • --cordon-control-plane is now --cordon-system-infra-components (#3210 by haiyanmeng). The env var is ATE_INSTALL_CORDON_SYSTEM_INFRA_COMPONENTS, the config key cordonSystemInfraComponents, and the shared pool label and taint ate.dev/workloadType=ate-system-infra. The old flag and config key fail; the old env var is ignored silently and the install stops cordoning. Relabel and retaint the pool, then redeploy with the new flag; the PR has the gcloud commands.
  • ActorTemplates may only add 12 capabilities (#3580 by Greg Castle (@destijl), #3487 by Benjamin Elder (@BenTheElder)). securityContext.capabilities.add accepts only AUDIT_WRITE, CHOWN, DAC_OVERRIDE, FOWNER, FSETID, KILL, NET_BIND_SERVICE, SETFCAP, SETGID, SETPCAP, SETUID and SYS_CHROOT; MKNOD, SYS_ADMIN, SYS_RAWIO and the rest are rejected at admission. Unknown names in add or drop are rejected too. Check your templates for typos and for capabilities outside the list. Unprivileged ping keeps working without NET_RAW.
  • Custom credential-exchange sidecars lose --actor-jwt-jwks-file (#3447 by Anish Ramasekar (@aramase)). The exchange now fetches keys through OIDC discovery. Drop the flag from custom sidecar configs, and set --actor-jwt-issuer-ca-file only when the issuer uses a private CA other than the service DNS one.
  • The public MintActorCertificate RPC is removed (#3427 by Eitan Yarmush (@EItanya)). Calls return UNIMPLEMENTED. Remove references to its client method and message types before upgrading. Actor JWTs and the worker certificates atunnel uses are unchanged.
  • Worker pods need the new worker image before the new ate-controller (#2428 by Tom O'Rourke (@tjorourke)). Workers get runtime-specific capability sets and the pod template changed. Move every WorkerPool to the new worker image first; a microVM pool left on the old image stops working once ate-controller rolls it. The PR has the upgrade and rollback steps.
  • Pools hold a little less (#3104 by Benjamin Elder (@BenTheElder)). A worker's capacity is its memory limit minus 64 MiB, and each microVM actor books its limit plus 128 MiB + 8 MiB/GiB. A pool sized to exactly fit its actors needs limits.memory raised by about that much. MicroVM guests now see their whole memory limit.

Behavior changes after upgrading:

  • Containers run as the image's USER and WORKDIR, and the image cache layout changed (#2392 by Benjamin Elder (@BenTheElder)). Images with a USER stop running as root and capabilities.add no longer helps a non-root user. After an upgrade or rollback, atelet refuses the old image cache: delete /var/lib/ate/image-cache on each node once no actor runs there, or reboot the node. A custom atelet securityContext needs CHOWN, DAC_OVERRIDE, FOWNER and FSETID.
  • /tmp and /dev/shm are tmpfs on both runtimes (#3533 by nybidari). ROOTFS snapshots no longer keep /tmp, and image-provided files there are hidden. MicroVM /tmp counts against actor memory, capped at half the guest's memory; gVisor /dev/shm is 64 MiB. Mount a durableDir at either path for persistence or more space. /run now comes from the image rootfs.
  • gVisor MEMORY snapshots from before this release don't restore (#3533, #3580, #2392). runsc restore rejects a checkpoint whose mounts or user differ from the new spec, so suspended gVisor actors crash on resume and templates created before the upgrade crash on first resume. RevertActor doesn't help. Recreate the templates and replace the actors. MicroVM actors are unaffected.
  • MicroVM assets move to kata 4.3.1, Envoy to v1.39.5 (#3181 by Davanum Srinivas (@dims)). A microVM MEMORY snapshot taken on kata 4.1.0 restores at ROOTFS fidelity on its next resume: a fresh guest with its rootfs writes and volumes, no memory state. Recreate templates whose golden you want warm again.
  • A microVM actor that overshoots its booking is killed, not its worker (#3429 by Benjamin Elder (@BenTheElder)). Each actor's cgroup gets memory.high at what it books; one stuck there is killed and reported crashed with "killed for exceeding its memory limit". Other actors keep running. Without PSI the worker sets no limit and warns at startup.
  • MicroVM guests get containerd's default masked and read-only paths (#3249 by Huy Pham (@hdp617)). A uid 0 entrypoint can no longer write guest sysctls such as vm.max_map_count, matching gVisor. Goldens and checkpoints taken before this release keep a writable /proc/sys until re-taken.
  • WorkerPool NetworkPolicies are named substrate-<pool> (#3130 by Yufan Su (@yufan-su)). Pools with similar long names no longer share a policy. On an in-place upgrade the old substrate-<truncated>-<hash> policy stays until the pool is deleted; once the new one exists, delete the old ones with the command in the PR.
  • An actor and its worker assignment are written in one transaction (#3223 by Scott Funkenhauser (@sfunkenhauser)). Earlier releases could leave them out of sync after an interrupted write, and this release no longer repairs that. A worker that can't be deleted or an actor stuck in RESUMING is the symptom.
  • Atespace names must not be reused (#3450 by Shruti Nair (@SHRUTI6991)). A request authorized against a deleted atespace can apply to a new one with the same name, and nothing enforces uniqueness. Pick a new name instead.

Bug Fixes

Features: Networking and Egress

Features: Security and Identity

Features: Workers and Actors

Features: Install and Operations

Documentation

Other Changes

New Contributors

Full Changelog: v0.6.0...v1.0.0-rc.2

Images

Container images for this release are published as ghcr.io/agent-substrate/substrate/<image>:v1.0.0-rc.2 for linux/amd64 and linux/arm64. The images a release publishes are listed in Release images.

Install

From a checkout of v1.0.0-rc.2, with Go, install the release images into the current kubectl context. On GKE, the default --sandbox-classes=gvisor,microvm downloads microVM guest assets from GitHub and uploads them with gcloud to gs://<bucket>-assets (created with atelet read access by tools/setup-gcp create bucket and create iam in https://github.com/ai-on-gke/substrate-gke). Add --kind for a kind cluster, or --sandbox-classes=gvisor to skip microVM staging.

git clone --depth 1 --branch v1.0.0-rc.2 https://github.com/agent-substrate/substrate.git
cd substrate
go run ./cmd/ate-setup deploy ate-system \
  --image-repo ghcr.io/agent-substrate/substrate --image-tag v1.0.0-rc.2 \
  --storage-bucket-name=<bucket> \
  --credential-provider='{"name":"k8s.io"}' \
  --ingress-allowed-spiffe-ids=<SPIFFE IDs of your ingress clients>

See Installing pre-built images for the flags.

kubectl-ate

kubectl-ate is attached for Linux and macOS on amd64 and arm64. Download the binary for your platform and its .sha256 file, check it, and put it on your PATH as kubectl-ate:

shasum -a 256 -c kubectl-ate-linux-amd64.sha256
install -m 0755 kubectl-ate-linux-amd64 ~/.local/bin/kubectl-ate

Upgrading from v1.0.0-rc.1

The version skew policy doesn't cover upgrades between release candidates, and several changes since rc.1 break compatibility. You can still upgrade an rc.1 install to rc.2 with the upgrade runbook if you accept the effects below. Otherwise, reinstall.

gVisor actors keep only their volumes through this upgrade (see below), so before you start, let their work finish or save the state you need in a volume. Suspending every microVM actor before you start lets you clear the image cache at step 2 right away and makes stale assignments less likely, at the cost of downtime for the whole upgrade.

  • Image cache (#2392): rc.2 changes the on-disk layout of atelet's image cache, and at step 2 the new atelet exits until the old cache is gone. Delete /var/lib/ate/image-cache on each node once no actors run there, or reboot the node. The cache only holds pulled images, so nothing is lost.
  • Worker capabilities (#2428): the worker pod template changed. At step 3, move every WorkerPool to the new worker image, because a microVM pool left on the rc.1 image stops working at step 4b. Step 4b then rolls every pool once more.
  • gVisor snapshots (#3533, #3580, #2392): rc.2 changes the mounts and user that gVisor actors run with, so MEMORY snapshots (the default) and golden snapshots taken on rc.1 don't restore. Create new templates with the same volumes, and move each suspended gVisor actor to its new template with UpdateActor before you resume it. The first resume then restores only its volumes, so its memory and rootfs changes are lost. An actor that already crashed needs RevertActor first. MicroVM actors are unaffected.
  • Stale assignments (#3223): rc.1 wrote an actor and its worker assignment separately, so an interrupted write could leave them out of sync, and rc.2 no longer repairs that. If it happened, you'll see a worker that can't be deleted or an actor stuck in RESUMING.

Rolling back to rc.1 needs the image cache step again. Check the breaking changes above for settings to change before upgrading, such as the snapshot location (#2784), capabilities in ActorTemplates (#3487, #3580) and custom sidecar flags (#3447).

Committers in this release

Contributors

dims, teemow, and 46 other contributors
Assets 10
Loading