substrate-gke

GKE packaging for Agent Substrate: an interactive terminal installer that provisions the required GCP resources and installs the Substrate control plane onto a GKE cluster.

• Quickstart
• What the installer does
• Sandbox runtimes
• Where the images come from
• How Substrate itself is obtained
• Logs
• Upgrading an installed cluster
• Tearing down
• Development
• Glossary

Prerequisites

# One-line install and launch:
curl -sSL https://raw.githubusercontent.com/ai-on-gke/substrate-gke/main/install.sh | bash

Or from a local clone:

git clone https://github.com/ai-on-gke/substrate-gke.git
cd substrate-gke
gcloud auth application-default login
make run          # launch the interactive installer

Useful variants

A terminal wizard walks the ten steps below, running the real command it shows and streaming its output as it goes:

Connect your cluster	Blocked: already installed

Provisioning	Complete

Substrate runs each actor inside a sandbox, and step 9 chooses which kind.

gVisor is the default and needs nothing extra: ate-setup applies the cluster-wide gvisor-default SandboxConfig as part of turning Substrate on.

Micro-VM is opt-in, because its sandbox binaries — cloud-hypervisor, virtiofsd, the guest kernel, the guest rootfs and the base kata config — are fetched from the snapshot bucket at boot rather than baked into the worker image. Step 9 (Choose your sandbox runtime) reports whether the cluster you selected has a KVM-capable node pool, and on choosing Micro-VM it runs upstream's hack/install-microvm-deps.sh --install to assemble those five assets, upload them under kata-assets/ in the snapshot bucket, and apply a cluster-wide SandboxConfig named microvm that points at them.

Actor templates must then name it explicitly:

sandboxConfig:
  sandboxClass: SANDBOX_CLASS_MICROVM
  configName: microvm

gcloud container node-pools list --cluster CLUSTER --location LOCATION \
  --format='table(name,config.machineType,config.advancedMachineFeatures.enableNestedVirtualization)'

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.capacity.ate\.dev/kvm}{"\n"}{end}'

Upgrading and teardown: The upgrade flow (--upgrade) updates the control plane and gVisor SandboxConfig, but does not re-stage kata-assets/; after upgrading a cluster that uses micro-VM across a Substrate revision bump, target the cluster's kube context and re-run NO_DEV_ENV=1 ARCH=amd64 BUCKET_NAME=<bucket> hack/install-microvm-deps.sh --install so the bucket assets and microvm SandboxConfig digests stay in sync. Full teardown (ate-setup delete ate-system) removes the SandboxConfig CRD and all SandboxConfig resources automatically; if you want to drop micro-VM without tearing anything else down, run NO_DEV_ENV=1 hack/install-microvm-deps.sh --delete against the cluster's kube context (the staged assets in kata-assets/ remain inert in the bucket).

The images step chooses between two ways of getting the Substrate control-plane images. Both name a commit of agent-substrate/substrate — ate-setup reads deployment manifests from that source tree either way. The repository is fixed; the commit is yours to choose.

For source builds using the default registry, the installer creates ate-images at <region>-docker.pkg.dev/<project>/ate-images. The region follows the selected cluster. Advanced mode can override the repository name or use an existing image registry. Pre-built installs and custom image registries skip repository creation.

ate-setup needs a Substrate source tree on disk at install time regardless of which images you chose — a pre-built install skips the build, not the checkout. That's why both paths in the images step ask for a revision.

The installer fetches it with a shallow git fetch pinned to an exact commit, into:

<user cache dir>/substrate-gke/substrate-<short commit>

One directory per commit, so two revisions never collide. The default pin is public, so no credentials are needed for it.

# Point at your own checkout instead (handy for testing an unmerged change):
cd installer && go run . --substrate-root=/path/to/substrate

A checkout you supply this way is used as-is and never modified or deleted.

GCP provisioning is the exception: the setup-gcp bootstrap step runs this repository's tools/setup-gcp, not a copy from the Substrate tree, whatever revision you chose. The tool moved here from upstream (agent-substrate/substrate#2304). The installer finds it above its working directory or its executable; pass --setup-gcp=/path/to/tools/setup-gcp if you run the installer from somewhere else.

Every command's full output is written to a timestamped log under <user cache dir>/substrate-gke/logs/. Press v in the wizard for a scrollable log viewer; failures show the extracted cause and the log path.

That flow names the cluster, reads what it currently runs, takes the new version from the images step, fetches both the installed and new source trees into <user cache dir>/substrate-gke/upgrades/, and prints the hand-over in runbook order for upstream's rolling upgrade runbook: the runbook itself, the variables its commands use, which tree to check out, the environment for ate-setup, and what a rollback changes. Nothing on the cluster changes until you follow the runbook.

When a source upgrade needs the default Artifact Registry repository, the hand-over includes its setup commands.

The installed tree is fetched at the commit the running API server reports it was built from (Go stamps every binary; ateapi --version prints it). If the cluster can't be read, or the binary carries no commit, you'll be prompted to enter the commit, version, and image registry. For source builds, enter the registry the images were pushed to.

An install creates billable resources: the GKE cluster, the snapshot bucket, IAM bindings, monitoring dashboards, and any image repository provisioned for a source build.

./tools/cleanup-gcp --project <project> --cluster <cluster> --location <zone> --bucket <bucket>
# or:
make teardown PROJECT_ID=<project> CLUSTER_NAME=<cluster> CLUSTER_LOCATION=<zone> BUCKET_NAME=<bucket>

The script asks for confirmation, then runs this repository's tools/setup-gcp/teardown.sh. It needs only gcloud. cleanup-gcp sets NO_DEV_ENV=1, so teardown.sh ignores any .ate-dev-env.sh in the working directory; run directly, teardown.sh sources that file. Already-deleted resources are skipped. Failed deletion steps are listed at the end, and the script exits non-zero.

Repositories are kept by default; the installer adds --delete-repository when it provisioned one, and --repository if the name is not ate-images. The region is derived from the cluster location. Replace --delete-repository with --keep-repository if the images are shared. With make teardown, use DELETE_REPOSITORY=true and, for a custom name, pass ARTIFACT_REGISTRY_REPOSITORY=<name> on the command line.

To remove only the Substrate control plane and keep the cluster:

ate-setup delete ate-system   # printed in the exit summary

APIs enabled by the install are left enabled — they cost nothing while unused.

make test         # unit tests for installer/ and tools/setup-gcp/, including a scripted dry-run walk of the wizard
make verify       # gofmt + go vet for both modules
make screenshots  # regenerate the README screenshots from the dry-run wizard

This changes what fresh installs get. Edit Commit in installer/internal/snapshot/snapshot.go, and update MinGoVersion next to it to match the go directive in that revision's go.mod.

make substrate-pin        # print the current pinned values
make substrate-pin-check  # verify MinGoVersion against upstream's go.mod — run after every bump

Bump ReleaseVersion and Commit together when a newer release is published — Commit has to be the manifest revision those images were actually built from. All three (Commit, ReleaseRepo, ReleaseVersion) are defaults, not limits: the wizard accepts any registry, tag, or revision, and the build-from-source track never uses Commit at all.