kubernetes-agent

Single static Go binary that drives Kubernetes node bootstrap on every VM in a Hypervisor-managed cluster — CP-init, CP-join, and worker-join. Replaces the bash scripts that used to live under resources/kubernetes-agent/scripts/.

The binary is private. It is built, published to the master server's authenticated storage, and downloaded by each VM during cloud-init using the cluster's short-lived bootstrap JWT. It is never on a public registry.

---

Why

The previous bash agent (bootstrap.sh + role scripts) was clear, version- controlled, and easy to debug — and that was the problem. The orchestration logic (phase ordering, retry semantics, certificate-key parsing, callback shape) was visible to anyone who looked at a running node. Porting the same logic to a compiled Go binary keeps the behaviour identical while making the implementation opaque.

External tools (kubeadm, kubectl, helm, openssl, containerd) are still invoked from disk via exec.Command. We are not hiding which tools run — we are hiding the orchestration around them.

---

Layout

kubernetes-agent/
├── go.mod
├── main.go
├── Makefile
├── README.md            ← you are here
├── internal/
│   ├── env/             /etc/cluster/env loader + validate
│   ├── callback/        POST events / kubeconfig / cert-key to master
│   ├── run/             exec.Cmd wrapper: retry, pipe, tee-to-log
│   └── cmd/             cobra subcommands
│       ├── root.go
│       ├── bootstrap.go bootstrap.sh
│       ├── cp_init.go   role/control-plane-init.sh
│       ├── cp_join.go   role/control-plane-join.sh
│       └── worker_join.go  role/worker.sh
└── dist/                build output (gitignored)

---

Subcommands

Command	Replaces
bootstrap	bootstrap.sh — dispatches by ROLE env (control-plane-init / control-plane-join / worker)
cp-init	role/control-plane-init.sh
cp-join	role/control-plane-join.sh
worker-join	role/worker.sh
version	prints build version stamp

Each subcommand emits the same callback events as the bash original, so the master-side UI / task logs continue working without changes.

Parity with bash agent

The Go agent is at functional parity with resources/kubernetes-agent/scripts/ as of this build:

• bootstrap: dispatches control-plane-init, control-plane-join, worker; emits unknown_role callback for anything else.
• cp-init: pre-seeds CA, runs kubeadm init phases with 3× retries, polls /healthz up to 10 min, captures the certificate-key from kubeadm-init.log, installs Cilium + hypervisor CCM + metrics-server, uploads admin.conf to master.
• cp-join: writes KUBELET_EXTRA_ARGS with --cloud-provider=external + --provider-id=hypervisor:///${INSTANCE_UUID} to the kubelet env file (auto-detected /etc/sysconfig vs /etc/default) before kubeadm join, then after join also patches the Node spec.providerID via kubectl patch so the CCM InstanceMetadata lookup never sees an empty provider-id.
• worker-join: identical kubelet-env build (provider-id + optional --node-labels / --register-with-taints), TCP-probe wait for CP LB, and kubeadm join with 5× retries × 30 s backoff.

Required env

Loaded from /etc/cluster/env (KEY=VALUE lines, identical format to the bash agent). Required keys:

CLUSTER_ID
ROLE                  control-plane-init | control-plane-join | worker
WAVE_ID
MASTER_API_BASE       e.g. https://panel.example.com/api/internal/cluster-controller/v1
BOOTSTRAP_JWT
KUBERNETES_VERSION
POD_CIDR
SERVICE_CIDR
CP_LB_ENDPOINT        host:port of the cluster's CP LB
CNI                   cilium

Role-specific extras:

worker, control-plane-join:
  KUBEADM_BOOTSTRAP_TOKEN
  KUBEADM_DISCOVERY_HASH

control-plane-join:
  KUBEADM_CERTIFICATE_KEY

Optional:

CILIUM_VERSION        defaults to 1.16.5
INSTANCE_UUID         used for --provider-id on workers

---

Build

make            # build dist/kubernetes-agent (linux/amd64)
make release    # adds -s -w + trimpath + SHA256SUMS
make publish    # release + drop to master storage
make clean

Single arch, single binary. We only target linux/amd64. Version stamp via -ldflags:

make release VERSION=v0.2.0

VERSION defaults to git describe --tags --always --dirty.

In WSL on a Windows-host path, go build complains about VCS — already worked around with -buildvcs=false in the Makefile.

---

Distribution

The agent is served by the master server, authenticated with the cluster's bootstrap JWT. No public registry, no embedded credentials, no versioned storage tree.

Storage layout on master

/opt/hypervisor/storage/kubernetes-agent      ← single static binary, mode 0755

make publish overwrites this file in place. Every cluster — old or new — pulls the same build on its next bootstrap.

Publishing

make publish                                   # default MASTER_STORAGE=/opt/hypervisor/storage
make publish MASTER_STORAGE=/custom/path       # override

Download endpoint

GET  /api/internal/cluster-controller/v1/cluster/{cluster_id}/agent
Authorization: Bearer <BOOTSTRAP_JWT>

No query params. Always linux/amd64. Response headers:

• Content-Type: application/octet-stream
• Content-Disposition: attachment; filename="kubernetes-agent"
• ETag: "<sha256>"
• X-Kubernetes-Agent-SHA256: <sha256>

The bootstrap JWT is per-cluster, short-lived, and scoped to role=bootstrap — a leaked JWT only buys access to that one cluster's agent download.

Cloud-init integration

The cloud-init runcmd for every cluster VM downloads the binary, verifies its SHA256 against the value injected in /etc/cluster/env, and execs the appropriate subcommand. Roughly:

curl -fSL -H "Authorization: Bearer $BOOTSTRAP_JWT" \
    "$MASTER_API_BASE/cluster/$CLUSTER_ID/agent" \
    -o /usr/local/bin/kubernetes-agent
echo "$CLUSTER_AGENT_SHA256  /usr/local/bin/kubernetes-agent" | sha256sum -c
chmod +x /usr/local/bin/kubernetes-agent
exec /usr/local/bin/kubernetes-agent bootstrap

This is the only bash left in the boot path.

---

Baking into the K8s VM golden image

You can pre-stage the agent inside the Kubernetes CP / Worker base images instead of pulling it on every cluster bootstrap. Net effect: faster boot (no download round-trip), one less failure mode in cloud-init, and the master agent endpoint becomes a fallback rather than the hot path.

Build flow for the image baker

# 1. Build release binary on a clean checkout
make release VERSION=v0.2.0
#    → dist/kubernetes-agent (linux/amd64, stripped)
#    → dist/SHA256SUMS

# 2. Hand the binary off to the image-build pipeline
cp dist/kubernetes-agent /path/to/image-builder/files/usr/local/bin/
cp dist/SHA256SUMS       /path/to/image-builder/files/usr/local/bin/

Inside the golden image, the binary lives at /usr/local/bin/kubernetes-agent (mode 0755, owner root:root). The SHA256 lives at /usr/local/bin/SHA256SUMS so a node can self-verify without phoning home.

Cloud-init behaviour with a pre-baked binary

The runcmd block becomes:

# Prefer the pre-baked binary; fall back to download if missing or stale.
if [ -x /usr/local/bin/kubernetes-agent ]; then
    /usr/local/bin/kubernetes-agent version
else
    curl -fSL -H "Authorization: Bearer $BOOTSTRAP_JWT" \
        "$MASTER_API_BASE/cluster/$CLUSTER_ID/agent" \
        -o /usr/local/bin/kubernetes-agent
    echo "$CLUSTER_AGENT_SHA256  /usr/local/bin/kubernetes-agent" | sha256sum -c
    chmod +x /usr/local/bin/kubernetes-agent
fi
exec /usr/local/bin/kubernetes-agent bootstrap

The master-served kubernetes-agent build is the source of truth. If the baked version drifts (because you released make publish but didn't rebuild golden images yet), cloud-init detects via SHA mismatch and pulls fresh. Recommended: add the sha256sum -c even when pre-baked, with the expected SHA piped in via /etc/cluster/env (CLUSTER_AGENT_SHA256).

Re-baking cadence

The agent's contract (env keys, callback shape) is stable. You only need to re-bake when:

• A subcommand's externally-observable behaviour changes (new env key required, new callback event, changed retry semantics)
• A security fix lands
• Go runtime bump (rare)

For minor / patch fixes that don't change the contract, ship via make publish and let cloud-init pick up the new SHA on next bootstrap.

Image-builder integration tips

• Pin the agent version in the golden-image build manifest (e.g. Packer variable, Earthly arg) so image lineage maps to a specific binary.
• Record the agent SHA256 + kubernetes-agent version output in the image's metadata (e.g. AWS AMI tags, Hypervisor image notes).
• Keep kubeadm / kubectl / kubelet baked at the same K8s minor as the image's intended cluster version (the agent's KUBERNETES_VERSION env must match what's installed; the agent does not install kube binaries itself).

---

Versioning

The agent's API surface (env keys, callback shape, manifest paths) is the contract. VERSION is stamped into the binary via -ldflags so kubernetes-agent version reports it at runtime, but there is no version negotiation — every cluster pulls whatever make publish last dropped at /opt/hypervisor/storage/kubernetes-agent. If you need version isolation, ship a new build and let nodes pull it on next bootstrap.

---

What's still on disk on the VM

The agent expects these tools to exist in $PATH. They are installed in the golden image, not by the agent itself:

• kubeadm, kubectl, kubelet — version matched to the cluster's K8s version
• helm — for Cilium install
• containerd + CRI plumbing

These are deliberately kept as native tools so users can kubeadm / kubectl themselves on the VM for debugging.

---

Hardening checklist (when ready)

Phase 1 (this branch) already does:

• -trimpath strips source paths
• -ldflags="-s -w" strips DWARF + symbol tables
• Static binary (CGO disabled) — no dynamic symbols at all

Future work for IP-paranoid deployments:

• garble build — scrambles function and identifier names
• //go:embed the YAML manifest templates; AES-GCM encrypt the embedded blobs and derive the decryption key from the cluster's bootstrap JWT so the binary alone reveals nothing
• Master-side rate-limit on the agent download endpoint per cluster
• Sign builds, master verifies signature before serving

---

Troubleshooting

• kubelet won't start, label rejected: check that worker-join did NOT write node-role.kubernetes.io/worker into KUBELET_EXTRA_ARGS — that label is blocked by NodeAuthorizer and master applies it post-join.
• apiserver health check times out: see /var/log/cluster/kubeadm-init.log on the first CP. The 10-min ceiling is intentional — if it expires, the CP node failed to come up, not the agent.
• callback events not reaching master: check MASTER_API_BASE is reachable from the VM and BOOTSTRAP_JWT is still within its TTL.
• agent download 404: verify make publish wrote /opt/hypervisor/storage/kubernetes-agent (single file, mode 0755) on the master.