Repository navigation
Runbooks Network
Operating the private network: connectors, sites, split DNS and the Headscale control plane. Concepts are on Private Network; the index is Runbooks. Every cdk deploy on this page is a hand deploy and passes --exclusively: without it CDK also deploys the DataStack and CredsStack dependencies from your local tree and can race the pipeline.
Manual, per target VPC — Epic 4 stacks never deploy from CI:
-
Declare the VPC in config — inside its account's SITE in
config/project.ts: add the account tonetwork.sites.<site>.accounts(an account lives in exactly one site) and aconnectorVpcsentry keyed by the account id:{ vpcId: "vpc-0abc…", routes: ["10.1.0.0/16"] }(routes = usually just the VPC's CIDR). For an account's SECOND VPC make the entry a list and give the new VPC aname({ vpcId, routes, name: "sandbox" }): it becomes that connector's hostname suffix, stack id suffix and function-name suffix, so set it once; redeploy the account's existing connector stack too (its SG-manager learns which VPC is its own). Addregion: "eu-west-1"when the VPC is outside the hub's region (step 2's NetworkStack deploy then also adds the relay the reconciler uses to reach that account's SG-manager and resolver). Synth fails on any CIDR overlap within the site (packages/infra-client/lib/network-validation.ts) — a colliding VPC goes in a different site instead (that's what sites are, the sites design note). Non-overlapping ranges share a site for free. -
Grant the account on THAT SITE's connector minter — add the account id
(or a role ARN for least privilege) to the site's
connectorAccounts, thennpx cdk deploy NetworkStack --exclusively --profile main. -
Deploy the connector into the target account:
npx cdk deploy Connector-<id> --exclusively --profile <target>(one stack perconnectorVpcsentry:Connector-<id>for the account's unnamed VPC,Connector-<id>-<name>for a named one; the deployed CloudFormation names stayConnectorStackandConnectorStack-<name>). A VPC in another region deploys there: bootstrap that account in the VPC's region first (npx cdk bootstrap aws://<id>/<region> --profile <target>). The account's site picks the control plane + minter automatically. The connector self-enrolls at boot (mints a key, joins, advertises its routes) — re-enrolling later is just restarting the task / redeploying. One-time note: connectors deployed before the per-account stack ids (2026-08-19) get an ECS task-family rename on their next deploy — a normal rolling task restart, not an outage. - If the VPC needs private DNS names, add its resolver to the SITE's
splitDns(VPC CIDR base + 2) and redeploy NetworkStack.
Reachability appears after the next reconciler pass (≤5 min) for users holding grants on resources registered in that account.
Retiring a VPC in another region: unregister its resources (or clear their
security group ids) and let one reconciler pass sweep the managed rules
BEFORE removing the connectorVpcs entry — once NetworkStack redeploys
without that region the hub can no longer reach it; the reconciler then logs
the account once, stops retrying, and the rules must be revoked by hand.
Retiring any connector VPC, in order: unregister its resources and let one
reconciler pass sweep their rules, cdk destroy its Connector-<id>[-<name>]
stack (the VPC itself only after that — the connector's network interfaces
live in it), remove the connectorVpcs entry and redeploy NetworkStack.
Then delete the retired connector's node on the site's control server — the
reconciler only prunes a connector that has a live replacement in the SAME
VPC, so this one stays listed as stale (and alarms) until removed:
aws ecs execute-command --cluster <NetworkStack cluster> --task <site's headscale task> \
--container headscale --interactive --command "headscale nodes list"
# then, with the connector's ID from that list:
# --command "headscale nodes delete --identifier <ID> --force"CloudFormation keeps the destroyed stacks' log groups; delete them in the target account if nothing needs them.
-
First time ever moving this deployment to sites (from the old single
headscale.…plane) → "One-time migration" below. Runs once, re-enrolls everything, never again. - Adding a VPC whose range doesn't collide with its site → "Adding a VPC to the network". No new site, no re-enrollment, users notice nothing.
- Adding a VPC whose range DOES collide (the synth error says so) → "Adding a SITE" first, then "Adding a VPC" into the new site. Existing sites and devices are untouched; users just gain a switcher entry.
Moving off headscale.portal.example.com to dev.network.… re-enrolls
every device and connector ONCE (login hostnames are baked into their state —
SITES.md calls this the deliberate break). Order matters twice: TestbedStack
must come DOWN before NetworkStack deploys (it imports the old connector
minter cross-stack, and CloudFormation refuses to delete an in-use export),
and the manual NetworkStack deploy goes BEFORE the CI push — pushing first
would make the API advertise site URLs that don't exist yet.
-
npx cdk destroy TestbedStack --profile main(temporary stack, nothing precious — its Aurora/nginx are test fixtures). -
npx cdk deploy NetworkStack --exclusively --profile main— replaces the old single Headscale withheadscale-devbehind the new wildcard ALB setup. The old EFS directory (/headscale) is abandoned in place; the site starts empty (that IS the re-enrollment). The deploy creates the site'sinfrauser and admin API key itself (since 2026-08-28 — no secret placeholder, no ECS Exec, no rotation stamp on a first deploy); the outputHeadscaleApiKeydevreadscreated. -
npx cdk deploy TestbedStack --exclusively --profile main— its connector re-enrolls itself into dev at boot; re-register/verify any testbed catalog resources whose endpoints changed with the rebuild. - Commit + push (CI deploys the API/frontend that advertise the site list).
- Each device: relaunch the desktop app (tray → Quit, reopen — the shell
itself changed), click Connect (fresh enrollment; first-connect rules take
≤5 min). The old
headscale.…DNS record and the oldinfra-client/headscale-apikeysecret can be deleted once everything is over — that secret deletion is a hand step by design (see "Removing a SITE").
Only when a new VPC's CIDR collides with an existing site (the synth error tells you). Site ids are permanent — pick a short meaningful slug.
- Add
network.sites.<newsite>to config (accounts, connectorVpcs, splitDns). Move NOTHING between existing sites — that re-enrolls devices. -
npx cdk deploy NetworkStack --exclusively --profile main— brings up the site's Headscale, ALB rule, minter, AND itsinfrauser + admin API key (the server container mints them at start;Custom::HeadscaleApiKeystores the key as<resourcePrefix>/headscale-apikey-<newsite>). DNS/cert need nothing (wildcard). OutputHeadscaleApiKey<newsite>=created. - Commit + push promptly — until CI updates the API, an old API omits the new site and mints default to the FIRST site.
- Deploy the site's connectors (previous runbook). Users see the new site's VPCs in the app and switch from the pill — one site at a time (desktop users must have relaunched the app since the sites update, or switching asks them to).
Delete network.sites.<site> from config and npx cdk deploy NetworkStack --exclusively --profile main: the service, ALB rule, minter, and the site's custom
resource go away; devices enrolled there lose that site. The one manual
step left in the whole VPN lifecycle: the site's API-key secret is kept on
purpose (the custom resource's Delete is a no-op — deleting it automatically
could strand a still-running server or a rollback mid-teardown), so remove it
by hand once you're sure:
aws secretsmanager delete-secret --secret-id <resourcePrefix>/headscale-apikey-<site> --profile main
and the certificate parameter the task published:
aws ssm delete-parameter --name /<resourcePrefix>/headscale-tls-<site> --profile main.
The site's EFS directory stays too (RETAIN under prod); delete it only if
the site will never return.
Connector hygiene is automatic: a redeployed connector enrolls as a new
node, and the reconciler deletes its offline predecessor once a trusted
replacement for the same account and VPC is online for 24h+ (logged as
prunedConnectors). Until then the predecessor is listed as
supersededConnectors, not as stale, so a routine task replacement (Fargate
retirement, a redeploy) does not raise the StaleConnectors alarm. An
account's ONLY connector is never deleted — it stays in the stale list and the
alarm, so an outage is seen rather than tidied away.
Relay map (DERP) failures heal the same way. Each control-server start saves
the live relay map to the site's EFS folder (derp-fallback.yaml), and
Headscale falls back to that copy when its daily refresh cannot download the
map, so nodes keep their relays. A connector that reports "could not connect
to the ... relay server" for about five minutes fails its ECS health check
and is replaced; its routes auto-approve as for any redeploy. When
NetworkStack has an alert topic, the DerpMapLoadFailed alarm fires on
every failed load (its OK only means no new failure was logged): if the site's log shows the saved copy is missing too (a site whose
first start could not download the map), force a new deployment of its
headscale-<site> service once the URL loads again.
No admin port is exposed; admin operations go through ECS Exec — per SITE
(each site is its own Fargate service, headscale-<site>):
TASK=$(aws ecs list-tasks --cluster <cluster> --service-name <headscale-<site> service> \
--query 'taskArns[0]' --output text)
aws ecs execute-command --cluster <cluster> --task $TASK --container headscale \
--interactive --command "headscale nodes list"The infra user and the admin API key are created BY THE DEPLOY (since
2026-08-28). Headscale's CLI only works inside the serving container (it
talks to the running server over a Unix socket on the container's own tmpfs
at /tmp; the rendered config and the self-signed TLS pair sit next to the
database on the EFS mount, /var/lib/headscale), so the headscale
container runs the two CLI steps itself once its server answers:
headscale users create infra and headscale apikeys create --expiration 87600h, printed to its own log stream headscale-<site>/headscale/<task>.
A Custom::HeadscaleApiKey resource per site then reads that stream and
stores the key under
<resourcePrefix>/headscale-apikey-<site> — a hand-made secret is adopted as
is. Every task start mints a key; only the newest is stored, and only when
the secret is absent or the rotation stamp changed (headscale apikeys list
/ expire via ECS Exec if you want to prune old ones). The stack output
HeadscaleApiKey<site> says existing, created, or rotated; the
handler's log group is /aws/lambda/NetworkStack-HeadscaleApiKeyFn*.
The secret is resolved by CloudFormation at deploy time. To ROTATE the key,
redeploy with -c headscaleKeyRotation=$(date +%s): the stamp rolls the
service (fresh task → fresh key), the resource stores it, and every
minter/reconciler picks it up in the same deploy (without the stamp
CloudFormation keeps the old key). Teardown keeps
the secret (the resource's Delete is a no-op) — delete it by hand with the
stack if the site is gone for good. The public ALB 403-blocks /api/* — admin traffic goes
Lambda → Cloud Map (headscale-<site>.tailnet.internal) → task over HTTPS,
verified against the certificate the task published to SSM
(/<resourcePrefix>/headscale-tls-<site>), never through the ALB.
Source: wiki/ in FNTIO/aws-infrastructure-client, mirrored by the Wiki workflow. Edit the files there; changes made on this site are overwritten by the next sync.
Start here
Use
How it works
Operate
Contribute