Troubleshooting Index

Troubleshooting index

Before this page: Lessons learned tells the stories behind many of these rows, and Spacelift and CI, tests and the cost gate explain the machinery that prints the messages.

This page is a lookup table: you see something wrong, you find the row that matches, and the last column says where the fix is written down. Every row comes from something documented in this repository, either a failure that happened during the build or one of the labs that breaks things on purpose. Nothing here is a general guess about how these tools can fail. If your symptom is not in the table, that is a gap worth an issue.

Two habits that make the table faster to use:

• Read the exact message. Several rows differ only by the status code or one word (403 against 422, skip against fail).
• Find which layer printed it. A line starting doppler-env: or cost-check: is this repository's own script. A 422 or 403 is DigitalOcean's API. A fail with a red box is tofu test. Each layer has its own section below.

Runbook links go to the files in the repository, outside the wiki: Spacelift and Doppler secrets, Stage 0 and Stage 1.

Spacelift runs

Symptom	Likely cause	Where to look
A red spacelift/cah-10-access check on a pull request that changed only docs	The stack's project root or project globs cover more than the stack, so a docs-only change reaches it. The proposed run has no credentials by design and can only fail.	Lessons learned, "A red check from a run that could never pass"; the free-plan table of roots and globs is in runbook section 9
A red Spacelift check on a pull request that does change a stack, and every GitHub Actions job is green	On the free plan the push policy cannot be attached, so a pull request that changes a stack still gets a proposed run that fails at the provider. Branch protection requires only CI green, so it does not block the merge.	Spacelift, "Push policies: paid plan versus free plan"
push.rego is in the repository and tests pass, but Spacelift behaves as if it did not exist; or there is no Policies menu	Push policies are a paid Spacelift feature. Spacelift reads policies from its own store, not from the repository, so the file takes effect only once it is created there.	Lessons learned, "Then a second surprise"; Spacelift, push policies
A second run for the same stack sits queued behind another one you did not start	A stack dependency starts its own run when the stack it depends on finishes. Starting one by hand queued a duplicate.	Lessons learned, section 9, "A stack dependency starts its own run"; check the stack's run list before you trigger anything
A tracked run reaches the plan and then just waits	Autodeploy is off on purpose. A human reads the plan and confirms or discards.	Spacelift, "Why autodeploy is off"
The plan phase had its secrets but the apply phase does not	The apply can start in a fresh container where the plan phase's environment is gone. Both before_init and before_apply must load the same list.	Spacelift, "Why before_apply fetches secrets again"
A downstream secret such as DIGITALOCEAN_TOKEN is missing in a run that should have it, and you are tempted to look for the Doppler token in the log	Do not diagnose by the token's absence. The attached cah-doppler context supplies DOPPLER_TOKEN, DOPPLER_PROJECT and DOPPLER_CONFIG to every run type on its own; the hook only decides whether to fetch the other secrets. Read the hook's own lines instead: doppler-env: loaded NAME as VAR for each fetched value, or an explicit doppler-env: skip or failure message such as the run type, branch or context rows in this table.	Doppler and secrets, "How the hook works"; runbook section 8
doppler-env: branch '<name>' is not main; skipping	The hook fetches only on main. A run from another branch gets none of the downstream secrets, by design.	Doppler and secrets, "How the hook works"; Labs: Lab 5
doppler-env: invalid secret name: <arg> or doppler-env: no secret names given	An entry in the stack's list in .spacelift/config.yml is not a plain identifier (letters, digits and underscores, not starting with a digit), on either side of TF_VAR_foo=FOO; or the stack's list is empty.	Doppler and secrets, "How the hook works", step 4; Spacelift, "Adding a stack"
A design that relies on a Spacelift run identity token (OIDC) finds no token in the environment	The run identity token is not issued on this Spacelift plan. It is a plan limit, not a misconfiguration.	Lessons learned, "The plan said no OIDC"; Spacelift, "No OIDC on this plan"

Doppler and the secrets hook

Symptom	Likely cause	Where to look
run type 'PROPOSED' does not get secrets; skipping	A pull request plan. Expected: proposed runs get no credentials, and the plan then fails at the provider.	Runbook section 7; Spacelift, "Why a pull request plan gets no credentials"
<VAR> is not set; is the cah-doppler context attached?, where VAR is DOPPLER_TOKEN, DOPPLER_PROJECT or DOPPLER_CONFIG (a missing environment variable)	The context did not attach to the stack. Check that the stack has the label cah and the context has autoattach:cah.	Runbook section 7; Spacelift, "Adding a stack"
could not read NAME from Doppler	The service token is expired or revoked, or NAME is missing from the prd config. An expired token warns nobody; the next run just fails.	Doppler and secrets, "Rotation, in short" and runbook section 7
NAME is empty in Doppler	The secret exists but has no value.	Runbook section 7
checksum mismatch ... refusing to run it	The downloaded Doppler CLI is not the pinned release. Do not bypass the check.	Doppler and secrets, "How the hook works", step 3; runbook section 7
A secret is in Doppler but OpenTofu behaves as if the variable were never set	Case. OpenTofu reads variable foo only from the exact name TF_VAR_foo, and Doppler names are uppercase. The hook maps them with TF_VAR_foo=FOO.	Lessons learned, "Doppler names are uppercase, OpenTofu variables are not"
The token prefix dop_v1_ appears in a run log	Masking did not work. Revoke the token at once and change the masking approach.	Doppler and secrets, "First run checks"; Lessons learned, section 8
401 or 403 from DigitalOcean during the plan	The token expired, or a custom scope is missing. The token cannot be edited after creation, only replaced. A 403 reading SSH keys meant the ssh_key: read scope was missing.	Runbook section 7; Lessons learned, "Token scopes are proven by a run"

tofu test failures

Symptom	Likely cause	Where to look
Invalid function argument ... r.source_tags is null, then 0 passed, 1 failed, 33 skipped	An inbound rule written with an address instead of a source tag breaks the module's output code before the real assertion is reached. The test failed for the wrong reason but still protected you.	Labs: Lab 1
Missing expected failure ... was expected to report an error but did not	A guard was loosened. The test that expects a bad input to be refused did not get a refusal.	Labs: Lab 2
Tests marked skip after one failed	A failing test stops the rest of its file. Read skip as "not checked", never as "fine".	Labs: Lab 1 and Lab 2, "What to take away"
tofu test seems to need a cloud account or a token	It should not. Every test file mocks its providers, which return made-up IDs, so the tests need no credential and cost nothing. Run init -backend=false first.	OpenTofu, stacks and modules, "Tests without a cloud account"
After adding a Rego rule about pull requests, an unrelated policy test fails (for example "push to another branch starts nothing")	null is neither false nor undefined in Rego, and every test input carries "pull_request": null, so the new rule fires on every event.	Labs: Lab 4
Three test_non_main_branch_gets_nothing cases fail in tests/test_doppler_env.py	The branch check in the hook was removed or weakened. The cases include mainline, which a sloppy prefix check would accept.	Labs: Lab 5

Cost gate failures

The gate is scripts/cost_check.py. Every message starts with cost-check:, and a failing run exits 1. Where it prints depends on the kind of failure, so read every cost-check: line, above and below the per-cloud summary:

• A price table, manifest or plan that cannot be read or priced (an unknown size, a missing price table, a number that is not a number) stops the gate at once with one line on stderr, cost-check: unreadable manifest, price table or plan input: ..., and no summary is printed at all.
• Every other failure prints the summary lines (cost-check: <cloud>: planned $X/mo, approved $Y, ... and cost-check: phase <id> ceiling $Z (reason: ...)) to stdout and the diagnostics to stderr. The two streams are separate, so in a terminal or a CI log the diagnostics can appear below the summary or above it. A summary that looks fine does not mean the gate passed; the exit code does.

The messages come from the current multi-cloud version of the script.

Message	What it means	Where to look
unreadable manifest, price table or plan input: ... (for example no price for size '<size>', no price for instance type ..., no price model for kind ... on <cloud>, backups requested but the backup price fraction is unverified, no price table for provider ..., or an instance missing root_gb and public_ip)	A resource in a cost.json that the gate cannot price. It refuses rather than guess, and stops before printing a summary.	CI, tests and the cost gate; Provider AWS: where the money goes for the AWS instance fields
resource type not on the free list, not approved, or outside its allowed file: <file>: <type>	A paid resource type that is not on its cloud's free list, or a costed one (a droplet, for instance) outside the file approved.json allows. The planned total can still read the old number, because an unpriced type is a failure, not a zero.	Labs: Lab 3; CI, "Key ideas"
The same message ending (the gate has no price model for it)	The type is not free and the gate has no way to price it. approved.json cannot approve a type the gate cannot price. A load balancer is the example in Lab 3.	Labs: Lab 3
The same message ending (provider <cloud> is not in approved.json)	Code uses a cloud that has no entry in docs/stages/approved.json. Every cloud needs its own approval.	CI, "Each phase has its own ceiling"
<file>: <type> is rejected: ...	A type the gate refuses at any budget, such as an AWS NAT gateway (about $36.50 a month before traffic). The text after the colon says what to do instead.	Provider AWS: where the money goes
<cloud>: planned $X is above approved $Y	That cloud's manifests add up to more than its approved_monthly_total. Raising it is a reviewed diff.	How money works, "Approve before it exists, not after"; CI, cost gate
<cloud>: approved $X is above the ceiling $Y	One cloud's approval was raised past the current phase's ceiling (today $30). Raise the phase's ceiling on purpose, or lower the approval.	How money works, "A ceiling, written down"
the approved figures of the active providers add up to $X, above the shared ceiling $Y; mark one inactive or lower an approval	Each cloud is under the ceiling alone, but the clouds marked active together are not (the ceiling is the current phase's).	CI, "Each phase has its own ceiling"
current_phase <id> is not in phases, exactly one phase must have status current, found N, or phase <a> has status current but current_phase is <b>	current_phase and the phase statuses in approved.json disagree. Exactly one phase is current, and it is the one current_phase names.	CI, "Each phase has its own ceiling"
phase <id>: ceiling $X is below <previous> $Y; a lower ceiling needs "lowers": true and a reason (or marked lowers but the ceiling does not go below ...)	A phase's ceiling went down from the phase before it without saying so. Raising or keeping a ceiling is free; lowering one must be explicit, so nobody weakens a limit quietly.	How money works, "A ceiling, written down"
phase <id>: ceiling ... must be a finite number, reason must be a non-empty string, decided ... must be an ISO date, or status ... must be one of ...	A phase entry is missing a part. Each phase needs a ceiling, a reason, a decided date (YYYY-MM-DD) and a status.	CI, "Each phase has its own ceiling"
phase <id>: ceiling $X is above hard_max $Y	The top-level hard_max is the absolute limit; raise a ceiling past it only by editing hard_max in the same reviewed diff.	CI, "Each phase has its own ceiling"
approved.json has phases and a top-level ceiling; remove the top-level ceiling	The old single ceiling is read only when phases is absent. With phases, the current phase decides.	CI, "Each phase has its own ceiling"
duplicate key 'X' in a JSON object (from approved.json, a cost.json or a price table)	A JSON file repeats a key. Python would silently keep the last one, so the value enforced could differ from the one a reviewer reads. Delete the extra.	CI, "Each phase has its own ceiling"
hard_max is required when phases exist ... or hard_max_reason must say why hard_max is that number	The absolute limit is mandatory with phases, and it carries a reason.	CI, "Each phase has its own ceiling"
<label>: X must be a JSON number or has more than two decimal places	A ceiling, hard_max or approval is a string, a bool or has fractions of a cent. Write it as a plain number with at most two decimals.	CI, "Each phase has its own ceiling"
phase <id>: status <s> comes after <other>; phases are done, then current, then planned, in declared order	The statuses are out of order, for example a done phase after the current one. Phases move planned, current, done, one step at a time.	CI, "Each phase has its own ceiling"
ci-cost-gate: FAIL: could not fetch ... or ... has no docs/stages/approved.json	On a pull request CI compares approved.json with the base branch and refuses to continue without it. Re-run the job; if it keeps failing the runner cannot reach the base branch.	CI, "Each phase has its own ceiling"
phase <id>: done at $X, above the current ceiling $Y	A finished phase may not exceed the live ceiling (a planted high phase could otherwise be switched on).	CI, "Each phase has its own ceiling"
phase <id> is done and may not change (or phase <id> (<status>) may not be removed, status may not go from X to Y, is new and may not start as done)	Only with --base-approved, and in scripts/cost_raise.py: a finished phase cannot be edited or switched back on.	CI, "Each phase has its own ceiling"
provider <cloud> has a manifest but no entry in approved.json	A cost.json exists for a cloud that approved.json does not list.	CI, cost gate
<stack>: has OpenTofu files but no cost.json; every stack needs a manifest	A stack directory under stacks/ was added without a manifest, so the gate would have counted it as free.	CI, cost gate
no stacks/**/cost.json manifest found; at least one is required	The gate has nothing to total, which it treats as a failure rather than a pass.	CI, cost gate
<stack>: N twingate-connector module block(s) but M machine entr(ies) in its cost.json; they must match	The code and that stack's cost.json disagree about how many connectors exist. Adding a third module block fails it one way; adding a third manifest entry alone fails it the other way. Neither is "the ceiling".	Labs: Lab 3, part 2
module '<name>' uses count (or for_each); the connector is costed as a single instance	A repeated connector module would be priced once. The gate refuses it.	CI, cost gate
<manifest>: instance_of ... (missing, malformed, points into stacks/sites, names no stack directory with its own cost.json, or must sit at stacks/sites/<resource_key>/<stack>), or instance_of is only allowed under stacks/sites/	A stack instance manifest (a second site of the same code) that does not name an existing stack with its own manifest, or sits at the wrong path.	Cost, stack instances
<file>: instances carry no code; stacks/sites holds data (cost.json) only	A .tf or .tofu file was put under stacks/sites/. An instance reuses the shared stack's code and carries only its manifest.	Cost, stack instances
<manifest>: N twingate-connector module block(s) in <stack> but M machine entr(ies) in this instance manifest	The instance lists a different number of machines than the stack it is an instance of has connector modules.	Cost, stack instances
<manifest>: resource names must equal the base stack's or resource '<name>' is ... but must equal the base stack's entry	An instance manifest lists different resources, sizes or flags than the base stack whose code it deploys. The shared code reads the base manifest, so they must be equal.	Cost, stack instances
.spacelift/config.yml stack id <id> has N manifest(s), expected exactly 1 (or .spacelift/config.yml is unreadable ...)	A stack id was added to the Spacelift config without a manifest, or two manifests answer to it.	Cost, stack ids
duplicate stack id cah-...: <dirs>	Two manifests map to one Spacelift stack id (for example a base stack literally named 10-access-tor2 and the tor2 instance of 10-access).	Cost, stack ids
<provider>: site <key> has a manifest but is not in the approved sites map, site <key> planned $X is above its approved $Y, approved sites add up to $X, above the provider approved $Y, or base stacks planned $X are above approved $Y minus sites $Z	The provider in approved.json has an optional sites map and an instance does not fit it.	Cost, approving sites
module '<name>' has a non-local source	The gate reads only local modules. A remote source is something it cannot read, so it is refused.	OpenTofu, stacks and modules ("Only local module sources")
*.tf.json cannot be read by this gate and is refused	The gate parses .tf files with regular expressions and does not read the JSON form.	CI, "What the gate cannot do"
<stack>: the plan costs $X but cost.json declares $Y; they must match, or --plan needs --stack, or --plan costing is implemented for aws only	Only when the gate is run with --plan on a plan from the trusted run. The planned values disagree with the manifest, or the options are incomplete.	The module docstring at the top of scripts/cost_check.py

Module and contract test failures

These are pytest failures in tests/test_module_policy.py and tests/test_module_contract.py. They guard rules that the per-module tofu test files cannot see.

Failure message	What it means	Where to look
not run by CI (add to directories in ci.yml): [...]	A module or stack with .tf files is not in the directories: list of .github/workflows/ci.yml, so its tofu test would never run.	CI, tests and the cost gate
ci.yml lists directories with no .tf files: [...]	The list names a directory that was renamed or removed.	CI, tests and the cost gate
no provider module calls the shared bootstrap, or a failure about passing the shared boot data straight to the machine, a templatefile, a cloud-init.yaml.tftpl beside a provider module, or a twingate_ resource in a provider module	A provider's connector module must reuse the shared twingate-bootstrap module, which is the only place the cloud-init template and the Twingate token resources live.	OpenTofu, stacks and modules; Terraform reference
provider <name> lacks modules: [...]	A provider directory under modules/ does not supply every role of the provider module contract.	OpenTofu, stacks and modules
.../outputs.tf mentions 'user_data' (or custom_data, metadata_startup_script, token, bootstrap)	An output may expose boot data or a token, because the machine's user-data holds the Twingate tokens.	Twingate, "Accepted risk: the connector tokens"
<module>/README.md missing, role table has no rows, empty extensions section, or does not describe contract output ...	Each provider module's README must list its roles, describe every contract output and explain its provider extensions.	The failing README and the test's own docstring in tests/test_module_contract.py

Twingate connectors and SSH

Symptom	Likely cause	Where to look
A connector shows offline in the Twingate admin console (not online)	The install did not finish, the service is down, or the token was rotated. Try the other connector first, since SSH to a connector goes through Twingate.	Stage 1, section 6a; Twingate, "Verify it yourself"
systemctl status twingate-connector says "Unit not found"	The installer never finished.	Stage 1, section 6a (read /var/log/cah-connector-install.log and /var/log/cloud-init-output.log)
The twingate-connector package is installed, but there is no /etc/twingate/connector.conf and the service never started, yet the log said success	The old install piped a vendor script into bash, which can swallow the rest of the script, and the pipeline's exit status hid the failure. Cloud-init now writes the file itself and counts success only if the service is active.	Lessons learned, section 1
The install fails loudly on the repository key check	The pinned fingerprint no longer matches, which is what the check is for. Confirm the new fingerprint with Twingate before updating the pin.	Stage 1, section 2b
Both connectors are down	There is no way in, and there is deliberately no console password. The droplets are stateless, so the recovery is to rebuild the affected one through a Spacelift run and read the plan first.	Stage 1, section 6a; Twingate, "Two connectors, not one"
An apply fails because a Twingate site named access-<region> already exists, or you end up with two	A hand-made site with that name collides with the one OpenTofu now creates. Delete or rename the hand-made one before the apply.	Lessons learned, section 4
The stack refuses to plan with an empty admin list	ADMIN_USER_IDS is required and may not be empty, so a missing secret cannot create a host nobody can reach.	Twingate, "Access is a group, and the group is code"
SSH through Twingate works to connector a's address but times out in the banner exchange to connector b's	The connector firewall needs both an inbound rule and an outbound rule for TCP 22 to the connector tag. A new outbound connection needs an outbound rule even inside the VPC.	Lessons learned, section 2; DigitalOcean network and firewalls
SSH to a droplet added later times out	The baseline firewall has zero inbound rules, and that applies between droplets in the same VPC. A later droplet needs its own firewall with the connector tag as the source for TCP 22.	Stage 1, section 7
SSH to the public IP from the internet gets no answer	Expected. Nothing is open to the internet. The scan from outside should report ports 22, 80 and 443 as filtered.	Twingate, "Verify it yourself"; DigitalOcean network and firewalls, "Verify it yourself"
SSH is refused for root, or asks for a password, or the key is rejected	The only login is user cah-admin with the admin key: root login, passwords and keyboard-interactive are off, and AllowUsers names only that user.	Droplet hardening, "Login" and the sshd drop-in section below it

Droplet first boot and the DigitalOcean API

Symptom	Likely cause	Where to look
Intermittent package install failures on first boot	Ubuntu's own apt-daily and unattended-upgrades timers start on first boot too and hold the dpkg lock. The fix is DPkg::Lock::Timeout "300";, so apt waits up to five minutes.	Lessons learned, section 5; Stage 1, section 2a
Apply fails with 422 SSH Key is already in use	The stack tried to register a key that is already on the account. The stack looks the key up by name with a data source and never owns it.	Lessons learned, section 3
The plan fails at the SSH key lookup	ADMIN_SSH_KEY_NAME does not match the name the key already has on the DigitalOcean account.	Stage 1, section 5, step 3
422 Size is not available in this region	The cheap droplet size is not offered everywhere. Ask the API (GET /v2/sizes, GET /v2/regions) before choosing a region.	Lessons learned, "The cheap droplet size is not offered everywhere"
403 Can not delete default VPCs, or 422 This range/size overlaps with another VPC network	The first VPC in a region becomes its default and cannot be deleted, and ranges may not overlap anywhere in the account, even across regions.	Lessons learned, "A default VPC cannot be deleted"; Stage 0, section 3
A rotated connector token destroys and recreates the droplet in the plan	The token is in the cloud-init user-data, and a user-data change forces a new droplet. That is expected: read the plan, then confirm.	Twingate, "Rotating a connector token"; Stage 1, "Accepted residual risk"

Wiki pages and link-checker failures

Symptom	Likely cause	Where to look
The Wiki check fails on a pull request that touches docs/wiki/	That check runs scripts/gen-wiki.sh on the pages and fails if generation or the link rewrite breaks. Run the script locally against an empty directory and read its error.	CI, "The wiki is published by a workflow too"
A link works in the repository but not in the published wiki	Page links are written as (Page-Name.md#anchor) and the generator rewrites them to (Page-Name#anchor). A link written any other way, such as an absolute path or a missing .md, is not rewritten.	CI, "The wiki is published by a workflow too"; tests/test_gen_wiki.py
A link lands on the right page but at the top, not at the section	The anchor does not match a heading. The anchor is the heading in lowercase, with punctuation removed and spaces turned to hyphens. Wording changes to a heading break every link to it.	Git, GitHub and pull requests, "Sending a correction to this wiki"
A page you edited in the wiki editor disappeared	docs/wiki/ is the only place a page is written. A page that exists only in the wiki is removed on the next publish.	CI, "The wiki is published by a workflow too"

Predict, then check

Do this before you need the table.

Predict. Pick the row you believe is the most likely to hit you first. Without opening the repository, write down three things: the exact message you would see, which layer prints it (your script, DigitalOcean, tofu test or Spacelift), and what you would look at first.

Check. Now reproduce one that is safe to cause on your own machine. The cost gate is the easiest, because it needs no account:

1. From the repository root, run python3 -I scripts/cost_check.py --root . and note the baseline lines (digitalocean: planned $12.00/mo, approved $12.00, active and phase stage-1 ceiling $30.00 (reason: ...)).
2. In stacks/10-access/cost.json, copy one of the two droplet entries to make a third, without touching any .tf file. Before you run the gate, predict which rows of the cost gate table it will trip, what the planned total will be, and whether a summary line will be printed at all.
3. Run the gate again and compare. Two rules should object, for two different reasons. Then find the same message in scripts/cost_check.py and read the condition that prints it.
4. Restore the file with git restore stacks/10-access/cost.json and confirm git status is clean.

If your prediction named the wrong rule or the wrong total, write one sentence on what you assumed. The labs do the same for the firewall, the VPC range, the push policy and the secrets hook.

Back to Lessons learned, the Glossary or the Reading order.