v0.18.1 — "gateway offline" wasn't the gateway, and the memory backups had never run

Fixed — memory backups had never run under systemd

memory-backup.service exited 127 on every firing: its Environment=PATH omitted
/snap/bin, and on the Ubuntu GCP image gcloud is a snap living only at
/snap/bin/gcloud (there is no /usr/bin/gcloud — verified 2026-09-06). memory-backup.sh
calls gcloud storage rsync, so every timer firing failed and the state backups silently
did not happen. /snap/bin appeared nowhere in the install package.

What kept it hidden: running the script by hand works. An interactive login shell has
/snap/bin on PATH; systemd's does not. Every manual verification of this script has
therefore always passed, while the scheduled path never once succeeded.

Fixed in systemd/memory-backup.service and applied to the live VM; the unit now exits
0/SUCCESS under systemd and the failed-unit list is empty. It is the only packaged
script affected — memory-backup.sh is the only one in the install that calls gcloud.

Not fixed here, but worth knowing: hermes-dashboard.service omits /snap/bin too, and it
hosts the agent runtime. Commands the agent shells out to inherit that PATH, so an agent
turn doing GCP work would not find gcloud on PATH either. Left alone deliberately —
widening the runtime's PATH is a behaviour change on a 24/7 install, not a bug fix, and
nothing has reported it. Noted so the next person recognises the same shape.

How long backups had been failing is undetermined: the diagnostic run that found this
also rsynced the tree, overwriting the bucket timestamps that would have dated it. Known
good: failing at the 20:01:35 UTC firing, succeeding under systemd afterwards.

Documented — the gateway failure that is not the gateway's fault, and not credentials

Hit live 2026-09-06, on Hermes v0.21.0, and initially read as the 0.16.2 credential
lapse because every local signal is identical. It is not. The tunnel, the credentials, the
service account, the firewall and the VM were all healthy throughout. The agent itself had
stopped answering, and no Mac-side action could have fixed it.

Ruled out with evidence, not assumption: the tunnel hangs identically under user creds and
service-account creds; a probe from inside the VM also returns 000; there was no OOM
kill and 12.9–13.7 GB of 16 GB stayed free throughout (it was not memory); and
hermes-autoupdate last ran 13 hours earlier.

The signature: curl localhost:9119 on the VM returns 000 while ss -ltn still
shows LISTEN — and the Recv-Q column on that LISTEN row climbs (19 → 44, live) because
those are completed TCP handshakes the application never accept()ed.

Root cause — catastrophic regex backtracking (ReDoS) in the agent's approval guard. The
launchd gateway-lifecycle pattern at tools/approval_detection.py:337 is unanchored and
both of its branches are lookaheads starting with [\s\S]*, so re.search retries at all
N start positions and rescans to end each time: O(N²). When the command contains no
launchctl — the normal case — nothing short-circuits and the full quadratic cost is paid on
every ordinary command. Python holds the GIL inside re, so one thread starves the whole
process, uvicorn's accept loop included. Captured with py-spy: one thread active+gil in
detect_dangerous_command, 47 threads parked in futex_do_wait, 13m34s of CPU in a
single search().

Measured against the exact pattern — the last row is the largest input the guard allows:

command size	one pattern, one variant
22 KB	1.2 s
44 KB	4.9 s
89 KB	19.7 s
127 KB	39.6 s

The existing size guard (128_000 chars / 4_096 separator-free / 25_000 separators) does
not help: a python3 - <<'EOF' heredoc that writes an HTML page is ~10–127 KB with a few
thousand newlines and passes all three. That is exactly how it was triggered — twice,
reproducibly, by an agent turn generating a 90 KB HTML page.

Upstream defect, not an install fault. The pattern is in upstream 6b2d4faf at the
same line; the install's single carried commit is unrelated. Do not hand-patch the VM —
hermes-autoupdate overwrites it. Report upstream.

Operational fallout this exposes: the 0.16.2 tunnel supervisor does its job correctly and
still misleads, because its HTTP probe cannot distinguish a broken tunnel from a wedged
agent — both are HTTP 000 at localhost:9119. It restarted a healthy tunnel every ~65s
for hours. And OPS-NOTES.md opened this symptom with "Check credentials first", which is
right for one cause and a dead end for the other.

• OPS-NOTES.md gains "The gateway wedges with the port still open (remote side)" —
  the VM-side probe that separates the two causes, the Recv-Q tell, a table of which
  checks lie, the measured numbers, the py-spy recipe for capturing the specimen before a
  restart destroys it, and the avoidance rule (write large files with a file-writing tool,
  not a giant heredoc; the exposure window is ~4 KB–128 KB with a newline).
• The existing credential section now starts by splitting local from remote, instead of
  sending you to gcloud auth login for a fault that has nothing to do with credentials.
  The split is read in three branches, not two: the diagnostic gcloud compute ssh
  authenticates with your user credentials, so whether the SSH succeeds at all is the
  first signal — a genuine credential lapse makes that command fail outright rather than
  return a discriminating 302. (And since 0.17.1 the tunnel authenticates as a service
  account, so your user credentials expiring does not imply the tunnel is down.)
• Recovery is systemctl --user restart hermes-dashboard.service — the dashboard unit
  owns 9119 (hermes-gateway.service is a different process; restarting it does nothing).
  The Mac needs no action: the supervisor reconnects itself in ~35s.

Not yet validated

• ⚠️ The memory-backup.service PATH fix has not been rebuilt from a virgin install. It
  was verified on the live 24/7 VM (unit patched, daemon-reload, run under systemd, exit
  0/SUCCESS, failed-unit list empty) — which is exactly the "re-run over a live VM" the
  from-scratch rule in AGENTS.md says not to sign off on. The change is a one-token PATH
  append, but per that rule it wants a teardown → 01 → 02 → 03 pass before it counts as
  proven. The ReDoS half of this entry is a diagnosis and doc change; it installs nothing.

v0.18.0 — the weekly update stops failing at itself, and teardown stops eating the live install

Fixed — the weekly update reported failure every week while succeeding every week

hermes-autoupdate.service sat in failed for five days. The updates themselves were
working: the 2026-08-30 run installed Hermes v0.20.6, restarted the gateway and the
dashboard, and passed 13 checks including shim chat, embeddings and Honcho end-to-end
recall. It then exited 1 on two checks — both of which assert on the autoupdate's own
state, and are circular when the autoupdate is what is running them. Both were
introduced by check 14 in 0.15.0.

1. The latch. 03-verify.sh failed if ~/.hermes/autoupdate/last-failure exists — and
the updater writes that marker when verification fails. So the first bad Sunday
(2026-08-23) guaranteed every later Sunday failed on the marker alone, wrote the marker
again, and never reached the rm -f last-failure that a success performs. Self-sustaining,
and invisible: the box kept updating.

Note a timestamp comparison does not fix this — once latched, last-failure is always
newer than last-success. The updater is about to record this run's outcome, so its
previous outcome is not evidence about the install. It is now skipped when the updater is
the caller, and still a hard FAIL for a human or hermesctl, which is who that warning is
for.

2. The timer check could not pass. The check failed when hermes-autoupdate.timer has
no next elapse — but while hermes-autoupdate.service is executing, its own timer
legitimately has none. Guaranteed to fail from inside the update, and says nothing about
the install.

hermes-autoupdate.sh now passes HERMES_VERIFY_FROM_AUTOUPDATE=1, and 03-verify.sh
gained a skip() state so the count stays honest instead of quietly passing. Deliberately
an env marker rather than "is the service active" — a standalone run during a concurrent
update would wrongly skip a check that should pass.

Added — teardown.sh refuses to destroy a RUNNING install

The typed confirmation was never a barrier to an automated caller: it reads stdin, so
echo "${VM_NAME}" | teardown.sh sails through it. That is how destructive code gets
"tested" against a live project.

What it cost, on 2026-09-04: the VM's service account was deleted at 12:31:11 UTC and
recreated 21 seconds later while teardown changes were being validated. The recreated
account has the same email but a new unique id, and a GCE instance is bound to the
id — so the metadata server returned 401 "Service account is deleted or disabled."
indefinitely, agent.vertex_adapter could not resolve credentials, and every turn failed
with "agent init failed" while the desktop app blamed Vertex. The VM never stopped; the
identity that made it useful was gone.

teardown.sh now refuses when the target instance is RUNNING and requires
--yes-destroy-live, naming the safe alternatives (a throwaway PROJECT_ID/VM_NAME, or
stopping the VM first). Verified both directions: the exact echo hermes-agent | teardown.sh invocation is refused with exit 1 (also with --vm-only), and an absent or
non-running target reaches the confirmation prompt exactly as before.

Recovery, for the record: gcloud iam service-accounts undelete <ORIGINAL_UNIQUE_ID>
restores the identity the instance is still bound to, so the metadata server works again
with no stop/start — the email must be freed first by deleting the replacement.
gcloud compute instances set-service-account is the alternative and needs a stopped VM.

Verified — re-probed live, 2026-09-04

Check	Result
03-verify.sh standalone	14 passed, 0 failed
03-verify.sh as the updater runs it, against the real latched marker	14 passed, 0 failed, 1 skipped (was: 1 failed, forever)
Vertex :generateContent @ global	gemini-3.8-flash, 3.7-flash, 3.5-flash — all HTTP 200
Hermes on the box	v0.21.0 (2026.8.31) — badge and CLAUDE.md said v0.20.4/v0.20.5
Gateway through the tunnel	HTTP 302

The 2026-08-18 from-scratch rebuild block in gcp/vpc-install/README.md is a dated
historical record and was left as written; only the re-probe lines were updated. Same
discipline the 0.16.1 entry had to correct after a blanket find/replace rewrote history.

v0.17.3 — README: what to do when you can't connect

Added — README: what to do when you can't connect

0.17.1 split the two credentials — the tunnel moved to the hermes-tunnel service account
while hermesctl kept running as the operator — and 0.17.2 explained the split in
OPS-NOTES.md §7a. But the README, which is the front door and the only page most people
read, still implied a single answer. The practical consequence: gcloud auth login is now
the fix for one of the two and does nothing for the other, and there was no
front-page guidance on telling them apart.

New "When you can't connect" section under Everyday commands, built around the one
check that distinguishes the cases — curl localhost:9119, because a bound-but-not-
forwarding tunnel looks alive to every process- and port-based check:

• 302 — tunnel healthy; if the app still fails it is the app or the password, and
  gcloud auth login will not help.
• 000 — tunnel not forwarding; wait, then launchctl kickstart, then read the log.
• Reauthentication failed from hermesctl while the app works — that is the split,
  not a broken tunnel.

Corrected

• Documented that a cold tunnel start has taken over two minutes to first answer.
  Observed repeatedly on 2026-09-04 while verifying 0.17.1. This matters because the
  obvious reaction to a slow start is to conclude the tunnel is dead and start
  re-installing — the README now says to wait first. The supervisor's 45s threshold applies
  to a running tunnel that stops forwarding, not to first connect.

v0.17.2 — teardown now removes the tunnel service account, and the auth advice stops contradicting itself

Fixed — teardown left the tunnel service account behind

0.17.1 added the hermes-tunnel service account and a local key, but teardown.sh was
never taught about either. A teardown-then-rebuild — this repo's own mandated validation
path — therefore left:

• an orphan service account still holding roles/iap.tunnelResourceAccessor on the
  project, invisible in gcloud compute instances list, so the "confirm it really is virgin"
  check passed while it was still there; and
• a stale key file on the Mac, which the installer's "Reusing existing tunnel key" branch
  would have handed straight to launchd — a key whose service account no longer exists.

Teardown now deletes the tunnel SA alongside the VM's, removes ${TUNNEL_SA_KEY} locally,
and lists both in the typed-confirmation preview so nothing is destroyed unannounced. Found
by review immediately after 0.17.1 shipped, not by a rebuild — the virgin-install rule exists
because this is exactly the class of defect that hides from incremental re-runs.

Fixed — gcloud auth login guidance contradicted itself

0.17.1's new §7a says gcloud auth login is not the fix for a dead tunnel; 0.16.2's entry
says an expired credential breaks all of hermesctl and to run exactly that. Both are
true — the tunnel moved to a service account, hermesctl did not — but side by side they
read as a contradiction. §7a now states the split explicitly: the desktop app and the CLI
now fail independently, and Reauthentication failed from hermesctl while
curl localhost:9119 returns 302 is that split, not a broken tunnel.

Corrected

• teardown.sh still told the operator the rebuild "must be 13/13". The suite grew to
  14 checks in 0.15.0. Same stale-count class as the 0.16.1 README defect.

Verified

• constraints/iam.serviceAccountKeyExpiryHours on test-disco-cm is allowAll — no
  key-expiry limit is enforced, so 0.17.1's "survives a week of idle" has no hidden shelf
  life from org policy. Probed 2026-09-04 via the Org Policy API.

v0.17.1 — the gateway tunnel authenticates as a service account, so idle stops killing it

Fixed — the gateway tunnel now authenticates as a service account, so idle no longer kills it

Two faults, hit together on 2026-09-04. The desktop app showed the same "Could not reach
this gateway yet" that 0.16.2 is about, from two causes that 0.16.2 did not fix.

1. The installer wrote a LaunchAgent pointing into a git worktree

0.16.2's supervisor, its plist and two commit messages were all written for a stable
~/.local/bin install. The one line in install-gateway-launchagent.sh that chooses the
path was never changed — it still passed ${HERE}, the checkout it was run from. So the
installed plist referenced
.claude/worktrees/hermes-flash-upgrade-test-44506f/…/gateway-tunnel-supervisor.sh; that
worktree was reset to main, the script vanished, and launchd could no longer exec it:

Check	Says	Reality
launchctl list | grep hermes	listed	agent is loaded
last exit status	-15	never actually ran
lsof -iTCP:9119	nothing	no listener at all
~/Library/Logs/…tunnel.log	stops at 14:07	died when the file disappeared

• The installer now install -m 0755s the supervisor to
  ~/.local/bin/hermes-gateway-tunnel-supervisor.sh and points the plist there.
• It also refuses to write a plist that contains its own checkout path (grep -qF "${HERE}"), the same shape as the existing unsubstituted-__TOKEN__ guard, so this
  cannot regress silently.
• Verified the way it actually failed, not from the source tree: copied the package to a
  foreign directory, installed from there, deleted that directory, restarted the agent
  — gateway still served HTTP 302. The original failure is now unreachable.
• The 0.16.2 entry claimed this was "caught before it could bite". That claim has been
  corrected in place. A commit message is not verification, and — for the second time
  after the 0.16.1 hermesctl PATH defect — exercising a script from its own checkout
  does not test how it is installed.

2. THE REAL ONE: a LaunchAgent can never satisfy a reauth prompt

Underneath the missing script, the log showed the tunnel had been refusing to start for
hours, once every 30 seconds:

Reauthentication failed. cannot prompt during non-interactive execution.

That is Google Cloud's periodic reauthentication requirement on user credentials.
01-gcp-setup.sh grants roles/iap.tunnelResourceAccessor to the human operator, and
the tunnel then runs unattended as that human. No supervisor, timeout, retry or
KeepAlive can fix this — reauth is interactive by definition, and a daemon has no one to
prompt. 0.16.2's supervisor diagnosed it correctly and could do nothing about it. The
requirement — "it needs to work after a week of idle" — was structurally unmeetable.

Fix: the tunnel gets its own service account. Service-account credentials are exempt
from reauth.

• TUNNEL_SA_NAME / TUNNEL_SA_EMAIL / TUNNEL_SA_KEY / TUNNEL_USE_SA in
  00-vars.sh. 01-gcp-setup.sh creates hermes-tunnel and grants it
  roles/iap.tunnelResourceAccessor and nothing else — no osLogin, no Vertex, no
  storage. It is deliberately not the VM's hermes-agent SA, whose roles would be
  wildly over-granted for a laptop.
• The key is minted on the Mac by install-gateway-launchagent.sh under umask 077
  (never briefly world-readable), chmod 600, and the plist passes it as
  CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE. It is .gitignored (*-sa.json) and never
  committed. This is the one key file in an install that otherwise deliberately has none
  — the reasoning is written out in 00-vars.sh next to the variables.
• The installer verifies the credential before handing it to launchd, so a bad key fails
  loudly at install time instead of becoming a dead gateway hours later.
• The supervisor no longer prints the wrong advice. Under a service account there is
  nothing for a human to log into, so a credential failure now names the real causes (key
  missing, revoked, or lost its IAM binding) instead of suggesting gcloud auth login.
• TUNNEL_USE_SA="false" restores the old operator-credential behaviour, and the installer
  says plainly that the tunnel will then stop at every reauth window.

Verified that it no longer depends on the human login at all — the only test that
actually proves the week-idle claim. With CLOUDSDK_CONFIG pointed at an empty
directory, so gcloud auth list reports no credentialed accounts whatsoever:

Test	Result
gcloud auth print-access-token with only the SA key	token minted
full start-iap-tunnel → curl localhost:9219	HTTP 302
live agent's plist env	CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE set
live gateway on :9119	HTTP 302

Corrected while verifying

• A freshly created SA key is eventually consistent. The first version of the installer's
  own verification failed immediately after keys create and told the operator to delete a
  perfectly good key; by hand, seconds later, it minted a token fine. It now polls for up to
  30s — the same trap 01-gcp-setup.sh already documents for SA creation.
• With TUNNEL_USE_SA=false the plist would have carried an empty
  CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE, which is worse than an absent one (gcloud would
  try to load ""). The installer now removes the key outright in that case.

v0.17.0 — default chat model to `gemini-3.8-flash`

Changed

• Default chat model is now gemini-3.8-flash (was gemini-3.7-flash). The catalog
  is {google/gemini-3.8-flash (default), google/gemini-3.5-flash}; gemini-3.5-flash
  remains the switchable, strict-EU-capable fallback. Only HERMES_MODEL /
  HERMES_MODELS in 00-vars.sh change — config.yaml is templated from them, and
  03-verify.sh derives the probe target from $HERMES_MODEL, so nothing else moved.

• VERTEX_REGION stays global, and the EU-residency exception is unchanged.
  gemini-3.8-flash has exactly the same availability shape as 3.7 and 3.6: 404 at
  every European regional endpoint, 200 only at global. This upgrade neither widens
  nor narrows the residency posture. Re-probed 2026-09-04 (:generateContent POST):

  Model	eu-w1	eu-w2	eu-w3	eu-w4	eu-n1	global
  gemini-3.8-flash	404	404	404	404	404	200
  gemini-3.7-flash	404	404	404	404	404	200
  gemini-3.6-flash	404	404	404	404	404	200
  gemini-3.5-flash	404	200	200	404	404	200
  gemini-3.5-flash-lite	404	404	404	404	404	200
  gemini-2.5-flash	200	200	200	200	200	200

  Every row other than 3.8 is byte-identical to the 2026-08-18 probe — the run reproduced
  the known-good result, which is what makes the new row trustworthy.

• Cost estimate revised down while introductory pricing lasts. gemini-3.8-flash is
  $0.75 / $3.75 per 1M input / output tokens at global through 2026-12-31, then
  $1.50 / $7.50 from 2027-01-01 — the latter being exactly what 3.7-flash costs
  today. The token line moves $75–180 → **$40–90**, taking the line-item sum from
  $234–367 to $199–277 (tabled as ~$200–275, matching how the previous estimate
  rounded). On 2027-01-01 the model line doubles back with no action required.
  Non-global endpoints carry a ~10% premium. Source: the
  Vertex AI pricing page,
  read 2026-09-04.

Verified

• The model id is real, proved the strong way. The global 200 echoed
  "modelVersion": "gemini-3.8-flash", matching the requested id — a 200 alone would not
  have settled it. Negative control run alongside: gemini-3.9-flash,
  gemini-3.8-flash-lite and gemini-3.8-pro all 404 at global, so 3.8 currently
  ships as a flash tier only, with no -lite or -pro sibling on Vertex.

Fixed

• The repo-map row in README.md still read "Currently 0.14.3" while the version
  badge four lines up read 0.16.1 — stale since 0.14.4. Now tracks the badge.

• INSTALL.md §2 and gcp/vpc-install/README.md both stated Honcho runs on
  gemini-3.5-flash. It does not, and must not — 00-vars.sh sets
  HONCHO_MODEL=google/gemini-2.5-flash, and §7 of the same document explains at length
  why any Gemini 3.x model 400s every dialectic query (dropped thought_signature). The
  summary tables contradicted the source of truth they were summarising. Both corrected.

Not yet validated

• ⚠️ This has not been rebuilt from a virgin install. The 3.8 evidence is an API fact,
  not an install fact: as of 2026-09-04 the live box still runs 3.7-flash, and neither
  03-verify.sh nor the live tool-calling turn has been re-run on 3.8. A
  :generateContent 200 proves the model answers — it does not exercise the
  thought-signature round-trip that tool-calling depends on, which is the failure mode
  that already bit HONCHO_MODEL. Per the from-scratch rule in AGENTS.md, run
  teardown.sh → full install → 03-verify.sh → the OPS-NOTES tool-call check before
  treating 3.8 as validated. HONCHO_MODEL is untouched and stays on gemini-2.5-flash.

v0.16.2 — gateway tunnel: supervise it, because KeepAlive structurally cannot

Fixed — the gateway failure that reports itself as healthy

Hit live on 2026-08-19, and initially misattributed to an unrelated upgrade. The desktop
app showed "Could not reach the remote Hermes gateway while refreshing its WebSocket
ticket" and gateway offline; Settings → Connection mode said "Could not reach this
gateway yet. Check the URL — the auth method will appear once it responds."

Cause: expired gcloud credentials. The tunnel process stays alive and keeps port 9119
bound; it simply cannot forward, because gcloud can no longer mint a token. That makes it
pathologically misleading:

Check	Says	Reality
launchctl list | grep hermes	status 0	process is running
lsof -iTCP:9119 -sTCP:LISTEN	bound	accepts local connections
curl localhost:9119	HTTP 000	← the only honest check

The app connects at TCP level, then fails refreshing its WebSocket ticket — exactly what the
message says. Every process- or port-based liveness check reports HEALTHY. Same lesson as
03-verify.sh test 13: liveness is not correctness.

The real defect: the tunnel could never recover, even after you re-authenticated

gcloud compute start-iap-tunnel binds the local port first, and when its token later
fails to refresh it does not exit — it retries internally, forever, holding the listener
open. Caught in the act 2026-09-02: one process alive 1d 9h, emitting ~1.5 auth errors
per second, having written a 27 MB / 413,000-line log.

KeepAlive only restarts a process that dies, so it never fired. And because the stuck
process never re-reads credentials, gcloud auth login did not fix it — only a manual
launchctl kickstart did. That is why the failure kept coming back and why it was
repeatedly misattributed to whatever had changed most recently.

• scripts/gateway-tunnel-supervisor.sh (new) — the LaunchAgent now runs this instead
  of gcloud directly. It probes the tunnel over HTTP (the only check that separates
  forwarding from listening) and kills and restarts the child after 45s of no
  forwarding. When credentials have genuinely expired it stops the child, waits, and posts a
  macOS notification naming the one command a human must run — then reconnects on its own
  once you do. Verified by SIGSTOPing the child to reproduce the exact zombie shape
  (port bound, HTTP 000): detected and recovered in ~35s with a new child.

• Logs moved out of /tmp to ~/Library/Logs/ and are rotated (2 MB cap). The previous
  setup pointed gcloud's raw stderr at a file nothing rotated, which is how it reached 27 MB.

• THE ACTUAL DEFECT: the tunnel could never recover, and KeepAlive could not help.
  gcloud compute start-iap-tunnel binds the local port first, and when its token later
  fails to refresh it does not exit — it retries internally, forever, holding the
  listener open. Measured 2026-09-02: one such process had been alive 1d 9h logging ~1.5
  Reauthentication failed errors per second, having written a 27 MB / 413k-line
  log. launchd's KeepAlive only restarts a process that dies, so it never fired — and
  since the stuck process never re-read credentials, gcloud auth login did not fix it
  either. Only a manual launchctl kickstart did. That is why this failure kept recurring
  and kept looking like something else had broken.

  Fix: scripts/gateway-tunnel-supervisor.sh, which the LaunchAgent now runs instead of
  gcloud directly. It probes the tunnel over HTTP — the only check that distinguishes
  forwarding from listening — and kills and restarts the child after 45s of not forwarding.
  A credential lapse now self-heals the moment you re-authenticate, and while it is waiting
  it sends a macOS notification naming the one command a human must run. It also rotates its
  own log, and logs moved from /tmp to ~/Library/Logs/.

  Verified by SIGSTOP-ing the child to reproduce the exact zombie shape (port bound,
  HTTP 000): detected and recovered in ~35s with a new child.

• The supervisor is installed to ~/.local/bin/, not referenced in the checkout. The
  first version pointed launchd at the script inside the repo working tree — and this repo's
  own workflow uses temporary per-agent git worktrees, so cleaning one up deletes the
  running supervisor and kills the gateway, failing in a way that looks exactly like the
  credential fault it exists to fix. The installer now install -m 0755s it to
  ~/.local/bin/hermes-gateway-tunnel-supervisor.sh and passes VM_NAME / ZONE /
  PROJECT_ID / DASHBOARD_PORT through the plist's EnvironmentVariables, because the
  installed copy has no 00-vars.sh beside it to source.

  It bit before this was true. An earlier revision of this entry claimed the move was
  "caught before it could bite, on 2026-09-02". It was not: the supervisor, the plist and
  two commit messages were all written for ~/.local/bin, but the one line in
  install-gateway-launchagent.sh that chooses the path was never changed — it still
  passed ${HERE}. So the installed plist pointed at
  .claude/worktrees/hermes-flash-upgrade-test-44506f/…, that worktree was reset to main
  on 2026-09-04, the script vanished, and launchd could no longer exec it: agent loaded,
  exit -15, nothing listening on 9119, no log. The desktop app reported the same
  "Could not reach this gateway yet" this entry is about — from a completely different
  cause.

  Two lessons, both already in this repo's rules and both ignored here: a commit message
  is not verification (three artifacts described the fix; the code did one thing), and
  exercising a script from its own checkout does not test how it is installed — the same
  shape as the 0.16.1 hermesctl PATH defect. The installer now also refuses to write a
  plist that references its own checkout, so this cannot regress silently.

• hermesctl now fails fast with the real fix. Every VM-side command goes over the IAP
  tunnel, so an expired credential breaks all of it — and breaks it confusingly, because
  gcloud compute ssh returns 255, which vm()'s retry loop treats as transient and
  retries 3× before giving up on something no retry can fix. A require_creds preflight now
  runs once per invocation and prints the two-command fix.

• gateway-tunnel.sh --status diagnoses instead of reporting up/down. It separates
  "nothing listening" (tunnel not running) from "listening but HTTP 000" (running, not
  forwarding), greps the log for TokenRefreshError, and names the fix. Verified against the
  real broken state — it identified the cause correctly.

• install-gateway-launchagent.sh names the cause (credentials vs. port conflict)
  instead of dumping 20 log lines.

• Documented in OPS-NOTES.md and INSTALL.md, including that this recurs by design:
  Workspace reauth policies expire the credential on a schedule, so a permanently-installed
  LaunchAgent will meet it periodically. Not a fault in the install, and not upgrade-related.

The fix, for the record:

gcloud auth login                                              # interactive, needs a browser
launchctl kickstart -k gui/$(id -u)/com.hermes.gateway-tunnel   # pick up the new token

Confirmed on the live install: recovered to HTTP 302 in ~6s.

---

• Log moved out of /tmp to ~/Library/Logs/hermes-gateway-tunnel.log, and the
  supervisor rotates it (2 MB cap). The old setup pointed gcloud's raw stderr at a
  /tmp file that nothing rotated, which is how it reached 27 MB / 413k lines of the
  same repeated auth error. All five references across INSTALL.md, OPS-NOTES.md and the
  plist's own instructions were updated to the new path — a stale tail -f in a runbook is
  worse than none, since it shows an empty file and looks like "no errors".

---

• The supervisor and the plist had drifted out of step, and the pair as committed could
  not start. 762a5ad moved the installed supervisor to ~/.local/bin (correctly — a
  LaunchAgent must not reference a git worktree), but two halves were left behind: the
  supervisor still did source "${HERE}/../00-vars.sh" unconditionally, and the plist
  template passed none of that config. In ~/.local/bin there is no 00-vars.sh, so
  VM_NAME was unset and the script died instantly under set -u — the gateway would
  never come up from a clean install.

  This is the same shape as the fresh-install defects in 0.13.0: it works in a checkout,
  because a checkout does have 00-vars.sh beside the script, and only fails once
  installed somewhere else. Fixed by making config arrive from the environment (the plist
  now passes VM_NAME, ZONE, PROJECT_ID, DASHBOARD_PORT), with 00-vars.sh used
  only as a fallback when the script is run straight out of a repo. A missing config now
  produces ERROR: VM_NAME/ZONE/PROJECT_ID not set and no 00-vars.sh beside this script
  instead of a bare unbound variable.

  Verified three ways: the isolated script with no config errors clearly; with config in
  the environment it starts and launches its child; and a full
  install-gateway-launchagent.sh run renders every placeholder, passes plutil -lint,
  and reports the gateway UP on HTTP 302.

v0.16.1 — fix hermesctl when run via the PATH (the only way it is used)

Fixed

• hermesctl failed for every invocation via the PATH — i.e. the only way anyone
  actually runs it. install-hermesctl.sh symlinks it into ~/.local/bin, and the script
  located 00-vars.sh with dirname "${BASH_SOURCE[0]}", which resolves to the
  symlink's directory. So it looked for ~/.local/bin/../00-vars.sh and exited with
  "cannot find 00-vars.sh". It only worked when run by its full path inside the repo,
  which is how it was tested. Now walks the symlink chain by hand — not readlink -f,
  which is GNU-only on older macOS.

  Testing lesson: exercising a script from its source directory does not test the way
  users invoke it.

v0.16.0 — hermesctl: one command for every routine operation

Adds hermesctl — one command for every routine operation, so day-to-day running of
the agent no longer means remembering gcloud compute ssh --tunnel-through-iap strings.

Added

• scripts/hermesctl (run on your PC) and scripts/install-hermesctl.sh to put
  it on your PATH. It reads 00-vars.sh, so a cloned install works with no edits.
  • health — status (the 14-point check), doctor,
    logs gateway|dashboard|shim|autoupdate|honcho|searxng
  • updates — update (server now), update-desktop (this Mac), update-all,
    update-check, autoupdate [show|off|on]
  • services — gateway status|start|stop|restart|kick, dashboard restart,
    restart-all
  • access — tunnel, open, ssh
  • machine — vm status|start|stop, disk
• README "Everyday commands" section covering the whole surface, and the split
  between updating the server (automatic, weekly) and the desktop app (manual, because it
  has no auto-update feed).

Design notes

• hermesctl update runs the autoupdate unit, not hermes update directly. That
  way a manual update gets the same cgroup isolation, dashboard restart and post-update
  verification as the Sunday run — rather than being a second, subtly different code path
  that could reintroduce the "updater killed by the restart it triggered" bug (0.15.0).
• gateway kick exists because restart cannot fix a wedged gateway. systemd only
  restarts a process that exits; a gateway hung on a stalled tool call stays active
  forever. kick stops it, clears stale gateway.lock/gateway.pid, and starts clean.
• vm stop confirms before acting and says what it costs: stopping halts the agent,
  cron and the weekly update, while the disk and Cloud NAT keep billing.
• Transient exit 255 from gcloud compute ssh is retried up to 3 times. Every
  wrapped command is idempotent, so a retry is always safe.
• restart-all restarts the Vertex shim first, because Honcho's first memory call
  fails if the shim is not up.

Fixed

• update-check reported the hint instead of the answer. hermes update --check
  prints its verdict and then Run 'hermes update' to install.; taking the last line
  showed that instruction even when the install was already current. Now picks the line
  that actually says N commits behind or up to date.

Verified

Every command exercised against the live VM: vm status → RUNNING; autoupdate →
timer armed for Sun 2026-08-23 04:09:15 UTC, last ok 2026-08-22T09:43:02Z, no
failures; gateway status → active, 13 min uptime; open → tunnel up; update-check →
9 commits behind origin/main on both server and Mac.

v0.15.0 — weekly backend autoupdate via systemd timer (+ install-blocking `hermes version` fix)

Adds automated weekly backend updates via a systemd timer, and documents the
update path for the desktop app — which is a separate, manual job.

Added

• scripts/hermes-autoupdate.sh + hermes-autoupdate.{service,timer} — weekly
  unattended hermes update on the VM, Sunday 04:00 UTC with a 30-minute randomised
  delay. Each run: --check (exits quietly when there is nothing to do) → hermes update --yes (keeping Hermes' own pre-update backup) → restart the dashboard →
  03-verify.sh, failing loudly if the new code does not pass. State markers in
  ~/.hermes/autoupdate/.
• AUTOUPDATE_ENABLE / AUTOUPDATE_MODE / AUTOUPDATE_SCHEDULE in 00-vars.sh.
  AUTOUPDATE_MODE=check reports without installing, for a human-in-the-loop
  production agent.
• 03-verify.sh check 14 — asserts the timer is not just enabled but has a real
  NextElapse, because a timer with a malformed OnCalendar loads happily and then
  never fires, which is indistinguishable from "updates are working". Also fails if
  ~/.hermes/autoupdate/last-failure exists, so a broken update cannot rot silently.
  Verification target is now 14/14.
• OPS-NOTES.md §11 — the full update story: automated backend, why a timer rather
  than hermes cron, the manual desktop-app path, how to make it check-only or turn it
  off, and recovery when an unattended update breaks the install.

Why a systemd timer and not hermes cron

Established by reading v0.20.5 on the live VM, not inferred:

• hermes update --plan reports exactly one service to restart —
  gateway [default] … systemd, restart: systemctl restart.
• A hermes cron job executes inside that gateway process, and the gateway unit sets
  KillMode=mixed plus an ExecStopPost cgroup cleanup. So an update scheduled as a
  cron job is reaped by the restart it triggers — it destroys its own runtime mid-run.
• The same trap catches updates launched from the desktop app or dashboard: the
  spawned updater is a child of the gateway. This is the real cause of in-app updates
  reporting failure.
• A timer unit has its own cgroup and is unaffected. No Hermes code change and no
  systemd-run --scope wrapper needed — the isolation is free once the updater is not
  launched from the service being restarted.

Fixed

• INSTALL-BLOCKING: 02-vm-install.sh died at step 3 on every current Hermes. It
  called hermes version, which was removed as a subcommand — v0.20.5 answers
  hermes: error: argument command: invalid choice: 'version' and only accepts
  --version. Because that call is guarded by || { …; exit 1; }, the installer aborted
  with "ERROR: hermes not on PATH after install" on a perfectly good install, and no
  amount of re-running helped. Found by applying this release to the live VM — a fresh
  clone today would have hit it immediately. Now tries --version first and falls back to
  the old subcommand for older pinned installs. 03-verify.sh check 1 and the
  OPS-NOTES.md snippets had the same stale form and are fixed too.
• hermes update leaves the dashboard on pre-update code. --plan restarts only the
  gateway; this install also runs hermes-dashboard.service, the endpoint the desktop app
  and browser actually connect to, which the updater knows nothing about. The autoupdate
  script now restarts it explicitly, and the manual path in §3 says to do the same. Symptom
  this removes: "I updated Hermes but the UI is unchanged."

Corrected — three claims that do not survive checking

Recorded because they are plausible, circulate as advice, and are wrong:

Claim	Reality (v0.20.5 source)
"The gateway drain defaults to 1800s, so set agent.restart_drain_timeout: 5 to speed restarts up."	restart_drain_timeout defaults to 0 — no drain at all; a restart interrupts in-flight agents immediately. Setting 5 increases the wait. The 1800 figure is HERMES_AGENT_TIMEOUT, the idle-agent timeout, unrelated to restart drain. The only real drain is cron_drain_timeout (default 30s). No config change applied.
"hermes cron create --schedule '0 4 * * 1' --prompt '…'"	hermes cron create takes positional schedule [prompt]; there are no --schedule / --prompt flags. The command as written fails. (And see above for why cron is the wrong mechanism regardless.)
"_spawn_hermes_action lives in hermes_cli/web_server.py."	It lives in hermes_cli/web_routers/{profiles,tools}.py. The underlying cgroup diagnosis is sound; the file reference is not.

The upstream systemd-run --user --scope code fix remains a reasonable idea for the
in-app path, but this install does not need it — the timer sidesteps the problem.

Verified

The whole path was exercised on the live VM, not just installed. 02-vm-install.sh
re-run to exit 0; timer armed for Sun 2026-08-23 04:25:21 UTC; then
systemctl --user start hermes-autoupdate.service — the same unit the timer fires — was
run against a VM that was genuinely 3 commits behind:

Unit result	Result=success, ExecMainStatus=0
Version moved	upstream 209e2ebd (+1 carried commit) → upstream 8e475ed2
Markers	last-success only — no last-failure
Restart order	gateway 09:42:35 → dashboard 09:42:42
03-verify.sh	14/14, including the new check 14

That restart order is the proof the design works: the updater restarted the gateway,
survived it (a cron- or dashboard-launched updater would have been reaped there), and
then restarted the dashboard.

hermes update --plan and --check were also run read-only beforehand. Desktop side: /Applications/Hermes.app carries no
app-update.yml, so there is no electron auto-update feed; hermes desktop is
documented as "Build and launch the native desktop app", confirming the app is built
from its own checkout and cannot be updated by a VM-side timer.