Skip to content

Managed Hugo Upgrades

Jump to bottom
Jm Rohmer edited this page Aug 20, 2026 · 4 revisions

Managed Hugo Upgrades

Since #980/#990 (v1.8.3), the server can optionally stage, activate, and roll back official Hugo releases through five tools: get_hugo_update, bootstrap_hugo, stage_hugo_upgrade, activate_hugo, rollback_hugo. This page is the runbook for the actual arleo.eu deployment — for the full config reference see docs/operator-guide.md § "Managed Hugo Upgrade Configuration" (that table is the source of truth for field meanings; this page is not a copy of it, see Documentation Source of Truth).

What This Does and Does Not Do

  • get_hugo_update is read-only and safe to call anytime. check_latest:true makes one bounded outbound request to the official Hugo GitHub API.
  • bootstrap_hugo is a one-time setup step for a deployment that has never used managed upgrades: it detects the currently-installed Hugo version, then re-downloads and checksum-verifies that exact version from the official release — it never trusts the pre-existing on-disk binary directly — and activates it as the initial managed baseline. Refuses with bootstrap_unavailable if a managed version is already active. Without this step, rollback_hugo has no target to restore on a system's very first real activation, because the pre-existing unmanaged binary was never itself a managed version. dry_run defaults to true.
  • stage_hugo_upgrade downloads, checksum-verifies, and extracts a release into a private managed directory. It never touches the live hugo binary.
  • activate_hugo atomically swaps a managed symlink — never /usr/local/bin/hugo or a package-manager path — and never restarts the service itself. An explicit operator restart is required afterward for the swap to take effect.
  • rollback_hugo atomically restores the previous managed target, same no-auto-restart rule.

get_hugo_update requires a write bearer. The other four — bootstrap_hugo, stage_hugo_upgrade, activate_hugo, rollback_hugo — require admin (see OAuth Clients and Scopes: admin is a separately-approved tier on top of write that exists specifically to gate these four managed Hugo binary lifecycle tools). Bootstrap/staging/ activation/rollback additionally require hugo_upgrade.enabled: true in config — status reporting works even when disabled.

Why the systemd PATH Change Is Required

build_site invokes the bare command hugo and lets the OS resolve it via $PATH (internal/tools/admin/build.go). activate_hugo only ever changes where hugo_upgrade.binary_link points — if that symlink's directory isn't first on the service's PATH, an activation "succeeds" but the next build_site silently keeps using whatever hugo resolved to before. This is exactly the kind of drift class that caused the #981/#983/#984 incidents earlier in the v1.8.x cycle — verify this before trusting an activation.

Live Configuration (hugo-vm)

/etc/systemd/system/mcp-hugo-server-go.service.d/hugo-managed-path.conf
/etc/mcp-hugo-server-go/config.yaml   (hugo_upgrade: block, appended)

The drop-in sets:

[Service]
Environment=PATH=/var/lib/mcp-hugo-server-go/hugo/current:/usr/local/bin:/usr/bin:/bin

/var/lib/mcp-hugo-server-go was already in the base unit's ReadWritePaths (used for the OAuth token store and other state), so managed_dir needed no new systemd grant — only the PATH drop-in.

config.yaml has:

hugo_upgrade:
  enabled: true
  managed_dir: /var/lib/mcp-hugo-server-go/hugo
  binary_link: /var/lib/mcp-hugo-server-go/hugo/current/hugo
  require_extended: true
  allow_downgrade: false

After editing either file:

ssh hugo-vm 'sudo systemctl daemon-reload && sudo systemctl restart mcp-hugo-server-go'

Verification Performed (2026-08-10, full live cycle)

Enabling this was config-only: prepending an empty managed directory to PATH is safe, since a missing/empty PATH entry is simply skipped by lookup and /usr/local/bin/hugo (the pre-existing install) resolves exactly as before. The full cycle was then exercised live on hugo-vm, in this order:

build_site                                    -> status: ok (proves PATH change didn't break Hugo resolution)
get_hugo_update(check_latest=true)            -> managed_upgrades_enabled: true, installed 0.147.0, latest 0.164.0 available
stage_hugo_upgrade(v0.164.0, dry_run=false)   -> checksum_verified: true, version_verified: true, staged: true
activate_hugo(v0.164.0, dry_run=false)        -> activated: true (first activation: previous_target empty, no managed baseline existed yet)
sudo systemctl restart mcp-hugo-server-go     -> confirmed current/hugo -> versions/v0.164.0/hugo
rollback_hugo(dry_run=false)                  -> rolled_back: true, restored v0.147.0 (checksum re-verified before symlink restore)
sudo systemctl restart mcp-hugo-server-go     -> confirmed current/hugo -> versions/v0.147.0/hugo (final state, deliberate)

hugo-vm's live .mcp-activation.json after this cycle:

{
  "active_version": "v0.147.0",
  "previous_version": "v0.164.0",
  "previous_checksum": "121847728886057e53cd093fcb99554a8437708663677a05b2a0119ff4bf1f4c",
  "previous_extended": true
}

Staged binaries live at /var/lib/mcp-hugo-server-go/hugo/versions/<vX.Y.Z>/, mode 0700, owned by the mcp-hugo-server-go service account (the jm operator account correctly cannot read them directly — that's the design, not a bug).

hugo-vm is already bootstrapped as a side effect of this test cycle — its first activate_hugo call created the managed baseline that bootstrap_hugo exists to provide on fresh deployments. Running bootstrap_hugo on hugo-vm today will correctly refuse with bootstrap_unavailable; that refusal is not a bug, it's bootstrap_hugo doing its job.

To Actually Upgrade

On a fresh deployment that has never used managed upgrades, run this once first — it seeds the initial managed baseline from whatever Hugo is already installed, so rollback_hugo has something to restore to after the very first real upgrade:

bootstrap_hugo(dry_run=false)

Then, for every upgrade after that (including the first one on an already-bootstrapped deployment like hugo-vm):

get_hugo_update(check_latest=true)
stage_hugo_upgrade(target_version="vX.Y.Z", dry_run=false)
activate_hugo(target_version="vX.Y.Z", dry_run=false)
ssh hugo-vm 'sudo systemctl restart mcp-hugo-server-go'
get_hugo_update    # confirm installed.version now matches
build_site         # confirm the new binary actually builds

If anything looks wrong after restart, rollback_hugo(dry_run=false) then restart again — the previous binary's checksum is re-verified before the symlink is restored, so a tampered or missing previous binary fails closed rather than silently reactivating something unverified. Then run build_site again: the on-disk public output was rendered by whichever binary was active at the last build, so after a rollback+restart it's still stale until you rebuild. Skipping this step is exactly what left hugo-vm reporting get_site_health → status: degraded, missing_public_pages: 40 for a period during this feature's own live test — the content and rollback were fine, the public tree just hadn't been rebuilt yet.

Common Mistake

Do not enable hugo_upgrade.enabled: true without also adding the PATH drop-in first — the tools will happily stage and "activate" a new Hugo version whose binary build_site never actually picks up, and you'll spend time debugging a phantom version mismatch instead of a one-line systemd config gap.

Clone this wiki locally