Skip to content

hal0 v1.6.0

Choose a tag to compare

@github-actions github-actions released this 23 Oct 18:28
· 122 commits to main since this release
ab096cb

Highlights

  • Hermes v0.22.4 on Python 3.14, installed as a release. The bundled
    Hermes moves from 0.19.0 to v0.22.4 (4d043038) with its venv on a
    uv-managed Python 3.14, built from upstream's lockfile into a release
    layout that hal0 update activates with an automatic flip-back on a failed
    start. hal0 agent upgrade hermes --to takes only a vetted commit, and
    building an older vetted release over a newer one's state.db is refused.
    (#2851, #2853, #2856, #2874, ADR-0032, ADR-0034)
  • Every Hermes profile is reachable from Open WebUI, with hal0's tools and
    memory.
    hal0 registers one pinned, read-only /v1 upstream per served
    Hermes profile, so the Open WebUI picker lists hermes-agent, coder,
    hal0-brain and the rest, and every named profile gets hal0-admin and
    hal0-memory under its own identity with its own private memory bank.
    (#2898, #2902, ADR-0035)
  • A terminal-off box stays off for Hermes's hidden side agents too. From
    v0.22.3 the gateway's preview-restart side agent ignores
    agent.disabled_toolsets; hal0 now also writes a terminal.backend Hermes
    rejects in the tool's own availability check, in the root config and every
    profile, so terminal and process_manage leave every schema. Opting in
    puts the operator's backend back. (#2878, #2912)
  • hal0 doctor watches the Hermes gateway and its API path. New rows fail
    when the gateway unit file is missing or an unmanaged gateway run --idle-exit is running, check each served profile's authenticated
    /v1/models and upstream, and warn rather than fail when the operator
    turned the gateway off. (#2874, #2898, #2905)
  • The memory engine runs on a local slot or an upstream model, switched
    without a restart.
    Pick it under Engine model on Settings → Data →
    Memory or with hal0 memory graph enable --model <id>; hindsight-api calls
    one virtual id, hal0/memory, that hal0 resolves per call, and a new
    Engine running row shows the model the engine really loads, including
    an override from /etc/hal0/hindsight-llm.env. (#2918, #2926)
  • Rotating a key ends every live connection that used it. Open WebSockets
    and Server-Sent Events streams holding the old credential close at
    rotation, and an expired session ends its streams. (see Removed)

Added

  • The memory engine can run on an upstream model, switched from the dashboard or CLI without a restart. [memory.graph].extraction_model names an upstream model id hal0's /v1 already routes (an OpenRouter or MiniMax id, say); unset, the engine stays on the local extraction_slot. hindsight-api now always calls one virtual id, hal0/memory, which hal0 resolves to your choice on every call, so picking a model is a config write that applies at once, uses hal0's own upstream credentials, and never restarts the engine — only a timeout or extraction-limit change still does, plus one repair of a stale or missing drop-in on the first save after upgrading. Pick it under Engine model on Settings → Data → Memory or the agents Memory tab (Local slots / Upstream models), or with hal0 memory graph enable --model <id> and --local. GET /api/memory/graph/status gains target, available_models and engine_llm, and the new Engine running row names the provider and model hindsight-api actually loads — including, in red, an override from /etc/hal0/hindsight-llm.env, which systemd applies after every hal0 setting and which used to leave hal0 reporting a slot the engine was not using (#2918, #2919, ADR-0023 amendment).

  • sudo hal0 agent upgrade pi re-pins an installed pi without an uninstall. It reruns the pinned installer, redeploys hal0's two extensions, and leaves the extensions you added, your default provider, model and theme alone. Run it the same way you installed pi (sudo hal0 agent upgrade pi); it refuses when pi's profile is in a different home or pi is not installed. --to stays Hermes-only. A fresh hal0 agent install pi from a git checkout now seeds a name-keyed banks alias (coding-agent::hal0 to coding-agent::hal0-mono) in ~/.hindsight/coding-agent.json instead of a path mapping, and upgrade pi adds that alias to an existing file, which other Hindsight-backed tools on the box also read.

  • A bank's description is editable from the Memory workspace. The text under the bank picker is Hindsight's reflect_mission for that bank — the persona reflect answers with — and until now every bank showed whatever it was created with (the hindsight coding-agent plugin stamps one text on all of its banks) and only hal0 memory bank profile set --reflect-mission could change it. Click the description to edit it in place; Save writes the same per-bank config override the CLI does, nothing else.

  • GET /api/agents/{name}/overview returns the state behind the Hermes and Pi cards in one call. For Hermes it reports the live and previous release against the pin, the agent and gateway unit states, the API server's health and served profiles, the configured platforms, the last provision run, four doctor rows, the command-approvals state with the pending count, memory counters, the desktop link, any running build job, and a phase with the reasons a running Hermes is degraded. A provision report older than the live release's build is marked stale and no longer degrades the card, because hal0 update rebuilds Hermes without rewriting it. For Pi it reports the installed and pinned versions, the profile checks, and whether the Hindsight API answers. Every probe is bounded and fails soft: a box without systemd answers with unknown values, not an error, and a stopped gateway reads inactive. A gateway the operator disabled or masked is not reported as degraded. The probes are cached for 30 seconds per agent (a Hermes build starting or ending is a miss), generated_at says when they were taken, and ?refresh=1 rebuilds them. The overview is rebuilt right after a restart, a command-approvals change or a repair, a probe that hangs past 8 seconds answers with partial: true instead of stalling the card, and the configured-platform list is probed at most every five minutes.

  • hal0 doctor has a Hermes gateway unit row. It fails when Hermes is installed but /etc/systemd/system/hermes-gateway.service is missing, or when a gateway run --idle-exit process is running, naming its pid and cgroup. Either state means a Hermes v0.22.3+ client could run, or is running, a gateway of its own where the operator turned the gateway off (ADR-0034 amendment, #2874).

  • [hermes] delegation_slot in /etc/hal0/agents/hermes.toml picks the slot Hermes's delegate_task children run on. Hermes's delegation.model was rendered from the agent slot and nothing else, so on a single-GPU box every child queued behind its parent on the same llama-server (--parallel 1), and a hand edit of delegation.model was undone by the next repair. The default stays agent. Any other slot name renders delegation.model = "hal0/<slot>", which hal0's /v1 route resolves to that slot and pins the lane even when two slots share a model id. The setting is applied at provision or repair while the slot is ready, and hal0 doctor has a row for it. There is no CLI or API writer yet (#2893) (#2885).

  • Every Hermes profile now has hal0's tools and memory, and hal0 doctor sees when the gateway would lose them. A profile cloned by the operator started with no mcp_servers, so through Open WebUI or Telegram it answered without hal0-admin and hal0-memory. hal0 now writes those two servers into every named profile's config.yaml, as hermes__<profile> with a private memory bank and the box bearer, at provisioning (new phase profile_mcp_wire) and on every hal0-api reconcile sweep, without touching anything else in the file; Hermes picks the change up on its own 60 s reconcile. Opt a profile out with hal0: {wire_mcp: false}. Two new doctor rows: hermes_profiles_mcp (each profile wired, with a bearer when auth is on) and hermes_generation, which warns when Hermes's package manager has committed an environment for the live release that lacks extras such as mcp, the drift that silently removed the tools from every Telegram, API and cron turn (#2891, ADR-0035).

  • Open WebUI and any /v1 client can now chat with each Hermes profile. hal0 registers one pinned, read-only upstream per Hermes profile that has an API key (hermes and hermes.<profile>), so /v1/models and Open WebUI's picker list hermes-agent plus one model per profile next to your slot models. A Hermes model id never falls back to a slot: if its upstream is offline or disabled the request fails with dispatch.upstream_unavailable. A gateway that is down or still starting keeps its entries; hal0 re-reconciles every 120 s and drops a named profile only when the running gateway answers 404 for it. The Services page's Open WebUI card gains an Agents chip, hal0 doctor gains a hermes-api row ("Hermes API upstreams"), and POST /api/agents/hermes/reconcile-upstreams re-registers them on demand. X-Hermes-Session-Id and X-Hermes-Session-Key are now forwarded to upstreams, and provisioning gains a profile_api_keys phase that writes a missing API key into a named profile's own .env and mirrors the default profile's key, which lives in hal0's root-only vault, into $HERMES_HOME/.env so hal0-api can read it. On a box where Hermes was installed before this release, run sudo hal0 agent reprovision hermes --repair once so the default profile appears; hal0 doctor says so. Uninstalling Hermes removes its entries at once, and /v1/models lists an id a Hermes profile owns as Hermes's even when a slot alias has the same name. /v1/responses is not available through hal0. See the Open WebUI guide, "Talking to Hermes", and ADR-0035.

  • POST /api/agents/hermes/repair re-runs the Hermes provisioner as a tracked job. It queues hal0 agent bootstrap hermes --repair --verbose as a child of hal0-api and returns the queued job with 202; GET /api/updates/status/{id} serves it under the repair:hermes phase, with each [run ] <step> line the provisioner prints appended to steps and the output copied to /var/lib/hal0/state/agents/hermes/provision-logs/job-<id>.log. Converge and repair see each other: while either is queued or running, or another process holds the Hermes builder lock, the route answers 409 agent.busy with the job (or the lock holder) in details, and 500 agent.builder_lock_unreadable when the lock file cannot be opened (a fault of the box, not a busy builder). A repair is bounded to 45 minutes. Restarting hal0-api during one (a hal0 update does not wait for it) kills the build; the live release is unchanged and the job reads failed with error_code: api_restarted. The job runs as the hal0 user and skips the root-only steps of sudo hal0 agent reprovision hermes --repair (the stale drop-in sweep, the default-key mirror, the CLI wrapper, the terminal-tool record, the ownership reconcile), so ownership, key or wrapper problems still need that command on the box.

  • POST /api/agents/hermes/gateway/restart restarts the Hermes messaging gateway without restarting the agent. It restarts hermes-gateway.service through the same systemctl wrapper as the agent restart, with the same {agent_id, unit, status, detail} answer and the same agent.restart_failed and agent.restart_timeout errors, and answers 404 agent.not_installed when Hermes is not installed.

  • The Agents view shows each agent as a state-aware card with a console on its back. The front follows the box and uses these status words: not installed (the install command to copy), broken and down (the last journal lines of the agent unit, with Start, Repair and Logs), degraded (up to three reasons, each with a one-click fix), working (the running upgrade or repair), ready and serving (persona, approvals and terminal chips, plus the chat platforms, API profiles and desktop link), and unknown (Restart and Logs, because hal0 cannot read the agent unit). A version label shows the live release, and an ↑ badge names the pin when the live release is behind it. Click a card to flip it to its Status, Controls, Lore and Lifecycle tabs. Controls restart the agent or the gateway, switch the persona, set command approvals to smart or manual, and re-mirror the profiles to Open WebUI. Lifecycle upgrades Hermes to the pin through the existing component converge job (offered only when the live release is behind the pin), repairs it with a new repair job that re-runs every provisioning phase and rebuilds the release at the pin (the card lists the steps the job reports, with elapsed time), and uninstalls it behind a type-to-confirm dialog. One action runs at a time per card, and a build started from the CLI or the Services page also shows on the card as working. Restarting hal0-api during a repair stops the build and keeps the live release. Pi is a live card too: it shows the installed pi and adapter against the pins and the profile checks, and offers sudo hal0 agent upgrade pi, uninstall and reinstall as copy buttons, because the daemon never installs, upgrades or uninstalls Pi.

  • The Agents view shows each agent as a state-aware card with a console on its back. The front follows the box and uses these status words: not installed (the install command to copy), broken and down (the last journal lines of the agent unit, with Start, Repair and Logs), degraded (up to three reasons, each with its fix where hal0 has one), working (the running upgrade or repair), ready and serving (persona, approvals and terminal chips, plus the chat platforms, API profiles and desktop link), and unknown (Restart and Logs, because hal0 cannot read the agent unit). A version label shows the live release, and an ↑ badge names the pin when the live release is behind it. Click a card to flip it to its Status, Controls, Lore and Lifecycle tabs. Controls restart the agent or the gateway, switch the persona, set command approvals to smart or manual, and re-mirror the profiles to Open WebUI. Lifecycle upgrades Hermes to the pin through the existing component converge job (offered only when the live release is behind the pin), repairs it with a new repair job that re-runs every provisioning phase and rebuilds the release at the pin (the card lists the steps the job reports, with elapsed time), and uninstalls it behind a type-to-confirm dialog. One action runs at a time per card, and a build started from the CLI or the Services page also shows on the card as working. Restarting hal0-api during a repair stops the build and keeps the live release. Pi is a live card too: it shows the installed pi and adapter against the pins and the profile checks, and offers sudo hal0 agent upgrade pi, uninstall and reinstall as copy buttons, because the daemon never installs, upgrades or uninstalls Pi.

Breaking

  • Updating from 1.3.x now takes one hop through 1.5.0. The release's upgrade_from floor moves from >=1.3.0 to >=1.4.0, so the 1.3.x updater's post-swap compatibility shim in hal0.updater.updater (the PROFILE_CATALOG_SCHEMA_VERSION re-export and the ceiling= parameter) is deleted. A box on 1.3.x gets UpdateUpgradePathUnsupported naming the floor; run hal0 update --target 1.5.0 first, then update again. A box on 1.4.x or later is unaffected (#2783).
  • hal0 agent upgrade hermes --to takes a vetted commit sha, and the Hermes venv moves to a release layout on the next update. --to now accepts only a full 40-hex commit listed in VETTED_HERMES_REFS; a version string or an ==X.Y.Z spec is refused, and the in-place pip install --upgrade path is gone. Every Hermes build (install, reprovision --repair, upgrade, and the venv repair and converge passes of hal0 update) now builds a fresh release under /var/lib/hal0/hermes/releases/, checked before it goes live, behind a current symlink; /var/lib/hal0/venvs/hermes becomes a symlink to current/venv. Activating a release stops the agent and gateway units that were running, switches, runs hermes config migrate, starts them again, and switches back to the previous release with config.yaml restored if the re-check fails. The first hal0 update after this release migrates every host this way, with a short Hermes outage. uv now ships in hal0's own venv, and hermes-prereqs.sh no longer installs it (#2615, ADR-0032, ADR-0034).
  • hal0 update now regenerates the Hermes gateway unit and enables linger automatically; run sudo hal0 agent install hermes only if the update reports it couldn't. An existing box keeps the gateway unit Hermes 0.19.0 wrote: no After=/Wants=user@<uid>.service and no linger for hal0. That unit still starts Hermes v0.22.2, but without the hal0 user's systemd manager Hermes cannot give its workers their own scope. Kanban workers never start: each dispatch tick logs kanban dispatcher: spawn of <task> deferred, host cannot place the worker and the card stays queued without spending its retries. Cron jobs still run, but outside a scope, so a gateway restart kills a job in flight. Once the v0.22.2 release is live, hal0 update calls a new hal0-update hermes-gateway verb. As root, that verb renders the v0.22.2 unit from hal0's own template and never runs Hermes code. It turns linger on for hal0 and restarts the gateway only if it was running. It never enables, starts or stops the gateway. A gateway you disabled and stopped is left alone. The sudoers grant does not change. If the verb fails, hal0 update, hal0 update status and hal0 doctor all (its hal0_linger row) still print sudo hal0 agent install hermes (ADR-0034 decision 6, #2615).
  • The bundled Hermes moves from 0.19.0 to v0.22.2 (d923a5eb), and its venv from Python 3.12 to 3.14. v0.22.2 declares requires-python = ">=3.11,<3.15", so it is the pin that moves hosts to 3.14. The first hal0 update to 1.6.0 builds a new Hermes release next to the live 0.19.0 one: a uv-managed CPython 3.14 under /var/lib/hal0/hermes/python/cpython-3.14*, dependencies installed from upstream's hash-locked uv.lock, then the switch. The release's release.json records that interpreter as python with "lock": true, the 3.12 release stays as previous, and /etc/hal0/hermes-python.env is rewritten to the 3.14 interpreter (the old 3.12 value is discarded with a hermes-python-discarded warning). The pin moves in lockstep: requirements line, VETTED_HERMES_REFS and its requires-python record, the drift-watch table and the contract fixtures (ADR-0032 decision 15, ADR-0034 decision 8). Three things change for an operator. The Hermes gateway now turns on systemd linger for the hal0 user, because Hermes's scheduled and kanban workers run under that user's systemd manager; hal0 doctor all shows it on a new hal0_linger row, and hal0 agent uninstall hermes and installer/uninstall.sh run loginctl disable-linger hal0 (ADR-0034 decision 6). Session history is kept: Hermes 0.22 deletes ended sessions after 90 idle days by default, and hal0 writes sessions.auto_prune: false so nothing is pruned (ADR-0034 decision 7). And hal0 no longer writes auxiliary.web_extract or auxiliary.session_search, which Hermes 0.22 ignores, and drops them from an existing config.yaml. The MCP client probe now works with the mcp 2.0 client in the new venv, and the unread HERMES_DASHBOARD_TUI variable is gone from the agent unit (#2615).
  • With Hermes v0.22.4, a profile-scoped chat in Hermes's dashboard and the hermes chat CLI need the Hermes gateway. From Hermes v0.22.3 the gateway daemon owns chat turns and the session store. The dashboard's default Chat tab still runs its turns in the dashboard process and needs no gateway. A chat opened for another Hermes profile (such as hal0-brain), and the classic hermes chat CLI, are gateway clients: they attach to hermes-gateway.service through /var/lib/hal0/.hermes/gateway.sock. If that unit is masked, failed or stopped, they report gateway inaccessible instead of starting a session. hal0 makes sure they never start a gateway of their own on a box where the gateway is disabled (ADR-0034 amendment, #2874). hal0 keeps the unit file installed even when the gateway is turned off, because an installed unit, in any state, is what stops a client from starting a gateway of its own; hal0 doctor has a Hermes gateway unit row that fails when the file is missing or a gateway run --idle-exit process is running. To chat in another profile, run the gateway (sudo systemctl enable --now hermes-gateway); with no messaging platform configured it serves only the API server and cron (#2615).
  • Rolling back from 1.6.0 to 1.5.x leaves the Hermes build failing until you update again. 1.6.0 records a Python 3.14 interpreter in /etc/hal0/hermes-python.env. A 1.5.x hal0 accepts only 3.12, so its Hermes converge stops with Hermes Python must be exactly 3.12: <path> reports 3.14 and reports the build as failed. The 1.5.x MCP probe also imports streamablehttp_client, which the mcp 2.0 in the v0.22.2 venv no longer has, so on 1.5.x every hal0 update reports the Hermes venv as broken and its repair fails, and hal0 agent bootstrap hermes and hal0 agent reprovision hermes fail at mcp_wire. The Hermes release 1.6.0 built keeps running and serving through all of this. Do not edit /etc/hal0/hermes-python.env to point 1.5.x at a 3.12 interpreter: 1.5's venv builder then replaces the release symlink at /var/lib/hal0/venvs/hermes with a 0.19.0 venv and leaves a stray /var/lib/hal0/venvs/hermes.rollback-<pid> behind. Update back to 1.6.0 instead, which clears every one of these errors (#2615).
  • Removing a NO_PROXY exemption from api.env now removes it from the Hermes agent's NO_PROXY. A driver env rewrite run as root (hal0 agent bootstrap hermes --repair, HermesDriver.install, an MCP exposure change) read the previous /etc/hal0/agents/hermes.env back in as an operator source, so every entry it had ever written stayed there, including an operator *. A rewrite now takes operator entries only from api.env and its own process environment, so an entry exported only in an interactive shell is dropped again by the next rewrite from the service. When hal0 cannot read the MCP registry to find the exposed loopback hosts, a rewrite keeps the previous NO_PROXY lines; when there is no readable previous file, the install_artifacts step of hal0 agent bootstrap hermes fails and names the fix, and HermesDriver.install raises, instead of writing a list without those hosts (#2534).

Removed

  • Rotating a key now closes the open WebSockets that were using the old credential. Before, every WebSocket checked its cookie or key only when it connected, so a realtime socket or a board socket opened before a rotation kept working afterwards. A rotation now closes those sockets with code 4401. That covers the default rotation that signs out other sessions, and client-key and admin-key rotations, including an admin-key rotation with keep_sessions. A client that gets 4401 should sign in again and reconnect (#2801).
  • Rotating a key now also ends open Server-Sent Events streams, and an expired session ends open streams and WebSockets. Before, a stream such as /api/events/stream, /api/journal/stream or a streamed /v1 completion was authorised once, when it started, and kept sending after the key was rotated. Every long-lived stream is now covered by the same check as the WebSockets, so a rotation ends it. The stream's last frame is event: revoked, which plays the part close code 4401 plays for a socket. A process-wide check every 30 seconds also ends streams and closes WebSockets (with 4401) whose session cookie has expired (8 hours, or 30 days with Remember me). Before, an open board or realtime socket outlived its session. A client should sign in again and reconnect. With auth off, nothing is revoked (#2816).
  • The agent chat proxy is gone: /api/agents/{id}/events, /submit, /session/handshake, /session/create, /session/resume and /session/history now return 404. Chat with Hermes happens in Hermes's own dashboard, which hal0 does not proxy (ADR-0033, amending ADR-0004). The proxy could not reach the pinned Hermes: Hermes rejected its credential, so the sockets closed with 1011 before any traffic, and the session routes called REST methods Hermes does not have. Nothing in hal0 has called these routes since the v0.4 dashboard rework (#439). Removed endpoints normally wait for a major release; these ship in a minor because no client used them and none of the proxied routes could work. /session/handshake only minted the same session cookie POST /api/auth/login mints, so a script that used it to get a cookie should sign in through /api/auth/login instead. hal0 agent bootstrap hermes no longer writes /var/lib/hal0/.hermes/runtime.json, and hal0 doctor perms no longer checks it. An existing host keeps its old copy: nothing reads it, and uninstalling the agent removes it with the rest of /var/lib/hal0/.hermes. The HAL0_HERMES_RUNTIME_JSON, HAL0_HERMES_HOST and HAL0_HERMES_PORT environment variables, which only the proxy read, have no effect (#2817).

Changed

  • Hermes is pinned to v0.22.4 (4d043038) instead of v0.22.2 (d923a5eb). v0.22.3 rolled up about 620 upstream PRs; v0.22.4 adds 14 commits of Desktop, plugin and dashboard fixes. The review found no change to anything hal0 relies on: the gateway unit hermes gateway install --system writes is identical, so the unit hal0 regenerates on update needs no change; requires-python, the lock's Python 3.14 markers and the web, mcp and platform extras are unchanged; hal0's provider and memory plugins load, the MCP probe connects, the dashboard still injects its session token and accepts ?token= on /api/ws, and Hermes still reads agent.system_prompt for the persona. The upstream changes that do reach a hal0 box are the gateway taking over chat turns (see Breaking), a config schema step from 50 to 51 that only removes cron.bot_chat_delivery_timeout_seconds, and dependency security fixes: mcp 2.3.0 (GHSA-84m7-p3x7-pcfv), urllib3 2.8.0 and PyJWT 2.15.1, plus tornado 6.5.10 and multidict 6.9.1 on a box whose release carries messaging-platform extras. The gateway API server gains POST /v1/runs/{id}/clarify and POST /v1/runs/{id}/resolve-unknown, behind the same key. v0.22.2 stays in VETTED_HERMES_REFS for the automatic flip-back after a failed activation, but once v0.22.4 is live, hal0 agent upgrade hermes --to d923a5eb487a17c0a68801bba9ef0f56c81040a8 is refused as a downgrade: v0.22.3+ writes gateway runtime state into state.db that v0.22.2 cannot run over, and hal0's release switch keeps only config.yaml (#2805 tracks a manual rollback). A box on v0.22.2 rebuilds onto v0.22.4 at its next hal0 update. The contract fixtures are re-vendored from v0.22.4 (ADR-0032 decision 15). (#2615).
  • The Hermes venv's Python follows the pinned commit, and hal0 can build it on 3.14. hal0 builds the Hermes venv on the newest of Python 3.14 and 3.12 that the pinned commit's requires-python admits. The previous pin, 3ef6bbd2, caps below 3.14 and builds on 3.12; the move to 3.14 comes with the v0.22.2 pin bump (see Breaking). Interpreters are resolved in this order: a HAL0_HERMES_PYTHON override, then a uv-managed CPython under /var/lib/hal0/hermes/python, then a system interpreter that links SQLite 3.51.3 or newer. hal0's own interpreter is never used. A resolve with no lock is pinned to the commit date with --exclude-newer, and hal0 doctor reports the live interpreter's SQLite version (#2615, ADR-0032 decision 9, ADR-0034 decisions 1 and 2).
  • A stale or invalid HAL0_HERMES_PYTHON no longer stops a Hermes build. Before, a value from the environment or from /etc/hal0/hermes-python.env that failed validation, or a file the parser rejected, made hal0 agent install hermes, hal0 update and the preflight fail until an operator fixed it by hand. The value is now discarded with a hermes-python-discarded warning that names it, why it failed and the interpreter chosen instead. hal0 then re-probes (a uv-managed CPython of the target minor first, then a system interpreter of that minor that links SQLite 3.51.3 or newer) and writes the new interpreter back to the file. When the file is read-only to the hal0 user, the re-probed interpreter serves that build and the file is left alone. If nothing is found the error is unchanged (ADR-0032 decision 9, #2615).
  • CI now requires Python 3.14. The 3.14 leg no longer carries continue-on-error, pull requests run 3.12 and 3.14, pushes to main also run 3.13, and the package declares the 3.13 and 3.14 classifiers (#2840).
  • The dashboard's optional browser Sentry no longer sends the browser's User-Agent or Referer. This only matters for a dashboard built with VITE_SENTRY_DSN. The Sentry SDK moved to v11, which replaced sendDefaultPii with per-category dataCollection settings that collect everything unless switched off. hal0 switches every category off: no user info or IP inference, no cookies, no request or response headers, no bodies and no URL query strings. Before, the SDK attached the User-Agent and Referer headers to every event. Spans still travel inside their transaction event, so the existing secret scrubber still sees all of them. (#2834).
  • Pi is no longer shown as "on the roadmap" in the Agents view. It is a live card with its own status; only Qwen and opencode stay behind the grey "Coming soon" mask.

Fixed

  • A fresh install no longer ends with "Verify FAILED: structured-output probe failed". The installer's last check sends a JSON-mode chat request to hal0/utility, but a fresh install binds no model until you pick one in the dashboard, so the check could never pass and every new box finished with a red summary. The installer now reads the slot list first. When no model is bound it prints "structured-output probe skipped — no model bound yet" and does not count it as a failure. When the slot the request would land on has a model but is not loaded, it skips the probe too, because sending the request would make hal0 load that model just for the check. It also skips, rather than guess, when the slot list cannot be read or the slot reports a state it does not recognise. A slot in error still fails the verify, now naming the slot, without sending the request. With a loaded model the probe runs as before. One known gap remains on CPU-only boxes, where first-run seeding binds and loads the small brain model. The probe sends its request while that model is busy with the other post-install checks, the request can outlast the probe's 120 s limit, and the install still ends with a false "Verify FAILED". Structured output itself works there, it is only slow (#2952).

  • A fresh install now lays down hermes-gateway.service, and hal0 update moves a gateway off the suffixed unit a 1.6.0 nightly install left behind. Under Hermes v0.22.4, upstream's hermes gateway install --system run as root resolves its default home from root's account home, so the installer and hal0 agent install hermes got hermes-gateway-<hash>.service instead: it named the release it was built from (which the next update prunes, killing the gateway), skipped hal0's secrets drop-in, could not be restarted from the dashboard, failed the Hermes gateway unit doctor row and left a stray /root/.hermes. Both now write the unit from hal0's own template as root, without running Hermes, and keep the masked and disabled off switches. hal0 update stops, disables and removes a hal0 suffixed unit and installs hermes-gateway.service with the same on/off state; the doctor row names the suffixed unit and the fix. A /root/.hermes left by such an install is not removed; hal0 doctor perms still reports it (#2943).

  • The Hermes agent card no longer reads "degraded" on a healthy box after a fresh install or a Repair. The provision smoke test counted any non-zero exit from upstream hermes doctor as a failure. Upstream exits 1 whenever it lists an issue, and what it lists describes a standalone Hermes install, which no hal0 box is: no provider API key in Hermes' .env (hal0 keeps keys in its vault), unconfigured optional tool keys, hal0's terminal-off sentinel as an unknown backend, no ~/.local/bin/hermes (hal0's entry point is /usr/local/bin/hermes), and npm audit findings in upstream's own workspaces. The smoke phase therefore always recorded a warning, and the agent card showed it as provision_warn with no fix to offer. The probe now treats that list as advisory and shows it in the smoke result's detail. It still fails if hermes doctor crashes, times out, prints no readable summary or exits with any other code. (#2951)

  • Settings → Secrets and provider credentials no longer accept loader or interpreter variables. Both write to /etc/hal0/api.env, which is hal0-api's own environment file, and set the value in the running process at once, so a "secret" named LD_PRELOAD, PYTHONPATH or PATH was inherited by every process hal0-api starts (model pulls, the Hermes provisioner, update helpers) and by hal0-api itself after its next restart: an LD_PRELOAD entry ran arbitrary code as the hal0 user, and a wrong PATH could stop the service from starting. On a box running without auth, anyone who could reach port 8080 could set one. Those names are now refused with secret.protected (403) on the secrets routes, and with a 400 on the credentials route and as an upstream's auth_value_env when it is created or edited: PATH, HOME, SHELL, IFS, ENV, BASH_ENV, NODE_OPTIONS, NODE_PATH, NODE_EXTRA_CA_CERTS, PERL5OPT, PERL5LIB, RUBYOPT, GCONV_PATH, SHELLOPTS, BASHOPTS, PS4, HERMES_HOME and anything starting with LD_, DYLD_, PYTHON, UV_, PIP_, NPM_CONFIG_ or GIT_. The credentials route also refuses HAL0_* names, as the secrets routes already did: on an upstream with no declared env name it used to accept any key, HAL0_PORT and HAL0_ADMIN_KEY included. Removing one that is already there still works, so check Settings → Secrets once after updating and delete any such entry you did not put there on purpose. Root was never reachable this way: hal0's sudo calls reset the environment (#2936).

  • The memory engine's Engine model picker lists upstream models right after a restart, and the Engine running row tells the truth about an override. The upstream catalogs were cached only by the dispatcher's first cold prefetch, so the picker's Upstream models group stayed empty until some chat request missed; the graph status now warms cold provider catalogs itself, at most once a minute per provider. And engine_llm read /etc/hal0/hindsight-llm.env, which is root-only on some boxes, so a provider/model override there showed as "not overridden"; it now reads the running hindsight-api process's own environment (same hal0 user), never opening the key file, reports "cannot verify" when neither can be read, and no longer blames that file for an engine that simply has not restarted onto a new drop-in (#2928).

  • Mental models no longer all read "stale", and each bank has a refresh floor you can set. Hindsight's is_stale only says the bank was written since a model's last refresh, so on a busy bank every model was painted amber within minutes of refreshing. The bank bar's Rules tab now shows when each model last refreshed, marks new memories since with a quiet dot, and warns only when a refresh failed after the last success. A new refresh selector on the same tab, and hal0 memory bank profile set --min-refresh-interval <seconds>, set the bank's mental_model_min_refresh_interval_seconds, which caps how often consolidation can re-run each model unless the model sets its own interval (0, the engine default, means after every consolidation) (#2920).

  • A fresh install's memory engine follows the Engine model picker from the first call. The installer-owned hindsight-api.service still pointed the engine at hal0/utility, so on a box that had never saved its memory graph settings the engine bypassed the hal0/memory alias and an upstream model picked in the dashboard took effect only after one save wrote the drop-in. The unit now defaults to hal0/memory, the same value the drop-in writes (#2925).

  • A bank whose id carries a dot can be managed and deleted from the dashboard again. The admin routes under /api/memory/banks/{bank_id} rejected every dot as a path trick, so a bank the hindsight coding-agent plugin named after a repo (coding-agent::ultrareview-1.4) browsed fine through the engine but answered invalid bank_id to stats, profile and delete through hal0. Dots are now legal inside an id, as colons already were; a leading dot, a dot-only segment and any .. run stay refused. The dashboard's new-bank form accepts the same grammar.

  • A terminal-off box now keeps the shell away from Hermes's hidden side agents too. From Hermes v0.22.3 the gateway's preview-restart side agent is built with the terminal toolset on and ignores agent.disabled_toolsets, the list hal0 used to switch the terminal off, so a preview.restart from an authenticated gateway client could run shell commands as the hal0 user. hal0 now also writes terminal.backend: hal0-terminal-off while the terminal is off, in the root config and in every profile it wires; Hermes rejects the unknown backend in the tool's own availability check, which every agent in the gateway process shares, so terminal and process_manage disappear from every schema. Off means off for every backend: a docker or ssh backend the operator chose is remembered under hal0.terminal_backend_saved and restored when the terminal is turned back on. hal0 update applies the gate on existing boxes and restarts Hermes when it changed; a provision that could not write it fails instead of reporting ok; cloned profiles also lose execute_code and delegate_task while the terminal is off. Hermes logs the rejection as Unknown TERMINAL_ENV 'hal0-terminal-off' at ERROR every 30 s while tools are listed; that line is expected on a terminal-off box (#2878).

  • Settings → Overview shows real CPU and GPU utilization. The API reports both as a fraction from 0 to 1, and the Live hardware panel rounded that fraction without multiplying by 100, so a busy GPU read 1% and an idle one 0%. The panel now shows the same percentage as the dashboard tiles (#2892).

  • The hal0-brain chat answers a request whose messages is a number or true instead of closing the stream empty. POST /api/brain/chat iterated messages without checking it was a list, so {"messages": 123} raised inside the event stream after the 200 response had gone out: the client got no frames at all and the hal0-api journal a traceback of about 1,500 lines. Any messages value that is not a list is now treated as no messages, as a string or an object already was, and the turn runs and ends normally (#2896).

  • A Hermes Desktop remote pairing now survives restarts and hal0 update. Hermes mints its dashboard session token fresh at every start unless HERMES_DASHBOARD_SESSION_TOKEN is set, and hal0 never set it, so a Desktop paired in token mode failed its next connection with "Reached the gateway over HTTP, but the live WebSocket (/api/ws) connection failed" and nothing in the journal: the HTTP leg hits the public /api/status, the WebSocket leg sends the stored token and is closed with 4401. hal0-agent@hermes now mints one token per install at /var/lib/hal0/state/agents/hermes/dashboard-session-token (0600, hal0) and exports it to the dashboard at every start. hal0 agent dashboard-token hermes prints it for pairing. A systemctl edit drop-in that sets the variable still wins. If the file cannot be written the agent starts as before and the shim says so on stderr. Existing pairings need the new token once (#2899).

  • hermes doctor no longer reports hal0's auxiliary task routing as broken. hal0 wrote provider: main for every auxiliary task without a dedicated slot (vision, and the utility tasks when no utility slot is loaded), and Hermes v0.22.2's hermes doctor rejects main as an unknown provider, so a fresh install or a hal0 agent bootstrap hermes --repair showed red rows such as auxiliary.compression.provider 'main' does not resolve. Hermes still routed those tasks to the chat model, so nothing stopped working, but a hand fix came back on the next repair. Every auxiliary.<task> block now names the custom provider with hal0's /v1 base URL. A task with no dedicated slot keeps an empty model, so it still uses the chat model. hal0 agent bootstrap hermes --repair rewrites an existing box's blocks (#2882).

  • sudo hal0 agent bootstrap hermes --repair no longer fails its install step with failed to open file /root/uv.toml: Permission denied. The builder dropped to the hal0 user without changing directory, so every uv it ran (the venv sync, uv python install, uv python find) started in root's /root and uv's upward search for uv.toml hit a directory hal0 cannot read. The hal0 child now starts in /, and the builder's own command runner defaults to / as a second guard. HOME was already pinned and was never the cause (#2883).

  • The Hermes plugin trees hal0 mirrors into $HERMES_HOME/plugins/ now converge on hal0 update and on a repair whose release build fails. The mirror ran only at the tail of the install phase, after the release build, so a failed build skipped it and an in-place update never ran it at all; one box carried a July kind: provider manifest for the hal0 model-provider plugin, which Hermes v0.22.2 downgrades to standalone with a warning on every start. The mirror now runs before the build, the updater re-runs it after activating a release (quietly, a copy failure never fails an update), and hal0 doctor reports a tree that differs from the shipped source instead of crashing when it cannot read it (#2884).

  • A Hermes update no longer fails and flips back when the gateway unit runs upstream's own launcher. If you ran hermes gateway migrate --multiplex, or hermes gateway install outside hal0, the gateway unit runs a launcher script inside one Hermes release's checkout (current/src/.hermes/bin/hermes). That script is tied to the release it was written in. After hal0 update built a new release, the gateway failed every start with "no dependency environment is committed for this install", the update switched back to the old release and reported activation_failed, and every later update did the same. The update now rewrites the unit to hal0's launcher, and turns linger on for hal0, while the gateway is stopped and before the new release starts. It does the same on an update with nothing to build. The rewrite goes through the same root step as before (hal0-update hermes-gateway), which takes no arguments, renders the unit from fixed paths, validates it, and never runs or reads the Hermes release. Any unit whose ExecStart, ExecStop or ExecStopPost is not hal0's launcher now counts as stale, not only units from before Hermes 0.22 (#2894).

  • The uv that ships in hal0's venv moves to 0.13.4, which drops a compromised CA root from its bundled certificate store (GHSA-hq85-5cpj-mr62). hal0 uses that uv to fetch the managed Python for Hermes releases and to install their dependencies. On 0.13.1, uv trusted an emSign root whose intermediate CA keys are compromised. A party holding those keys could intercept those downloads. Hermes package installs stay hash-checked against upstream's lock either way. The version floor moves to 0.13.4 as well. 0.13.4 is a day old, so this is a deliberate exception to the 7-day uv quarantine. Supersedes #2858.

  • A Hermes release now installs the SDKs for the messaging platforms the box uses, so moving a box to Hermes v0.22.2 no longer takes Telegram, Discord, the webhook or the gateway API server down. The release builder installed only the pin's web and mcp extras. The 0.19.0 venv had python-telegram-bot, discord.py and aiohttp because Hermes installs them lazily when a platform is configured. Hermes v0.22.2 cannot run that lazy install inside a hal0 release, so after the update the gateway logged Platform 'Telegram' requirements not met, Platform 'telegram' is registered but adapter creation failed and Webhook: aiohttp not installed, and the bridges stayed down. The gateway API server needs aiohttp too, and hal0 enables it on every box with API_SERVER_KEY, so it could not start on any v0.22.2 release. A build now adds upstream's extra for each platform the box configures: telegram, discord, slack, teams, dingtalk, feishu, google-chat and wecom for those adapters, and sms (aiohttp alone) for the API server, the webhook and the other aiohttp adapters. Every box therefore gets sms, because hal0 always configures the API server; a config.yaml block with api_server: enabled: false turns it off. A platform counts as configured when its credentials are in the gateway's secrets vault or HERMES_HOME/.env, or when a config.yaml platform block says enabled: true; a block that says enabled: false wins over a token. hal0-api runs as hal0 and cannot read the 0600 vault, so it asks the hal0-agentenv seam, whose new secret-keys verb prints the names of the keys that are set and never their values. Extras the live venv already has are carried into the new release, including packages an operator installed by hand. Platform extras install only from upstream's lock, with --require-hashes, and release.json records them as platform_extras. A box already on v0.22.2 whose live release lacks a configured platform's extra is rebuilt on its next hal0 update; one where the operator installed the packages by hand is left alone. matrix is never added, because its python-olm builds from source. Hermes's own lazy install could fix only one platform per gateway start. It built a separate environment under HERMES_HOME/installs, holding that platform's extra but not web or mcp, and every later Hermes process for that release booted from it instead of the verified release venv. hal0 now treats such an environment as a reason to rebuild: the new release gets its own clean install key and carries every extra Hermes recorded for that environment, a TTS engine or search provider as well as a messaging platform. release.json keeps those extras under platform_extras, and every later rebuild carries them again. A lazily installed feature therefore costs one rebuild, not one on every update. matrix and the umbrella extras (all, termux, termux-all) are never carried (#2876, ADR-0034 decision 1).

  • hal0 update now shows a component's note, such as "Hermes config keys applied; restart hal0-agent@hermes to load agent.system_prompt". The update restarts hal0-api, whose boot pass re-records every component's status before the CLI reads the table, and that dropped the note the update's own converge had written. A note now survives that pass and is replaced by the next update's result (#2615).

  • The persona prompt now lands in Hermes's agent.system_prompt. The #2839 fix wrote it with hermes config set, but Hermes v0.22.2 refuses a multi-line value whose lines look like a YAML list (the persona's - hal0-admin: … lines) and writes nothing. The failure was not reported, so after hal0 agent install hermes a box had no agent.system_prompt at all and its sessions got only SOUL.md. The prompt is now written through the same YAML merge as hal0's list keys, and overrides.yaml still wins (#2839).

  • hal0 update now writes sessions.auto_prune: false and the persona prompt before the updated Hermes first starts. The update path builds and activates a Hermes release; it never ran the provisioning config overlay, so a box updated to v0.22.2 kept agent.system_prompt_prelude and the retired auxiliary blocks and had no sessions setting, and v0.22.2 then pruned ended sessions idle for more than 90 days at its first start. The release activation now writes sessions.auto_prune: false, moves the old prelude into agent.system_prompt, drops the retired keys and lays overrides.yaml on top, while the agent and gateway are stopped and before they start; if it cannot, the release does not start and the previous one comes back. A box already on the pinned release gets the same keys on its next hal0 update, which reports "Hermes config keys applied; restart hal0-agent@hermes to load agent.system_prompt" when it changed the file. The build's hermes --help check also ran Hermes's boot maintenance on the live HERMES_HOME, migrating config.yaml before activation and before the copy kept for a flip-back; it now runs against a throwaway home inside the release (#2615).

  • A stream that ends now releases what it held at once, instead of whenever Python's garbage collector runs. When a client disconnected from a streaming response, or a key rotation or session expiry ended it, the route's generator could be left suspended instead of being closed, so its cleanup (an event-bus unsubscribe, closing the upstream connection of a streamed /v1 completion) waited for a garbage collection, which an idle server may not run for a long time. Two things caused it: the request-ID middleware ran every route in a separate task behind a memory stream, where the cancellation could land outside the generator, and the revocation guard raised into the stream while it was already winding down. The request-ID middleware is now a plain ASGI middleware, and a revoked stream that is winding down has its last chunks dropped instead (#2816).

  • The active persona's prompt now reaches Hermes. The config overlay wrote it to agent.system_prompt_prelude, a key no Hermes version reads, so the persona and its MCP-tool and approval text never reached the model. It now goes to agent.system_prompt, which Hermes applies to every new session, and provisioning drops the stale key. hal0 also stops writing display.personality, which held the persona's display name and never selected anything; it is left to Hermes's own /personality command (#2839).

  • The Hindsight memory engine now installs on Python 3.14 without --ignore-requires-python (#1537). The engine venv accepts Python 3.11-3.14 (litellm 1.100+ lifted its 3.14 cap), so Ubuntu 26.04's system Python works as-is. The installer's bypass fallback is gone: when no 3.11-3.14 interpreter exists it warns, naming HAL0_HINDSIGHT_PYTHON, and skips the memory engine for that run, and it imports hindsight_api after pip install so a broken engine is reported (and its venv removed for a clean retry) instead of passing. The rest of the install carries on. hal0 update's engine upgrade follows the same rules.

  • The Hermes dashboard no longer runs an npm build every time the agent starts. _resolve_web_dist looked for the built UI only under the venv's site-packages, which an installed host never has, so --skip-build was never passed. Each release now builds the UI into its own checkout when the commit ships upstream's build script, and the agent finds it through venvs/hermes → current/src/hermes_cli/web_dist (#2837).

  • hal0 agent bootstrap hermes --offline now refuses a build instead of ignoring the flag. It set HAL0_HERMES_OFFLINE, which nothing read. When a Hermes build is needed offline, the install step now fails, names the step that needs the network, and leaves the live release running (#2838).

  • Re-running the installer leaves a disabled hermes-gateway, hal0-bench.timer or hal0-bench-worker off. The installer ran systemctl enable --now on these units every time, so a re-run silently turned back on anything the operator had disabled. It now records each unit's state before touching it. A disabled unit is refreshed but left off, a masked unit is not written over at all, and the Hermes gateway generator runs without starting the gateway. The installer prints the command that turns each one back on. Fresh installs, and units that were already enabled, behave as before (#2807).

  • hal0 agent install hermes leaves a disabled or masked hermes-gateway off. It used to run the Hermes gateway generator and then systemctl enable --now every time, so re-running it, for example to repair the agent venv, reconnected a gateway the operator had turned off. It now checks the gateway's state first. A disabled or runtime-masked gateway has its unit refreshed without being started, a masked one is not touched, and the command prints how to turn it back on. A new or enabled gateway behaves as before (#2815).

  • hal0 update no longer starts a stopped hermes-gateway after repairing the Hermes venv. The repair bounced the gateway with systemctl restart, which also starts a stopped unit. It now uses systemctl try-restart, so only a running gateway is bounced (#2822).

  • The gateway bounce after a Hermes venv repair now works during hal0 update. The updater runs that step as the hal0 user, where a plain systemctl try-restart fails because it needs interactive authentication. The failure was ignored, so a running gateway kept its broken MCP client until something else restarted it. As hal0, the updater now goes through the hal0-systemctl seam, as hal0 agent install already does, and logs a warning if the bounce fails (#2828).

  • hal0 agent bootstrap hermes, --repair and the installer now clear a stale Hermes agent drop-in. The sweep that renames a leftover hal0-agent@ drop-in (the kind that makes the unit fail with status 241 and restart in a loop) only ran as root, but every CLI and installer path had already switched to the hal0 user before reaching it, so it never ran and the broken unit survived every bootstrap, reprovision and upgrade. The sweep now runs as root before that switch, and the warning names the file the drop-in was actually renamed to, including a numbered .hal0-disabled.N name (#2803).

  • Re-running the installer no longer reports false memory-graph and structured-output failures. After restarting hal0-api the installer only waited for systemd to call the service active, so its post-install checks could run before the API answered HTTP and print hal0 API not running or structured-output probe FAILED. It now waits up to 60 seconds for /api/health to answer before those checks, and prints a warning instead of stopping the install if it never does (#2804).

  • On Python 3.12, a successful rootless image pull is no longer reported as failed. After reading the runtime's output to the end, hal0 killed the process before reading its exit code. On Python 3.12 that kill collects the exit status of a child that has already exited, so asyncio reported exit code 255 and the pull showed as failed. It only happened when the child had exited but asyncio had not collected it yet. The runtime is now killed only when a pull is aborted. Only the rootless pull path was affected, which runs in development and CI or on a host without the hal0-podman-rw grant. An installed host pulls through that seam and never had the problem. Python 3.13 and later were not affected (#2814).

  • The dashboard's model-pull and slot-image-pull progress recovers when its stream is revoked. When a key rotation or session expiry ends the progress stream, the pull keeps running on the server. The model-pull bar used to freeze at its last value, and a slot-image pull still in progress showed as failed. Both now check the pull's status, reopen the stream while the pull is still running (backing off up to 16 seconds, and stopping after 5 attempts in a row with no progress), and show that the user is signed out when the status check returns 401 (#2824).

  • hal0 model rm --keep-files no longer promises a way back that cannot work. When a kept file's extension is outside [models].file_extensions, the closing hint names hal0 model pull <id> for a curated model instead of hal0 model add; for a model with no curated entry it names no recovery command, because removing the registry row also removed the Hugging Face coordinates a bare re-pull needs. An unreadable config is compared against the schema's default extensions rather than assumed addable (#2748).

  • A file-set pull clears every installed file's tombstone. A kept voice or projector file no longer stays hidden from scans after the pull that reinstalls it; before, only the entry file's tombstone was cleared (#2748).

  • The slot log drawer and the Logs page "slot" channel no longer sit in "reconnecting…" forever when hal0-api cannot read the journal. On a box whose hal0-api.service predates the SupplementaryGroups=systemd-journal grant (hal0 update does not rewrite the unit), journalctl -f printed No journal files were opened due to insufficient permissions. and exited at once; the per-slot log stream then closed after zero bytes, the browser's EventSource treated that as an error and retried on a loop, and the one-shot /api/slots/{name}/logs claimed "no journal entries — the slot may not have started yet". A follow never finishes by itself on a healthy host, so journalctl exiting is now reported: the stream emits a degraded frame naming the cause and the fix (re-run the installer once; hal0 doctor all row "API journal read" tracks it) and is held open with keepalives like the synthetic-slot stream, so the drawer shows the notice instead of looping; the one-shot tail carries the same hint. hal0 slot logs --follow prints the reason and returns.

  • The dashboard's persona editor now edits the persona it opens. It posted name, slot and tone, which the API ignores, and always saved a hard-coded system prompt over the persona's own. It now loads the persona and saves its display name, system prompt, tool patterns (fnmatch over MCP tool names, one per line; blank allows every tool) and preferred model. The tone picker is gone; personas have no tone.

  • POST /api/agents/hermes/restart and the restart after a command-approvals change no longer fail with "Interactive authentication required". Both ran a bare systemctl restart hal0-agent@hermes.service as the unprivileged hal0 user, which polkit denies on every provisioned box, so the restart never happened and a command-approvals PUT rolled its change back. They now go through the same systemctl wrapper the Services page and the slot restarts use. A restart that outruns its 90 s budget reports agent.restart_timeout, an unknown outcome, and a command-approvals change is kept in that case rather than rolled back (#2913).

  • An update, converge or repair job that a hal0-api restart interrupted now reads failed instead of running for good. Job tasks run inside hal0-api, and GET /api/updates/status/{id} re-seeded a job from /var/lib/hal0/update-jobs/ as it was left, so a poll across a restart sat on a job nothing was running, and every later component converge and repair answered 409 (the update /apply, /prepare and /commit have no busy check). A job read back from disk while still queued or running now records failed with error_code: api_restarted (a Hermes converge or repair says the live release is unchanged; any other job says to check hal0 update status before retrying). A job that finished before the restart, including an apply or commit whose own restart of hal0-api is part of the update, reads as it was.

  • A Hermes install, repair (CLI or dashboard) or reprovision never loads a slot that is not already loaded to smoke-test it, and never restarts one with a pending config change. The last provisioning step sent a test chat to the agent's model through hal0's gateway, and the gateway loads a cold slot to answer, so a repair on a box whose agent slot was not loaded started loading the whole model. With a large model the host stalled for minutes and the agent's watchdog killed hal0-agent@hermes with SIGABRT. The chat check now runs only when hal0-api reports the slot behind model.default as loaded (ready, serving or idle) and that slot has no pending config change. A loaded slot whose running config lags its slot file, as it does after hal0 update until the slot restarts, would be restarted by the test request, so it is skipped too. A skipped check names its reason, for example skipped: chat model not loaded (slot agent is offline; skipped so the smoke test does not cold-load it), or what hal0-api could not answer. The memory round-trip check is also skipped when the memory engine's slot is cold, because writing a memory makes the engine call its model through hal0. The context-window check only reads the model list, which loads nothing, so it still runs on a cold slot and still reports a window below Hermes' minimum. A skip is not a failure: the step stays ok, the Agents card shows no "provision warnings", and hal0 agent status lists the skip reasons. The agent's watchdog health probe now waits five seconds instead of one, so the agent is no longer killed for answering a health check slower than one second.

Migrations

  • Hermes profile models: no operator step is required. The first hal0-api start after the update writes a missing API key into each named profile's own .env and registers every profile. hal0 agent reprovision hermes and the Hermes component converge job write missing keys too.