v0.5.0

Upgrade notes

• Rollout-step approvals recorded before the upgrade stop counting (the approval key now includes the step's and the experiment's rings): give them again.
• Upgrade every halo-server replica, the halo CLI, halo mcp serve and halo controller run before adding the first identity.roleBindings entry.
• halo-server takes an exclusive lock on --data-dir: a second process on the same directory refuses to start.
• FIPS builds refuse webhook, audit-export, cloud-webhook and telemetry-salt secrets under 14 bytes at startup; standard builds warn.
• When shadow pair keys are configured, unencrypted pair rows are refused; pass --allow-plaintext-pairs (MCP allow_plaintext_pairs) once to migrate a store sealed part way through.
• POST /api/v1/service-tokens rejects unknown, repeated or case-variant keys with 400.

Added

• Fleet-scale load test for halo-server (make load-fleet, test/load/fleet, .github/workflows/load-fleet.yml): simulated halod devices enroll, poll the signed kill list, fetch rings and post reports against file and Postgres modes, with latency percentiles, error rates and server CPU and memory; a manual or monthly 10,000-device workflow with a Postgres service container. Sizing guidance with the measured numbers is in the Operations guide ("Sizing halo-server"): up to 10,000 devices per replica, one replica more than needed, memory and Postgres storage per device.
• Scoped role bindings for halo-server (ADR-0037): identity.roleBindings grant a console role to groups and users over a scope of rings and units (a unit includes its descendants; rings and units intersect). Every route declares how its resource is scoped (routeScopes); a scoped grant covers a resource only when it lies wholly inside the scope, derived from policy and server records, never the request. Lists (fleet, devices, releases, experiments, rollouts, toggles, kill list, freeze, requests, service tokens) are filtered, unknown resources deny scoped callers with the same 403 as out of scope, and the audit log, budgets, policy and a few admin routes stay global only. Service tokens take a scope within their creator's grants (halo service-token create --scope-ring/--scope-unit); scoped approvals count toward ADR-0030 quorums only while their scope covers the change; scoped actions are audited with authzRole/authzScope/authzBinding. /api/v1/me and /capabilities return bindings; halo whoami shows console roles and scopes. identity.roles, adminGroups and existing tokens are unchanged. Upgrade every halo-server replica, the halo CLI and halo mcp serve before adding the first binding: older binaries refuse such policies and scoped tokens.
• Controller notifications beyond Slack: Microsoft Teams (Workflows/Power Automate webhook, Adaptive Card; the retired Office 365 connector URLs are refused), PagerDuty (Events API v2: trigger with a stable dedup_key per rollout or experiment for rollbacks, kill-switch holds and paused rollouts, resolve when a human merges or advances, or the kill is lifted; informational events never page) and the signed generic webhook, which can now receive all-clears. Per-sink event filters (--notify-<sink>-events, server.controller.notify.<sink>.events), bounded retries with backoff on 429 and 5xx that honour Retry-After (never on other 4xx), halo_controller_notifications_total{sink,result} and a HaloControllerNotifyFailing alert. Secrets stay in files and Secrets (--notify-teams-url-file, --notify-pagerduty-routing-key-file). Slack output is unchanged. See the notifications guide.
• PagerDuty and webhook all-clears also fire when an experiment or rollout is deleted from policy while an incident is open, so a deleted name no longer stays paged forever.
• GitLab merge requests and a GitHub REST path for every change Halos proposes (promotion, rollback, rollout, controller, console proposals, halo upgrade, MCP propose_*): --forge gh|github|gitlab and --forge-base-url on halo exp promote, halo rollout advance|pause|rollback, halo controller run, halo upgrade check|watch, halo mcp serve and halo-server; portal config policyForge / policyForgeBaseURL; Helm server.controller.policyRepo.forge, forgeBaseURL, gitlabToken. GitLab uses REST v4 with GITLAB_TOKEN (api scope, nested group paths, self-managed hosts), GitHub uses GITHUB_TOKEN/GH_TOKEN (GitHub Enterprise Server via base URL); both are idempotent by marker and send no merge, auto-merge or approve call (a human still merges). The default is unchanged: with no flag the forge is autodetected from the origin remote (gitlab if the host names it, else the gh CLI). Tested against httptest fakes only, not a real GitLab or GHES (UNVERIFIED). halo doctor reports GITLAB_TOKEN instead of gh when the policy repo's origin is a GitLab host, --forge-base-url under a URL prefix (https://host/gitlab/api/v4) strips that prefix from the remote project path, and the MCP propose_* tools now embed a change marker and return the existing open PR/MR instead of opening a second one (merged or closed ones never block a new proposal). See the new GitLab and GitHub Enterprise Server guide.
• An OpenAPI 3.1 description of halo-server's HTTP API: api/openapi.yaml, published at /halos/openapi.yaml on the docs site and served unauthenticated and read-only at GET /api/v1/openapi.yaml. It lists every route with auth schemes, request and response schemas derived from the Go types, status codes, the RBAC permission each route needs (x-halo-permission, from the route table) and a stability marker (x-halo-stability, stable on every operation under the existing /api/v1 promise). Generated by make openapi (also run by make docs-gen); go test ./internal/server -run OpenAPI fails when a route is registered without documentation, a permission differs from the route table, an example stops decoding into its Go type, the file is not valid OpenAPI 3.1, or a handler answers a status or body the file does not describe.
• Signing keys in AWS KMS or an HSM (ADR-0038): --key-uri file://|awskms://|exec:// on every signing command (HALO_KEY_URI), halo-server --killswitch-key-uri, and halo keys public --key-uri to export the public key and key id. awskms:// signs with ECC_NIST_P256 or ECC_NIST_EDWARDS25519 keys through the KMS API (no new dependency); exec:// runs an external signer speaking the documented halos.signer.v1 JSON protocol (PKCS#11, GCP, Azure, Vault adapters). Every remote signature is verified against the key before use. Guide: Signing keys in a KMS or HSM.
• ECDSA P-256 (ecdsa-p256-sha256) release, pointer, attestation and kill-list signatures, verified by halod, halo-proxy, halo-kong, the portal and exports next to ed25519. Each signature is checked under its recorded algorithm only; unknown algorithms are refused. Manifests and kill envelopes record the signing key id. --co-key-uri co-signs with the old key of the other algorithm while rotating, so halods that trust only it (including older releases of halod) keep verifying. Releases signed before this change verify unchanged.
• FIPS 140-3 builds: *_fips_* archives and -fips image tags built with GOFIPS140=v1.0.0 (Go Cryptographic Module), make build-fips / make test-fips, and a CI job running the signing, verification, TLS, session and encryption packages under GODEBUG=fips140=only. Guide: FIPS 140-3 builds. Halos itself holds no FIPS certificate.
• Service-account API tokens for halo-server (ADR-0036): halo service-token create|list|revoke and /api/v1/service-tokens. A token (halo_sat_ + 80 hex digits, for secret scanners) holds one role no stronger than its creator's, always expires (--service-token-max-ttl, default 90 days), is stored only as SHA-256, revocable alone on every replica, and audited as service-account:<name>:<id>. Only an interactive admin session can create one. The CLI's server-backed commands and halo mcp serve --server read HALO_TOKEN, which wins over HALO_SESSION. Postgres: migration 4 adds halo_service_tokens (run halo-server migrate); halo-server import copies a data dir's tokens.
• Break-glass local admin: halo-server breakglass-hash (PBKDF2-HMAC-SHA256), --breakglass-hash-file, --breakglass-user, --breakglass-enabled (Helm server.breakGlass). Sealed unless enabled; POST /auth/breakglass gives a fixed 30-minute admin session; every login, failure and request is a high-severity audit entry, and a login whose audit entry cannot be written is refused. SIEM export marks high-severity entries (OCSF severity High, syslog critical). Runbook: Admin access guide.
• Console session policy: --session-ttl (default 12h, unchanged) and a sliding --session-idle-timeout (default 1h; 0 turns it off), Helm server.session.
• Policy identity.groupsOverage (deny default, or graph to read groups from Microsoft Graph, --entra-graph-url for national clouds) and identity.requireAcr / identity.requireAmr for the console login. Okta and Entra ID setup guides.
• Helm chart operations kit. monitoring.prometheusRule.enabled renders a PrometheusRule (only when the cluster serves the CRD) from deploy/helm/halos/files/prometheus-rules.yaml, now the single source of the alert list: the Grafana alerting provisioning (deploy/observability/grafana/provisioning/alerting/halos-alerts.yaml) is generated from it by make obs-assets, and go test ./internal/obsassets fails when the rules, the generated file, the dashboards or the operations guide drift, or when a rule or panel names a metric no binary exports. New alerts: HaloServerErrorRateHigh, HaloServerGatewayLookupErrors.
• Four Grafana dashboards for Prometheus under deploy/observability/grafana/dashboards/ (server, gateway, fleet as the server sees it, rollouts and experiments), built only from exported metrics; monitoring.dashboards.enabled ships them as ConfigMaps for the Grafana sidecar.
• PodDisruptionBudgets for every component with more than one replica (the OTel collector included), optional HPAs for the Postgres-backed halo-server, halo-shadow and the collector (<component>.autoscaling.enabled, off by default), a default one-pod-per-node podAntiAffinity (podAntiAffinity.*), and global.imageRegistry / global.imagePullSecrets applied to every pod and image (the policy sync sidecar and the collector included) for mirrored and air-gapped installs. A render test asserts the hardened security context on every container for each CI values file.
• The release workflow packages the chart and pushes it to oci://ghcr.io/<owner>/charts/halos with chart version and appVersion equal to the tag, signs it keyless like the images, and the verify job checks signature, version and default images. Available from the first release that includes this change (v0.5.0 or later); earlier versions install from a source checkout.
• Disaster-recovery drill: scripts/uat-dr.sh (make uat-dr, CI job uat-dr) installs the HA chart on kind with Postgres, writes state through the API, runs pg_dump, destroys the database, restores, and asserts the kill list, the audit chain and halo audit verify, a single controller leader and the gateway endpoint. New guides: backup and restore (RPO and RTO per mode, PITR, restore order) and air-gapped install. A new chart CI job lints, renders and kubeconform-validates the chart with every values file.
• halo test, the offline tier of testing a policy change: in seconds, with no network or credentials, it validates the policy, renders every ring x harness x OS through the adapters (surfacing adapter warnings; --strict fails on them), builds each ring's release and checks its manifest and reproducibility, asserts no permission-bypass token in any rendered file, with --against REF previews who changes ring, variant or settings and fails above --max-impact percent (default 25), runs PolicyTest cases from tests/ (identity -> expected ring, variants and rendered setting values; schemas/policytest.schema.json) and simulates every rollout. Output text, json (with next_steps) or junit; exit 0 pass, 1 could not run, 2 a check failed. The MCP server gains the read-only policy_test tool. examples/acme-corp/tests/ has passing cases. Guide: Testing a policy change.
• Compliance docs: a control mapping (SOC 2, ISO/IEC 27001:2022, NIST SSDF and SLSA, EU AI Act) to the code and evidence Halos produces, a data-flow and PII inventory with retention defaults, and a security overview with known gaps. No certification is claimed.

Fixed

• halo-server on a file backend refuses to start when another process holds --data-dir (exclusive lock on <data-dir>/.lock); two processes on one directory lost each other's writes. A failed write while revoking a device no longer un-revokes it in memory (the request answers 500 and the retry persists it), and a panicking coalesced catch-up no longer leaves the coalescer stuck or crashes with an unlock of an unlocked mutex.
• halo-server at fleet scale (found by the load test above). With 10,000 devices enrolling, every ring, report and kill-list request waited behind each enrollment's fsync because the device store held its lock across log writes: steady-state ring lookups fell from a mean of 5.0 s (p99 10.4 s, 0.9% timing out) to 5 ms (p99 15 ms, no errors), reports from 4.1 s (p99 9.4 s) to 5.5 ms (p99 14 ms), and the kill-list p99 from 7.8 s to 42 ms (10,000 devices, file mode, adjacent runs). The per-request policy-directory walk is now shared between concurrent requests (360 us to 43 us per read under load), and the fleet report history shares unchanged values between a device's reports (45.7 KB to 16.9 KB per device, 450 MB to 165 MB at 10,000 devices). Behaviour and protocol are unchanged.
• GET /api/v1/policy, /api/v1/experiments and /api/v1/experiments/{name} returned null instead of [] for an experiment with no rings or variants, and null instead of {} for resolved.harnesses of a profile that pins no harness. They now return empty lists and objects, as the OpenAPI description declares.
• Helm: the schema-migration hook Job (server.database.migrateJob) failed a fresh helm install because the pre-install hook ran before the chart's ServiceAccount existed, so its pod was never created. On install it now uses the namespace default ServiceAccount (no token is mounted either way).
• halo onboard local (and the MCP init_policy tool) failed with exit 2 when cursor-agent was installed: autodetect copied its date-style version (2026.04.17-787b533) into halos.yaml and validation rejects anything but an exact semver. A CLI whose version is not a semver is now left out of the autodetected pins with a warning that names it and the --tools name@x.y.z flag; any remaining validation failure also prints that hint instead of only the validator errors.
• halo keys generate wrote halo.key into the current directory, often the policy repo, where git add -A would commit the signing private key, and halo onboard company generated no .gitignore. The key now goes to <user config dir>/halos/keys (directory 0700, key 0600) unless --out is given, and a private key is refused inside a git work tree unless --force. The company repo gets a .gitignore for *.key, *.token, .halos/local/ and dist/.
• halo onboard verify exited 0 when every CLI was skipped (no credentials), so a run that proved nothing looked green. It now ends with verified X of N and exits 3 when X is 0 (1 when a CLI failed, 0 when at least one verified and none failed). The MCP verify_harness result carries the same summary.
• The README halo onboard company generates told you to "merge this repo's PR" although the directory is not a git repository: it now gives the git init / commit / remote / push steps, lists the publish command for every ring (it listed only the GA ring), and says the Helm chart is installed from a Halos checkout because it is not published to an OCI registry.

Changed

• Kill-list envelopes carry alg and keyId (absent alg means ed25519); gateways and devices accept the envelope they already hold as a no-op. halo-server caches a remotely signed kill list for up to 30 seconds while it is unchanged.
• Shadow pairs are sealed with cipher.NewGCMWithRandomNonce (required by fips140=only); the stored format is unchanged and existing rows decrypt.
• Re-publishing the same release reuses its existing, verified manifest instead of re-signing (ECDSA signatures are randomized).
• A console session now ends after an hour without a request. A HALO_SESSION copied from a browser stops working an hour after the browser last used it: move automation to HALO_TOKEN, or set --session-idle-timeout=0.
• MCP server (halo mcp serve): one response envelope for every tool. Every result, success or error, carries next_steps, an ordered list of {tool | command, why, args}: a tool call with an example args, or the exact human command at a gate. It is applied centrally by a middleware around tools/call, so new tools get it without per-tool code. Handler-written string steps are folded into the same shape, and the next key of init_policy and onboard_company results is replaced by next_steps.
• MCP argument errors are actionable and arrive with next_steps: the error names the field, its type, an example and the allowed values (plan without ring lists the policy's rings), unknown fields list the valid ones, and a string where an array is expected (delivery: "mdm") or name@version items where tools is a map are accepted. Tool input schemas now carry examples and default, and output schemas advertise next_steps.
• onboard_company over MCP requires only org. Every other answer has a documented default (gateway_kind halo-proxy, delivery halod, CLIs found on PATH, provider anthropic, example.com placeholders for issuer, gateway, registry, policy repo and console URL); the dry run echoes them in assumed, lists the placeholders in placeholders, and a real write is refused until they are replaced.
• The onboard and autopilot MCP prompts spell out the tool call sequence with argument shapes, the human gates and the stop conditions.
• The install pages are one: getting-started/installation (components, build from source, prerequisites) is merged into getting-started/install.
• The quickstart timing claim is now "about a minute with a warm Docker cache, 2 to 3 minutes cold" (a fresh cold-cache run took 2:31), replacing "55 s warm, 86 s cold".
• POST /api/v1/service-tokens decodes its body strictly (ADR-0037 review): an unknown, repeated or case-variant key is now a 400 (it was ignored), and a scope naming no ring or unit is refused. halo service-token create --scope-* revokes a token whose returned scope differs from the request (an older server ignores scope) and fails.
• Rollout-step approvals are keyed by the rings they approve (the next step's and the backing experiment's, kept apart): retargeting either invalidates them. The key changed shape, so every rollout-step approval recorded before the upgrade stops counting and must be given again.
• An access request's approval rules and freeze windows (ADR-0030) now cover what approving it changes: an opt-in also matches the requested ring; an MCP server request matches every ring its profile reaches (rings on profiles extending it, experiment variants on them) and every unit.
• The rollout gate without an approval rule no longer counts the requester's own approval.
• The console's request form has a "Change ticket" field. GET /api/v1/catalog gives each item the ticket patterns the caller's request needs (ticket), the form shows them up front, and a refused ticket is shown next to the field with what was typed kept. Filing an MCP server request that reaches GA needs a ticket again from the console.

Security

• exec:// signers are re-checked before every run: the binary, its symlink target and every ancestor directory (owned by root or the current user, not group- or world-writable unless sticky), plus an optional sha256= pin. They run in / with absolute PATH entries only, and their process group is killed after every exit, so a backgrounded helper cannot outlive them.
• A hanging remote kill-list signer costs one in-flight call: polls for other lists share one pending slot (newest list) instead of starting a call each.
• awskms:// endpoint= hosts must be plain ASCII (checked before case folding).
• MCP eval_harvest gains allow_plaintext_pairs (default false); the plaintext-row error names the switch of the front end that hit it.
• Shadow pair stores: with pair keys configured, an unsealed (plaintext) row is now refused when read by halo eval online|harvest, the MCP eval tools and halo-shadow decrypt-pairs, so a row injected by anyone with write access to the store cannot pass as sealed. Stores that hold rows written before sealing was turned on: pass --allow-plaintext-pairs (-allow-plaintext-pairs for halo-shadow) once to read them, off by default.
• FIPS builds refuse to start with an HMAC key Go's FIPS module would panic on (under 14 bytes): halo-proxy telemetry.unitSalt, --notify-webhook-secret-file, --audit-config hmacSecret, --cloud-webhook-secret-file. The error names the setting. Standard builds log a warning and keep running. The GitHub webhook verifier never panics on such a key.
• awskms:// endpoint= must be a KMS host (kms[-fips].<region>.amazonaws.com[.cn], kms[-fips].<region>.api.aws, vpce-*.kms.<region>.vpce.amazonaws.com) of the SigV4 signing region, or loopback; other AWS hosts (API Gateway, S3) are refused. FIPS builds use the kms-fips endpoint unless fips=false.
• exec:// signers run with only PATH, HOME, HALOS_SIGNER_* and the variables named by env=; on Windows the URI is exec:///C:/path/to/signer.exe (a host component or a UNC path is refused with the correct form); their stderr is logged at debug level and never put in errors; on timeout their process group is killed (Unix); a signer binary that is group- or world-writable, or owned by anyone but root or the current user, is refused.
• Re-publishing a release reuses its existing manifest only when it carries exactly the signatures the publish writes; adding or removing a co-signer since is an immutability error instead of a silently dropped co-signature.
• A remote (KMS, external) kill key: the signature runs detached from the poll with its own timeout and the cache lock is not held across it; during a signer outage an unchanged list is served from the cache for up to 3 minutes (halo_server_killswitch_stale_served_total, halo_server_killswitch_sign_failures_total), while a changed list is an error, never the old list presented as current. A local kill key is recommended (Signing keys in a KMS or HSM guide).
• Gateways and devices accept a kill list with the kills they hold and an older issuedAt (another replica a little behind) as a no-op instead of logging a replay error.
• An Entra ID login whose token signals groups overage (_claim_names / hasgroups) logged in with no groups, silently dropping the user's roles. It is now refused with a clear, audited error (login.denied), or resolved from Graph with identity.groupsOverage: graph.
• Service tokens and break-glass cannot file, approve or deny access requests or approve rollout steps (403, audited as authz.deny with need: human), and a token counts as its creator in every separation-of-duties check: a person refused as their own approver could otherwise approve, or count twice, through a token they minted.
• Agent credentials delegated from a console session expire no later than the session's idle deadline, so an agent chain cannot outlive an idle session.
• Break-glass: --breakglass-allowed-cidrs (Helm server.breakGlass.allowedCIDRs) is required to enable it (halo-server refuses to start and the chart to render without it; a range broader than IPv4 /8 or IPv6 /32 (0.0.0.0/0, ::/0) is refused, and IPv4 /8-/15 or IPv6 /32-/47 is warned at startup). Outside the ranges it answers 404 before the password is read and refuses sessions used from elsewhere, so outsiders cannot lock it. 20 failed logins in 15 minutes from allowed addresses lock it on every replica for 15 minutes, audited once (breakglass.lockout, high severity); an SSO admin lifts the lock with POST /api/v1/breakglass/unlock (breakglass.unlock). A state-changing break-glass request is refused (503) unless its audit entry is written first.
• The 1,000 active service-token cap is enforced atomically, also across replicas.
• IdP logins whose user id starts with service-account: or breakglass:, in any letter case, are refused, so audit attribution cannot be spoofed. Invalid identity hardening fields make halo-server refuse the policy at load.