v0.19.0

Hub 0.19.0 removes the last master-realm bypass: reads and writes of analyses, records and realtime rooms now reach exactly as far as each grant's realm_scope. It also moves to Authup 1.0.0-beta.70, recovers stuck builds on its own, and moves the message bus into the shared hubwire library. It covers everything since v0.18.0.

📣 Some News

The master realm is no longer special

Up to v0.18.0, Hub decided "may this caller touch every realm?" by looking at the caller's realm name: a user or client of the master realm could read every analysis and write every record. Authup grants have carried a realm_scope (any, ownOrNull, own, none) all along, and Hub now uses it everywhere instead: analysis reads, record writes, analysis commands, realtime namespaces and the rate-limit exemption.

For most deployments this is invisible, because the built-in admin role and the system client hold their grants at any. It is not invisible for custom cross-realm roles that relied on sitting in the master realm. Give them their permissions at realm_scope: any before upgrading.

Upgrade order matters

Do these in order. Each step is explained in a highlight below.

1. Upgrade Authup to 1.0.0-beta.70 or later.
2. Add analysis_read and analysis_node_log_create to your Authup provisioning, and re-grant cross-realm roles at realm_scope: any.
3. Upgrade server-core-worker and server-storage.
4. Upgrade server-core, stopping the old replicas before the new ones serve traffic.

👀 Highlights

🔭 Reading analyses needs analysis_read and reach

Every analysis read used to be open to any logged-in caller. GET /analyses, /analysis-nodes, /analysis-node-events, /analysis-buckets and /analysis-bucket-files (list and record) now need a grant and return only the analyses the caller reaches.

A read is allowed by the new analysis_read or by any analysis write or lifecycle permission (analysis_create, _update, _delete, _approve, _execution_start, _execution_stop). Holding none of them answers 403; a record out of reach answers 404, like a missing one.

Caller	Reaches
User	what the grant's realm_scope covers, plus every analysis a node of the user's realm takes part in
Node client	the analyses its node takes part in
Analysis client	its own analysis
Any other client	what the grant's realm_scope covers

Buckets follow the analysis, by type: CODE for every participant (a node whose approval is pending needs the code to review it), TEMP and RESULT only through an approved node. Result files additionally need analysis_result_read, which narrows that reach and never widens it.

The analysis log proxies were the worst case: server-core forwarded GET/DELETE /analysis-logs and /analysis-node-logs to telemetry under its own credentials, so any logged-in caller could read or delete any analysis's logs. They now follow the same reach. A node reads only its own logs, and deleting needs analysis_update through the grant.

analysis_read                 # new: view analyses without changing them
analysis_result_read          # now also required to list or download result files

Warning

Add analysis_read to your Authup provisioning before upgrading. The built-in admin role (*) only picks up names that exist when provisioning runs, and until it exists every node save fails. Grant it to roles that view analyses, and make sure the worker's service client holds it at realm_scope: any, or builds of other realms' analyses fail with 404.

Note

Node clients need nothing: server-core grants them analysis_read in the background, at start and then hourly. Analysis clients created before the upgrade do not get it.

📖 Reading analyses requires analysis_read and reach

✍️ Writes take their realm reach from realm_scope

Every create, update, delete and command (analyses, analysis nodes, buckets and files, nodes, projects, project nodes, registry projects, client and registry credentials, storage buckets and files, telemetry events) now needs its permission through a grant that reaches the record's realm. Out of reach answers 403. An update must be allowed for both the stored and the updated record.

any        →  every realm, and records without a realm
ownOrNull  →  the holder's own realm, and records without a realm
own        →  the holder's own realm
none       →  no realm

Three behaviour changes ride along:

• Analysis commands check a permission. Starting a build or distribution needs analysis_execution_start; every other command needs analysis_update. Before, only realm membership was checked. client-ui shows each command's button by the same rule.
• Node clients are authorized by binding, not by realm. They hold analysis_read, node_read and the new analysis_node_log_create, all at realm_scope: none, and lose analysis_update. A node still reports its own execution status and writes and reads its own logs, but can no longer change other analyses in its realm. The hourly sync rewrites existing grants.
• A node's client is assigned by Hub only. clientId can no longer be set through POST/PUT /nodes, and a client belongs to at most one node. Before, a node manager could point a node at any client, including system, and the node-client sync then rewrote that client's permissions.

Warning

Add analysis_node_log_create to your Authup provisioning before upgrading, alongside analysis_read. The sync checks for both before it changes a grant, so a node client keeps its previous permissions until they exist. Give roles that start builds or distributions analysis_execution_start.

Note

The upgrade migration keeps a client that several nodes shared on the oldest node. The others get a fresh client on their next save, so their node software needs the new credentials.

📖 Writes take their realm reach from realm_scope · Realm scoping

📡 Realtime works for non-master users

Two older bugs meant that realtime updates never reached a user outside the master realm: the namespace pattern had no capture group, so every realm namespace was treated as the global one and refused, and client-vue connected lists with an empty realm to the global namespace. Lists of every other user updated only on reload. Both are fixed, and the namespaces now follow realm_scope too:

• The own realm's namespace (/resources/<realmId>) is open. Another realm's namespace and the global one (/resources) need a hub grant at realm_scope: any.
• Each room subscription is checked against the room's own permissions, so an any grant for one entity no longer opens another entity's rooms.
• Admin → Realms now offers "set management realm" on every realm, which picks the namespace the UI connects to.

The rate limiter no longer exempts a client for being in the master realm. Only a client holding every hub permission at any (the system client) is exempt; users are always limited.

📖 Realtime and rate limiting follow realm_scope

🛡️ Authup beta.70

Hub now requires Authup 1.0.0-beta.70, on the server as well as in the client libraries. The record-reach helpers Hub wrote for the changes above now ship in @authup/access, and token verification runs through vouchly. What you notice:

• An Authup outage answers 502, not 401. A failed introspection (Authup unreachable, 5xx, 429, Hub's own credential refused) used to surface as an invalid token, which made client-ui sign the user out although the token was never judged. An inactive or expired token still answers 401.
• registry_manage reveals registry-project secrets only within its reach. An own grant used to return accountSecret for every registry project the caller could read, in any realm.
• Cached introspection results are kept per Authup client under new vouchly:* Redis keys. The old token:* keys are not read again and expire with their tokens.
• A service booted without AUTHUP_URL answers 401 instead of crashing at boot.

Warning

Upgrade the Authup server first. Hub keeps its own malformed-bearer guard, because an Authup server older than beta.70 still authenticates Bearer not-a-token as the system client. Admins who manage other realms from Hub's realm pages need realm_update / realm_delete at any, and attribute-policy grants no longer join realm realtime rooms.

📖 Authup upgrade to beta.70

♻️ Stuck builds recover on their own

A build or distribution whose worker died used to stay started until someone clicked "check", and the worker judged staleness with a 15-minute rule that also failed healthy long builds. Now:

• server-core reconciles. Every 5 minutes it re-checks each build or distribution that has been starting/started without an update for 15 minutes, and marks it failed if the image is not there.
• Workers send a heartbeat every 5 minutes, so one long silent step is not judged stale. A step with no progress for 2 hours is treated as hung and stops the heartbeat.
• Transient worker failures are retried. A connection error, a 5xx/429 from an API, GitHub or the registry, or a failing docker daemon is retried with backoff up to 5 attempts, then reported as failed. A broken Dockerfile step or a 4xx still fails at once.
• Status transitions are race-safe. Every status change runs on a row held with SELECT … FOR UPDATE. Two concurrent start requests no longer both dispatch, and a late progress or finished event no longer writes an older status over executed, or brings back a run that was already failed or unlocked.

Warning

Upgrade the workers before server-core. An old worker sends no heartbeat and still applies its own 15-minute rule, so the new reconciler's checks would fail its healthy long builds. Set RabbitMQ's consumer_timeout above your longest build.

Note

With a single worker replica, a hung step is detected but not recovered: the re-check waits behind the hung run on the same queue. Restart the worker, run a second replica, or let consumer_timeout end the delivery.

📖 Message delivery (RabbitMQ)

🪣 Buckets: one per analysis and type, and uploads need ownership

A repeated or concurrent storage check could leave several buckets for one analysis and type, and the storage reply was matched to its analysis through a cache entry that a restart or another replica could lose. Bucket replies are now matched by the bucket's own ref and name, and both services enforce uniqueness:

• server-core keeps one analysis bucket per (analysisId, type), server-storage one bucket per (refType, refId, name). A repeated create answers with the existing bucket.
• POST /buckets with a refId answers 400: storage cannot verify who owns the referenced resource, so only the owning service sets refs, over the bus.
• POST /buckets/:id/upload into a bucket without a refId answers 403 unless the caller created it or holds bucket_update reaching its realm. Before, any logged-in user could upload into any bucket, and the creator was never recorded, so owner rights never applied.

Warning

The upgrade migrations remove duplicates and their down() does not restore them. server-core keeps the oldest link per analysis and type and deletes the newer ones with their file rows. server-storage clears the ref of every newer duplicate. No stored file is deleted; an orphaned bucket can be removed with DELETE /buckets/:id. A third migration recovers the ref of analysis buckets created before the ref columns existed, so uploads into older analyses keep working.

Warning

Stop the old server-core replicas before new ones serve traffic. An old replica holding the storageBucketEvents consumer drops the reply for a bucket a new replica created. If they did overlap, run storageCheck for every analysis created during the overlap.

📖 Uploading into a bucket requires ownership or bucket_update

🔌 Components and the message bus come from hubwire

Hub's components, scheduled jobs, dispatchers, workers and message bus moved out of @privateaim/server-kit into hubwire, a library shared with Authup. This only affects you if you build on the @privateaim/server-* kits.

- import { BaseComponent, MessageBusDispatchComponentCaller } from '@privateaim/server-kit';
- await caller.callExecute(payload);
+ import { Component, RemoteDispatcher } from 'hubwire';
+ await dispatcher.dispatchExecute(payload);

Every hub *Caller class is now a *Dispatcher with dispatch* methods, and the cache module and TaskManager are gone, with no aliases. On the wire nothing changes except the retry counter, which moved from the message body to the x-attempt AMQP header. Both directions stay compatible during a rolling upgrade.

Note

Without AMQP_URL, log lines and entity events for server-telemetry are no longer published at all, because the in-memory bus would keep them until shutdown with no consumer.

📖 @privateaim/server-kit message bus

🧹 Smaller fixes

• A Redis outage no longer drops audit events. The entity-event publisher stopped at the first handler that threw, and Redis runs before the telemetry audit bridge. Every handler now runs.
• Registry commands run only from the queue. Harbor link, relink and unlink used to run inside the request first and fall back to the queue when that threw, so a failing command could run twice. A registry-project rename is now one relink command, which also fixes the unlink being sent the new name. A node's robot credentials appear shortly after the node is created.
• Registry commands over HTTP work again. server-core set up its HTTP module before the registry dispatcher, so every registry command answered 400 and the Harbor webhook controller was never mounted.
• A component that fails to start is logged at error level instead of leaving a running server with a silently dead consumer.

⚠️ Heads-Up Before Upgrading

Change	What to do
Authup 1.0.0-beta.70 required	Upgrade the Authup server first
New permissions analysis_read, analysis_node_log_create	Add them to Authup provisioning before upgrading; grant analysis_read to roles that view analyses
Master realm no longer reaches every realm	Grant cross-realm roles their permissions at realm_scope: any
Result files need analysis_result_read	Grant it to roles that download results
Build and distribution start need analysis_execution_start	Grant it to roles that start them
Node clientId is Hub-assigned and unique	Update node software whose node got a fresh client; stop sending clientId
Reconciler and heartbeat	Upgrade server-core-worker before server-core; set consumer_timeout above your longest build
Bucket replies matched by ref	Upgrade server-storage before server-core; stop old server-core replicas before new ones serve traffic
Bucket dedupe migrations	Back up before upgrading; down() does not restore removed duplicates
POST /buckets with refId → 400, upload into ref-less buckets → 403	Create ref'd buckets over the bus only; upload as creator or with bucket_update
Authup outage → 502 instead of 401	Treat 502 upstream_error as retryable in API clients
realm_update / realm_delete at own manage only the own realm	Grant them at any to admins who manage other realms
@privateaim/server-kit component, bus, cache and TaskManager exports removed	Import from hubwire / @hubwire/amqp; rename *Caller → *Dispatcher, call* → dispatch*
createMasterRealmActor / createNonMasterRealmActor removed; fakes throw PermissionError	Use createAllowAllActor / createScopedActor; assert PermissionError from @authup/access

⬆️ Upgrading

docker pull ghcr.io/privateaim/hub:0.19.0

npm consumers of the server kits (@privateaim/server-kit already depends on hubwire; install it directly only where you import from it):

npm install @privateaim/server-kit@^0.19.0 hubwire@^0.2.1

Migrations run at boot. Read the deployment configuration sections linked above before rolling out.

👉 Changelog

v0.18.0...v0.19.0

⚠️ Breaking Changes

• assertRecordReach, reachesRecord, hasUnboundedReach, holdsPermission, isPermissionDenial, buildRecordPolicyData and RecordReach are removed from @privateaim/server-kit. The server-test-kit actor fakes deny with authup's PermissionError instead of PermissionDeniedError.
• createMasterRealmActor and createNonMasterRealmActor are removed from @privateaim/server-test-kit (use createAllowAllActor / createScopedActor). hasUnboundedReach and assertRecordReach take Pick<ActorContext, 'permissionEvaluator'>. Master-realm clients without unbounded grants are no longer exempt from rate limiting or admitted to foreign realm namespaces.
• provision analysis_read in Authup before upgrading (until then every node save fails), grant it to roles that view analyses, and grant cross-realm roles - including a non-system worker client - their analysis permissions at realm_scope any. Master-realm users are no longer able to read every realm's analyses by realm name alone.

Features

• adopt authup 1.0.0-beta.70 (#1970) (5cdba0f)
• realtime namespaces and rate-limit exemption reach through realm_scope (#1964) (e42cd06)
• bucket uniqueness, transient worker retries, queue-only registry dispatch, dead routings and cache removal (#1962) (f9568e3)
• record writes reach through grant realm_scope (#1961) (eba3ef8)
• server-core: reconcile stuck analysis builds and distributions, judge staleness in core (#1960) (bb6cf17)
• scope the analysis log proxies to analysis reach (#1958) (26a7f79)
• scope analysis buckets, files and node events to analysis reach (#1957) (da20dd5)
• require analysis_read and participation reach for analysis reads (#1955) (9e798d6)

Fixes

• close five regressions of the v0.18.0 to beta.70 range (#1973) (41b6800)
• server-kit: run every entity-event handler when one fails (bc6351f)
• server-core: race-safe process-status transitions (#1954) (68e802b)

Refactors

• migrate from synergos to hubwire 0.2 (#1968) (a051e5f)
• components and message bus come from synergos (#1965) (dbbbbfb)
• server-core: remove TaskManager, correlate bucket replies by their refs (#1959) (710729c)
• server-core: aggregators depend on core ports, not the DataSource (#1956) (7e26b5f)