v0.18.0

Hub 0.18.0 moves to Authup 1.0.0-beta.69, puts reads of catalogs, registry projects and events behind permissions, and moves node approval from the build to distribution. It covers everything since v0.17.0.

📣 Some News

This release closes authorization gaps, and most of them are breaking

v0.17.0 passed complete Authup grants through to the permission evaluator. This release uses them. Reads that were open to any logged-in caller now need a grant. Writes that checked the wrong realm now check the right one. server-messenger, which never authenticated its REST surface, now does. Plan the upgrade with the heads-up table below open: five new permission names must exist in Authup before the new server-core starts, and the worker and storage services must be upgraded before server-core.

👀 Highlights

🔐 Reading catalogs, registry projects and events needs a read grant

Listing or reading nodes, master images, master image groups, registries, registry projects and telemetry events now needs a grant. Each read accepts the new read name or any of the entity's write and manage names, so a role that can already edit an entity can still read it. A caller holding none of them gets 403, with no fallback.

Five permission names are new:

node_read
master_image_read
master_image_group_read
registry_read
registry_project_read

Nodes, master images, master image groups and registries are global catalogs, so the grant is a capability gate and every row stays readable across realms. Registry projects and events are narrowed by the grant's realm scope: own returns the caller's realm, ownOrNull adds realm-less rows, any returns all. This closes the hole where one event_read grant read every realm's audit trail. A record read (GET /<entity>/:id) uses the same scope, so a row out of reach answers 404, the same as a missing one.

The same permissions gate realtime subscriptions to node and master-image updates, and the first segment of an include (/nodes?include=registry needs a registry permission).

Node clients need no manual step. server-core re-grants every existing node client its permissions (analysis_update, node_read) in the background, once at start and then hourly, so pod orchestration, the storage service and the message broker keep reading /nodes.

Warning

Add the five names to your Authup provisioning before upgrading. The built-in admin role (*) and the worker's system client only pick up names that exist when provisioning runs. Then grant node_read, master_image_read and master_image_group_read to every role that creates or edits projects or views analyses, and registry_read to roles that manage a node's registry connection.

Note

Events stored before this release carry no realm, so an own grant does not see them; ownOrNull does. Each background pass also removes any permission a node client was granted by hand, as node saves always have.

📖 Reading nodes, master images, registries, registry projects and events requires *_read

🛡️ Authup beta.69 and the authorization fixes that came with it

Hub now requires Authup 1.0.0-beta.69, a security release: realm-bound tokens, two-stage rate limiting, realm-gated client reads. Hub needed no code change for the bump itself, but your Authup deployment does need attention (Redis key flush, trustProxy, client grants).

Reviewing Hub against it surfaced several gaps of Hub's own, all closed here:

• server-messenger authenticates. Its REST surface attached one fixed master-realm user to every request, so every caller read and acknowledged the same mailbox. Every request now needs a bearer token, including the GET /messages/stream SSE stream. Treat messages stored by earlier versions as exposed.
• Malformed bearers no longer authenticate. Authup up to beta.69 introspected an out-of-bounds token as the introspection request's own Authorization header, which is Hub's service client. Bearer x acted as system. Every Hub service now refuses anything that is not a compact JWS of 16 to 2048 characters before introspecting it.
• Secrets no longer leak through a relation. include=registry&fields[registry]=+accountSecret on /registry-projects, /nodes, /project-nodes and /analysis-nodes returned the Harbor secret. The field is now dropped for every caller.
• Every service fails closed without Authup. The no-Authup fallback (a fixed master-realm user, plus a dryRun mode that granted everything) is removed.
• Status codes are honest. A permission denial answers 403 (was 400 or 500), a dead token 401 (was 500), throttling 429, an Authup outage 502.
• Attaching or detaching a project node needs project_update. The project form hides its node picker from users without it.

# Redis-backed Authup: once after upgrading, so pre-upgrade tokens stop bypassing the realm binding
redis-cli --scan --pattern 'oauth2_token_claims*' | xargs -r redis-cli del

Warning

Rotate Harbor robot and registry secrets, and analysis client secrets: they could be read through the closed relation path. Grant the messenger's service client permission_read, and set AUTHUP_URL, CLIENT_ID and CLIENT_SECRET for it.

📖 Authup upgrade to beta.69 · server-messenger authenticates against Authup · Malformed bearer tokens

✅ Nodes approve the built image, not just the code

Node approval now gates distribution instead of the build. The build starts without any approval. Once it has finished, each node's operators review the image (hash, size, OS) together with the code and approve or reject it. Distribution starts only after every node has approved.

Before, a node approved the code before any image existed, and a node that rejected after the build did not stop distribution.

• Starting a build, including a rebuild, resets every non-aggregator approval and rejection, because a new image needs a new decision.
• Approving before the build is executed answers 400. Re-sending an existing approval still works, and rejecting is possible at any time.
• Starting distribution without every approval fails with AnalysisError.nodesApprovalRequired().

The analysis field moves with the step it gates. There is no alias:

- if (analysis.buildNodesValid) {
+ if (analysis.distributionNodesValid) {
      // every node has approved
  }

A migration renames the column in place, so stored values carry over. In client-vue the "Node(s) Approval" requirement moves from the Build card to the Distribution card, and the approve command is disabled until the image is built.

Warning

Every API consumer that reads buildNodesValid, in fields, filters or response handling, must switch to distributionNodesValid. That includes the Python hub client (flame_hub): use a version that knows the new name.

📖 Analysis Approval

🗂️ Analysis bucket writes check the analysis, not the caller

Creating a bucket file stamped the caller's realm and then checked the caller against it, so the check always passed. Any logged-in caller could register a file in any analysis bucket, including a code bucket after the configuration was locked, and flag it as the root file. Creating an analysis bucket had no check at all.

Buckets and files are now authorized against the analysis's realm:

	Code	Result	Temp
create	analysis realm + analysis_update, unlocked	master realm, an approved node's realm, or the analysis's own client	same as result
update / delete	analysis realm + analysis_update, unlocked	analysis realm + analysis_update	master realm only

Only a code file can be the root file. Uploads that go straight to server-storage are checked when server-core registers them: a rejected code upload is deleted from storage, a rejected result or temp upload is kept but not registered, so a misjudged uploader never costs result data.

Warning

Grant analysis_update to every role that uploads or deletes analysis code.

Note

The direct-upload check is realm-level only, and master-realm uploads that go straight to storage are rejected. Upload through a node or the owner realm.

📖 Writing analysis buckets and files requires analysis_update

📬 Commands are retried, and outcome events wait for server-core

Two message-delivery defects lost work silently:

• The AMQP driver acked and skipped every redelivered message, so a build command got exactly one attempt. A worker crash mid-build lost it. A failed or interrupted command is now re-published with exponential backoff (1s, 2s, 4s, 8s), up to 5 attempts, then dropped with an error log line.
• Every server-core replica bound its own temporary queue for worker and storage events. Every replica applied every event, and events published while server-core was down were lost. The six outcome routings (analysisBuilderEvents, analysisDistributorEvents, masterImageBuilderEvents, masterImageSynchronizerEvents, storageBucketEvents, storageBucketFileEvents) are now durable single active consumer queues: one replica applies each stream in publish order, and another takes over if it goes away.

A late started or progress event no longer moves a finished process back to started, a failed master-image build reports failed instead of finished, and failure events carry the error's name, message and code instead of {}.

A command is acknowledged only when its handler finishes, so RabbitMQ's consumer timeout must outlast your longest build:

# rabbitmq.conf: milliseconds, default 30 minutes
consumer_timeout = 7200000

Warning

Upgrade server-core-worker and server-storage before server-core. An upgraded server-core consumes only the new queues and misses events from a worker that still publishes the old way. After a rollback, delete the six queues before upgrading again: a stale finished or failed event left in them would overwrite the current status.

📖 Message delivery (RabbitMQ)

🕰️ Timestamps are UTC everywhere, and replicas migrate once

On MySQL and PostgreSQL every Hub service, and the migration CLI, pins its database session to UTC, reads created_at / updated_at back as UTC and binds Date parameters in UTC, whatever zone the database server or host runs in. Before, the database stamped in its session zone and the driver read in the process zone, so they agreed only when every clock was UTC.

Boot-time migrations now run under a database lock per service (server-core:migration, …), shared with migration run / revert. Replicas that start together apply the chain once, and a migration job cannot overlap a booting replica. A replica that waits more than 60 seconds fails its boot instead of migrating unlocked.

Check your database's zone before upgrading:

-- PostgreSQL
SHOW timezone;
-- MySQL
SELECT @@global.time_zone, @@system_time_zone;

Warning

If it is not UTC, rows stamped before the upgrade read shifted by that offset. Convert them with every old replica stopped, before the new version starts; the docs have the conversion SQL. A driver setting that contradicts the pin (a MySQL timezone, dateStrings or typeCast, a PostgreSQL TimeZone) now stops the boot, and the connection pool needs at least two connections.

📖 Timestamps are stamped and read in UTC · Replicas and boot-time migrations

🧹 Intermediate files are cleared once every node is done

Once every node of an analysis is idle (executed, failed or stopped), server-core empties the analysis's TEMP bucket. The files go, the bucket stays, so a re-run still has its intermediate storage.

The clear is also available on its own, with the same authorization as deleting the bucket:

// POST /buckets/:id/clear
await storageClient.bucket.clear(bucketId);

Deleting a bucket through server-storage's HTTP API now also reaches server-core, which detaches the analysis bucket instead of leaving a stale row behind.

📖 Analysis Buckets

🔎 Look up an analysis by its Authup client

Analysis containers authenticate to node services with tokens that carry the analysis's Authup client id, not the analysis id. /analyses now accepts that id as an indexed filter, and /analysis-nodes accepts it through the relation:

GET /analyses?filter[clientId]=7c9e6679-7425-40de-944b-e07fc1f90ae7
GET /analysis-nodes?filter[analysis.clientId]=7c9e6679-7425-40de-944b-e07fc1f90ae7

Note

Deploy this Hub before a node-storage-service release that uses the lookup. An older Hub rejects the filter key.

🧩 For package consumers: authup's evaluator replaces Hub's checker

@privateaim/server-kit's ActorContext now carries Authup's IPermissionEvaluator directly, with Authup's method names. There are no deprecated aliases:

- await actor.permissionChecker.preCheck({ name: PermissionName.NODE_CREATE });
+ await actor.permissionEvaluator.preEvaluate({ name: PermissionName.NODE_CREATE });

• preCheckOneOf / check / checkOneOf become preEvaluateOneOf / evaluate / evaluateOneOf.
• buildActorContext is exported once, from @privateaim/server-http-kit, and gives every service the full audit metadata. Telemetry and messenger audit rows used to have no actor, IP, path or method.
• The 'robot' identity type is gone from ActorContext, RequestIdentity and SocketIdentity. Authup identities are user or client.
• scopeReadQuery and appendQueryConditions are new in @privateaim/server-kit for authorized collection reads.
• Tests use FakePermissionEvaluator from @authup/server-test-kit. tokenVerifier is required in the server-http-kit middleware options.

The rate limiter now exempts master-realm clients. The old exemption named 'robot' and never matched, so the worker's system client was capped at 1000 requests a minute.

⚡ Smaller changes

• Indexes. New lookup and composite indexes on the node junctions and analysis bucket files, and master_image_id indexes that make a catalog sync's ON DELETE SET NULL stop scanning analyses and projects on PostgreSQL. The inbox order sort=approvalStatus,-createdAt is now allowed.
• Deleted users now detach their analysis bucket files. The handler queried the wrong table, so their userId stayed behind.
• server-messenger no longer returns expired messages in the gap before the sweep removes them.
• Socket subscribe denials answer 403 instead of 401, and any other failure no longer poses as a denial.
• Log writes that VictoriaLogs rejects are dropped with a warning instead of retried, so an outage cannot pile every service's log lines up in RabbitMQ.

⚠️ Heads-Up Before Upgrading

Change	What to do
Five new *_read permissions	Provision node_read, master_image_read, master_image_group_read, registry_read, registry_project_read in Authup before upgrading, then grant them to your roles
Authup 1.0.0-beta.69 required	Upgrade Authup, flush Redis oauth2_token_claims*, check trustProxy, give Hub's client any scope on client_*
Reply queues are durable and ordered	Upgrade server-core-worker and server-storage before server-core; raise RabbitMQ consumer_timeout above your longest build
server-messenger authenticates	Grant its client permission_read, set AUTHUP_URL / CLIENT_ID / CLIENT_SECRET, send a bearer token from every node
Services fail closed without Authup	Configure Authup for every service; dryRun is gone
Secrets were readable through relations	Rotate Harbor robot/registry secrets and analysis client secrets
buildNodesValid → distributionNodesValid	Update every API consumer, including flame_hub
Approval moved after the build	Tell node operators: approve once the image is built; approving earlier answers 400
Code uploads need analysis_update	Grant it to every role that uploads or deletes analysis code
Project nodes need project_update	Grant it to every role that creates projects with nodes
UTC pin	Check the database zone; convert local-time rows with every old replica stopped
ActorContext.permissionChecker removed	Use permissionEvaluator and Authup's method names
'robot' identity removed	Handle only user and client

⬆️ Upgrading

npm install @privateaim/core-kit@^0.18.0 @privateaim/core-http-kit@^0.18.0

Each service applies its migrations on boot.

📖 Deployment configuration

👉 Changelog

compare changes

🚀 Features

• deps: bump authup to v1.0.0-beta.69 (#1923) (09f55fa)
• server-core: allow filtering analyses by clientId (#1930) (3b942b3)
• adopt typeorm-extension 4.2.0 (UTC timestamps, boot migration locks) (#1934) (eddee85)
• align hub with authup's permission evaluator and close authorization gaps (#1932) (ec2d560)
• clear analysis temp bucket once all nodes are idle (#1939) (5094f2e)
• require read permissions for catalogs, registry projects and events (#1940) (476cb60)
• server-kit: durable ordered reply queues on amqp-extension 5 (#1947) (ec599bf)

🩹 Fixes

• server-core,server-storage,server-telemetry,server-messenger: migrate tsdown bundle to unbundle (0c700d1)
• server-kit,server-core,server-core-worker: bounded command retries and awaited result events (plan 018, phase 1) (#1935) (54c4174)
• server-test-kit: wait for the real mysql server before starting authup (#1936) (03bb068)
• server-core: authorize analysis bucket and bucket-file writes (#1944) (8fce9bb)
• gate distribution, not the build, on node approval (#1946) (67ca4a9)
• release prep: squash unreleased migrations, boot-time node-client permission sync, non-retrying log writes (#1948) (d4e94d9)
• deps: bump the minorandpatch group with 30 updates (#1931) (bee4598)
• deps: bump dotenv from 17.4.2 to 18.0.7 in the majorprod group (#1942) (a68562a)

⚡ Performance

• lookup and foreign key indexes, plus fixes found in the index audit (#1945) (cf8af59)

💅 Refactors

• one buildActorContext, no robot identity, authorization in core (#1941) (ee1bde7)

📖 Documentation

• add rapiq filter-helper convention and lint guard (#1933) (c693472)
• rework README header with animated hub logo (6fb3082)
• add how-it-works, mermaid architecture and fix docker run docs (68fa81f)
• point package READMEs to the hub docs site (#1951) (4c8b123)

⚠️ Breaking Changes

• Reading nodes, master images, master image groups, registries, registry projects and events needs a read permission; event_read is realm-scoped (#1940)
• server-messenger authenticates every request; every service refuses requests without Authup; dryRun is removed (#1932)
• Attaching and detaching project nodes requires project_update (#1932)
• ActorContext.permissionChecker → permissionEvaluator (IPermissionEvaluator); IPermissionChecker, RequestPermissionChecker, useRequestPermissionChecker, FakePermissionChecker, AbstractEntityService.isActorMasterRealmMember, createFakeTokenVerificationData and FakePermissionProvider removed (#1932)
• The 'robot' identity type is removed from ActorContext, RequestIdentity and SocketIdentity (#1941)
• Analysis.buildNodesValid → distributionNodesValid; approving a node before the build has finished answers 400 (#1946)
• Analysis bucket and code-file writes require analysis_update in the analysis's realm (#1944)
• Worker and storage reply routings are ordered WORK queues; upgrade worker and storage before server-core (#1947)