Repository navigation
Releases: BotResources/br-service-engine
Release list
br-service-engine 0.4.1
0.4.1 - 2026-10-08
A patch of 0.4.0: two defects and one additive gap found by the first
services running on the engine. No adopter code changes; a service bumps its
pin. The compatibility entry for this release is chart br-service-engine-chart
2.0.0 ↔ engine 0.4.1.
Fixed
- A led mirror no longer drops a KV change when it runs on two or more pods.
Every pod watches the consumed bucket, and through 0.4.0 every pod took the
mirror's advisory lock without waiting before it checked the lease. A
standby held that lock for a moment on each event only to find it held no
lease; the leader, receiving the same event at the same moment, read the busy
lock as "not the leader", discarded the change it had already recorded as
seen, and the next change advanced the watermark past it. The row stayed
stale until the periodic reconcile (300 s by default) or a restart — a user
deactivated upstream stayed admitted for that long. A pod now reads the lease
first, without any lock: a standby answers "not me" and never touches the
lock; the pod the lease is open to (its own, expired, or never taken) waits for
the lock, then renews the lease and projects. A pod that loses the lease while
it waits logs awarn, and the new leader projects the change from its own
watch. Conformances179holds the lock across a change and loads two pods
with forty changes: every change is projected from the live watch. - Confirming a blob works against a signature-strict S3 store. The
HEAD
that confirms an upload (and that the blob reaper uses) sent
x-amz-checksum-mode: ENABLEDoutside the presigned signature; a store that
refuses an unsignedx-amz-*header on a presigned request answered400, so
every confirm failed withINTERNAL. The header is now part of the signed
headers and sent unchanged, so the store still returns
x-amz-checksum-sha256and the check against the declared digest keeps
working.
Added
- Full scope specs. A host can declare each scope's label and description
i18n keys and itsplatform_onlyflag, and the service's own label and
description keys:ScopeDef::new(key, label_key, description_key)(with
.platform_only()),Engine::contribute_scope_defs(&[ScopeDef]),
Engine::describe_scope_service(label_key, description_key), and
ScopeManifest::of_defs/with_service_labelsfordeclare_scopes. Through
0.4.0 the engine declared every scope's key as both i18n keys and
platform_only = false.contribute_scopes(&[..])andScopeManifest::of
keep that behaviour (ScopeDef::bare). The manifest checks apply to full specs
unchanged, and one key given two different specs is refused with the new
ScopeError::ConflictingScope(an identical repeat still collapses).
Conformances077reads the declaration on the wire for both paths.
br-service-engine 0.4.0
0.4.0 - 2026-09-24
The breaking release after 0.3.4. Paging, the last gesture bound to one pod,
becomes subscription arguments, so any replica serves any subscription.
PassportPrincipal::from_passport no longer sees the database, so a database
fault can never answer 401. The rough edges of the 0.3.1–0.3.4 rollouts are
fixed: a Recreate rollout that changes versions no longer restarts the new pod,
and an error chain renders each cause once. Good practice is built into the
primitives: a batched read is required, one aggregate, a parent's children and
hand SQL are read behind the view's gate or under RLS, a cohort window is read
through the store's cohort index and bounded, never loaded and filtered in
memory, a one-key view fetch reads its one row, a failure a service builds is
logged, and a mirror detects a change on any column. No engine service is in
production; every break has one migration line below. It ships with chart
br-engine-service 1.1.1 (next section); the compatibility entry for this
release is chart 1.1.1 ↔ engine 0.4.0.
Changed (breaking)
- Paging is subscription arguments. The window a session renders is what
the projector'spopulatereturns for the subscription's arguments (size, a
beforecursor, a filter), carried in the view's typedQuery. A window change
is a new subscription and a new session: itsResetat revision 1 carries the
whole new window, including a write committed during the switch. The engine
mints everySessionId, and nothing ties a window to a pod: no session
affinity, no cross-pod relay, any replica count. Thepagemutation, the
paging runtime and the client-chosen session are removed. window_capacitybounds a window at attach, never after. An attach whose
population holds more thanwindow_capacitykeys (default 10,000,
EngineConfig::with_window_capacity) is refused before the render with the new
AttachError::WindowTooLarge { projector, keys_read, capacity }, answered as
WINDOW_TOO_LARGE(WINDOW_TOO_LARGE_CODE); the message names the capacity
only.keys_readis the countpopulatereturned before the refusal:
window_capacity + 1when it binds its ceiling, never the collection's size.
An attach of several windows populates and admits them all before it renders
any. A
whole-collection view (type Query = ()) is refused once its collection
outgrows the capacity. Through 0.3.4 the capacity bounded the pages one session
held and evicted the oldest page.- A one-shot window fetch is bounded like an attach.
Query::fetch_window,
fetch_window_jsonandfetch_view_windowpopulate under the attach ceiling
(window_capacity + 1) and refuse a window abovewindow_capacitywith
WINDOW_TOO_LARGEbefore any row is loaded or rendered; through 0.3.4 they read
and rendered the whole window, however large. The read before the refusal is
bounded whenpopulatebinds its ceiling (cohort_window,full_eda::keys,
cx.limit_all(),cx.limit(&page),ceiling.limit()); apopulatethat
ignores it still reads its whole population before it is refused. - A raw one-key fetch is bounded like a window fetch.
Query::fetchand
fetch_jsondecide membership from the projector's default window populated
under the same ceiling, and refuse withWINDOW_TOO_LARGEwhen that window
holds more thanwindow_capacitykeys, since a truncated window cannot decide
membership. The client has no window argument to narrow, so the message says
that the service must read the key by its row, and the engine logs onewarn
per such refusal naming the projector, the keys read and the capacity (the fix
isfetch_view). Through 0.3.4 they read the whole default population
(KeyCeiling::NONE) on every one-key call. CohortIndex::keys_in_cohorts(conn, cohorts, ceiling: KeyCeiling)takes the
read bound and bindsceiling.limit()as itsLIMIT.view::cohort_window
passes the populate's ceiling, so a whole-collection cohort attach reads at most
window_capacity + 1keys before it is refused; in a debug build
cohort_windowpanics when a store returns more keys than the ceiling.full_eda::keys::<T, P>(cx: &Populate<'_, P>)takes the populate context
instead of a pool and reads at mostcx.limit_all()keys.PassportPrincipal::from_passport(passport) -> Result<Self, PrincipalRejected>
is pure and synchronous: noPgPool, noBoxFuture. The passport alone
decides the one401a principal can cause, with its exact message. Every
database fact goes through aregister_principal_factloader, run on every
POST /graphqland/graphql/wsupgrade and again at each principal refresh
(the refresh never calledfrom_passport, so a fact read there went stale on a
live session). A loader fault answers the coded500INTERNALwith no detail.
A loader has no rejecting outcome: a denial read from local data isFORBIDDEN
or an affordancereasonCode, never a401.max_body_bytesmoves toEngineConfig(with_max_body_bytes,
config::DEFAULT_MAX_BODY_BYTES, same 16 MiB default, still fed by no
environment variable). It bounds every authenticatedPOST /graphqlbody.
EngineConfig::validateis the one judge of the body bounds;
MultipartConfig::validateis removed.MultipartConfig::max_file_bytesis an
Option<u64>: unset, the part bound is 8 MiB ormax_body_byteswhen lower;
set abovemax_body_bytes, boot refuses it and names both settings.- Faults are errors.
MutationFaultandReactionErrorrequire
std::error::Error + Send + 'static, andMutationFault::as_erroris removed.
A mutation fault without a reason is logged aserror::describe(&fault); a
reaction fault's dispatch and dead-letter reason isdescribe(&fault), source
chain included.gate::Reasonis an error whoseDisplayis its code, so
type Error = Reasonstill works. MutationErroris a refusal or a failure, with no public field.
MutationError::refused(reason, detail)takes aReason;
MutationError::internal(detail)is the failure, and it logsdetailat
errorwhen it builds it, so a failure a service builds and forwards with?
keeps its cause in the log; the engine's own failures are logged once where
they occur.reason(),code()andrefusal_detail()answerSomefor a
refusal only. A failure writes only
mutation failedand keeps its internal text as itssource(), which no
accessor hands out: a resolver that forwards it with?answersINTERNAL
with no database or engine text, anddescribe(&error)renders the chain for a
log.?in a resolver converts only an error that says what the client gets.
The engine turns on async-graphql'scustom-error-conversion, and Cargo's
feature unification turns it on for every crate of the build, so the blanket
From<T: Display>intoasync_graphql::Erroris gone.?on agate::Reason
(whatGate::requirereturns) answers its code; on aMutationErrorits
refusal code withmutation refused: <detail>, orINTERNALfor a failure; on
anAttachErrorWINDOW_TOO_LARGE,UNAUTHENTICATEDorINTERNAL; on a
WindowSizeOutOfRangeWINDOW_SIZE_INVALID.?on any other error
(EngineError,sqlx,serde_json,std::io, …) no longer compiles: a fault
goes throughOrInternal::or_internal(context). A bareStringor
&'static strstill converts (async-graphql keeps those two conversions) and
reaches the client with no code: build a refusal withcoded_errorand a fault
withor_internal. Through 0.3.4 a bare?sent
a refusal without its code, and a fault'sDisplaytext (a database message
included) to the client uncoded. The engine does not compile without the
feature: its conversions conflict with the blanket one.- The schema-version claim is internal. The
schema_versionmodule is
crate-private (claim_schema_version,refresh_schema_version,
version_conflict_reasonleave the API); the claim and its handover run only
insideEngine::boot, the refresh only in the beat.EngineConfig::validate
refuses abeatthat is not belowschema_version_liveness, and
establish_transportvalidates its config before any database work. - One batched read per store, one way to read a row.
Persistence::read_manyis required and has no per-key default, so a store
without a batched read does not compile.loadleaves the trait: it is
PersistenceExt::load, implemented once for every store as the one-key
read_many, so the mutation path, render and the gated reads cannot read a row
two ways. - A mirror projection receives its whole batch:
Project::project(cx, keys: Vec<K>)gets every key its lead transaction
touches, one live change's keys or every key of a rescan (boot, periodic
reconcile, leader takeover), so a rescan of N keys costs the statements a
rescan of one costs. Projection::replace_rows(RowScope, rows)overKnownRowrows is the mirror
kit's one row write. It detects a change on any column: one multi-row upsert
per ~30,000 bound values writes a row only when a value column differs, one
NOT EXISTSdelete removes the scoped rows the call did not name, and it stages
exactly the rows inserted, changed or deleted. Rows compare by typed key
values.Projection::replace,upsert,retire,replace_one,remove, the
KnownandKnownScopetraits andWritten::Insertedare removed; hand SQL
goes throughProjection::connplusimpact_*. The change test is
IS DISTINCT FROMon every value column, so a value column needs an equality
operator: a document column isjsonb, neverjson(ajsoncolumn fails
every write, as the 0.3.4 upsert did).KnownRowdeclaresKEYandPRINCIPAL.KEYnames the key columns
key()returns, in order; a write whose key names other columns...
br-service-engine 0.3.4
0.3.4 - 2026-09-24
Engine services become reachable for subscriptions through the gateway. The
gateway reaches a subgraph for a subscription only as POST /graphql with
Accept: text/event-stream (graphql-sse over HTTP), never over a WebSocket;
through 0.3.3 the engine served subscriptions on GET /graphql/ws alone and
answered that request with the JSON error "Subscriptions are not supported on
this transport.", which the gateway reports as SUBGRAPH_REQUEST_ERROR — seen on
dev for every engine service. Additive: no adopter code changes.
Added
- The gateway's subscription transport on
POST /graphql. A request whose
Acceptnamestext/event-streamis authenticated and its body read exactly as
before — the passport from the headers first (401with the body never read,
the same status and body a JSON request gets), then the bounded body — and is
then answered as a graphql-sse "distinct connections" stream (graphql/sse.rs):
Content-Type: text/event-stream,Cache-Control: no-cache, oneevent: next
per GraphQL response (data:the response JSON on one line), one
event: complete(data:empty) at the end, and a:comment after 15 s
without a frame. Any operation may be sent this way: a query or a mutation
answers onenext, thencomplete; a coded refusal rides anext, then
complete, as on the WebSocket. A response that cannot be serialized is logged
and sent as a codedINTERNALresponse, never as an empty frame. A request
withouttext/event-streaminAccept(a wildcard does not count) is answered
as before. - Bounded like a WebSocket session, on the same session machinery. The stream
runs the same schema, so a subscription attaches the same engine session as on
/graphql/ws: sameReset/Upsert/Removewire and revision, same cohorts,
same principal-facts refresh. Atsession_max_age, measured from the request,
and when the engine shuts down (the signal that closes the WebSockets), it sends
completeand ends, so the gateway re-subscribes with a freshX-Passportand
an open stream never holds graceful shutdown; a stream requested during shutdown
answerscompletewithout executing. A client that goes away drops the
response, and with it the operation and its session. - Conformance:
bb16(black-box, the realexample-servicebinary) — over the
gateway's transport a subscriber gets theResetthen theUpserton a
contiguous revision, read withbr-test-harness'sSseSubscription(the client
service e2e suites use); a passport that is absent, undecodable or not a
passport is refused with the very401a JSON request gets, before a byte of
the body is read (a body the client never finishes sending is answered at once,
while an authenticated control is waited for); and pastsession_max_agethe
stream sendscomplete, ends, and a new request attaches afresh. TheReset
and max-age scenarios fail on 0.3.3 with the dev error; the401scenario
pins that the refusal still precedes the transport choice. Unit tests pin the
framing, a query over the stream, a refusal as anext, the keep-alive
cadence, the max-age and shutdown completions, the refusal to start during
shutdown, the release of the operation when the response is dropped, and which
Acceptvalues ask for a stream.
Changed
conformance-service-enginedev-depends onbr-test-harness1.2.0 (the
br-e2e-harnessrepository,ssefeature only);deny.tomlallows that git
source.- README: the ops contract names the WebSocket route
/graphql/ws(it read
/ws) and documents both subscription transports; the black-box command list
namesbb12tobb16.
Adopter migration
- None. Move the pin to
v0.3.4:appserves the event stream with no new
configuration and no environment variable, and a service's schema and SDL are
unchanged. Its subscriptions become reachable through the gateway as soon as
the new image rolls.
Known limitation
- Paging a gateway (SSE) session requires the session's pod. The
page
gesture serves only a live session this pod holds. Through the gateway the
session rides an event stream and thepagemutation is a separatePOST
that may reach another replica, where it is refused
(EngineError::NoLiveSession,NOT_FOUND). Paging a gateway (SSE) session
requires the session's pod (one replica) until 0.4.0 moves paging to
subscription arguments.
Engine services run one replica today.
br-service-engine 0.3.3
0.3.3 - 2026-09-23
A security patch over 0.3.2. POST /graphql now resolves the passport before it
reads the request body, and then reads every body within explicit bounds: its size
(JSON as well as a GraphQL multipart request) and the time it takes to arrive. No
adopter code changes; one new optional configuration group and one timeout, both
with defaults.
Security
- An unauthenticated multipart request could make the pod write to disk.
Through 0.3.2 the handler took async-graphql-axum'sGraphQLRequestextractor,
which parses the body — and, for a GraphQL multipart request (operations+
map+ file parts), spools every file part to a temporary file — before the
handler looked atX-Passport. Any client that reached a pod could therefore
fill its writable filesystem (on a read-only root the request failed400and
nothing was written), and a JSON body of any size was buffered in memory before
the same401. The parser also bounded nothing: no body, file or file-count
limit, and a file part spooled even whenmapdid not name it or the schema
declared noUpload. Found while verifying the engine under a read-only root
filesystem (chartbr-engine-service1.1.0, PR #17). - The fix: authentication first, then a bounded read. The handler resolves the
principal (passport decode,PassportPrincipal::from_passport, principal facts)
from the headers alone; a request whose passport is absent, does not decode or
is rejected is answered401with its body never read, whatever its content
type. The401status and body are those JSON clients already received;
principal facts the pod cannot load are the500INTERNALof 0.3.2, now also
answered before the body is read. Only then is the body read: amultipart/*body through the engine's own receiver
(graphql/multipart/), every other body streamed to the function
async-graphql-axum's extractor calls, so it is parsed exactly as before (same
content-type dispatch, same single-request rule, same rejection). - An authenticated body is bounded in memory and in time. Through 0.3.2 an
authenticated JSON body of any size was buffered whole, and a client that sent
its body slowly (or never finished it) held the request, its buffer and any
spooled file open for as long as it kept the connection. Every authenticated
body — JSON and multipart alike — is now capped bymax_body_bytes(a declared
Content-Lengthabove it is refused before the body is read, a chunked body is
cut on the chunk that crosses it) and must arrive whole within
body_read_timeout; past it the read is abandoned and whatever was received,
buffered bytes or spooled files, is dropped with it.
Added
- Multipart bounds —
EngineConfig::multipart: MultipartConfig
(with_multipart; the type atservice_engine::andservice_engine::graphql),
validated at boot:
max_body_bytes(default 16 MiB; a declaredContent-Lengthabove it is refused
before the body is read, a chunked body is cut),max_file_bytes(default 8 MiB,
any single part),max_files(default 4, the uploadsmapbinds — every path
counts;mapitself may weigh at most 1 KiB per allowed upload plus 1 KiB, so a
paddedmapis refused before it is parsed),spool_dir(default unset:
std::env::temp_dir(), i.e.$TMPDIR, else/tmp). No environment variable is
added to the ops contract. - The receiver enforces the spec's order (
operations,map, then the filesmap
names), so the upload count and each file part's binding are judged before
anything is spooled; a partmapdoes not name is refused unspooled. A schema
that declares noUploadscalar (read from its SDL) accepts no file (its
effectivemax_filesis 0): multipart stays accepted, and such a service never
writes to disk. Files spool to anonymous temporary files, freed when the request
ends. - Coded refusals, a GraphQL-shaped body with
errors[0].extensions.code:
413 MULTIPART_TOO_LARGE,413 MULTIPART_FILE_TOO_LARGE,
413 MULTIPART_TOO_MANY_FILES,400 MULTIPART_MALFORMED,
500 MULTIPART_SPOOL_UNAVAILABLE(the path and the OS error are logged, never
returned). The codes are public constantsgraphql::MULTIPART_*_CODE. - Every authenticated body —
MultipartConfig::max_body_bytes(default
16 MiB) now bounds a JSON body too, refused413 BODY_TOO_LARGE; and
EngineConfig::body_read_timeout(with_body_read_timeout, default 30 s,
config::DEFAULT_BODY_READ_TIMEOUT, validated non-zero at boot) bounds the time
from the passport resolving to the last byte of the body, for every content type,
refused408 BODY_READ_TIMEOUT. Both are GraphQL-shaped like the multipart
refusals; the codes are public constantsgraphql::BODY_TOO_LARGE_CODEand
graphql::BODY_READ_TIMEOUT_CODE. No environment variable is added. - Conformance:
s241— an unauthenticated multipart request (passport absent,
undecodable, or not a passport) is401before a byte of its body is read,
proven on a body the client never finishes sending (an authenticated control
shows the same stalled body is waited for) and on a spool directory an
authenticated control shows the request would otherwise reach; an authenticated
upload within bounds reaches the resolver intact; each bound — declared and
chunked body, part, upload count — and the malformed order are refused with
their code; a schema withoutUploadaccepts multipart without files and
refuses a file.s115pins the anonymous JSON401and its body. Unit tests pin
every bound (fed in small reads, so a bound trips mid-part), themapweight,
the order rules, the no-Uploadpolicy and its SDL detection, the spool failure,
the content-type dispatch (any case ofmultipart/form-datareaches the bounded
receiver; an unparseable type is refused unread) and the config validation. - Conformance:
s242— an authenticated JSON body overmax_body_bytesis
413 BODY_TOO_LARGE, by its declared length (answered at once, the body never
awaited) and as a chunked body cut in flight, while an anonymous oversized body
is still401; a JSON body and a multipart body stalled inside a file part being
spooled are408 BODY_READ_TIMEOUTat the read timeout, not before it, leaving
no named file in the spool; after every refusal the pod serves JSON and uploads
within bounds. Unit tests pin the declared and streamed JSON bound (nothing past
the crossing chunk is read), the inclusive limit, the deadline for both body
kinds, and the rendered codes.
Changed
- An anonymous request is now refused before its body is parsed, so an anonymous
request whose body is also malformed answers401where 0.3.2 answered400. - A multipart request must follow the spec's part order and bind every file part
it carries (0.3.2's parser accepted parts in any order and silently spooled
unmapped ones); amultipart/*type other thanform-datais400 MULTIPART_MALFORMED. - An authenticated JSON body over 16 MiB is refused
413 BODY_TOO_LARGE(0.3.2
buffered any size), and an authenticated body that has not fully arrived 30 s
after its passport resolved is refused408 BODY_READ_TIMEOUT(0.3.2 waited with
no end).
Adopter migration
- None required. A service whose schema declares
Uploadand runs under a
read-only root filesystem mounts a writableemptyDir(with asizeLimit) at
/tmp— or at itsMultipartConfig::spool_dir— through the chart's
extraVolumes/extraVolumeMounts; no engine service declaresUploadtoday
(blobs use presigned URLs), so none needs it. A service that expects larger
uploads raises the bounds withEngineConfig::with_multipart. - A service whose clients send JSON documents over 16 MiB inline raises
MultipartConfig::max_body_bytes(the name predates its reach: it bounds every
body); one whose clients reach the pod unbuffered over slow links raises
EngineConfig::with_body_read_timeout. Behind the gateway, whose nginx buffers a
request body before it proxies it (its default), the defaults leave a wide margin. EngineConfiggains themultipartandbody_read_timeoutfields (the struct is
#[non_exhaustive]).
Follow-up (documented, not implemented)
- No limit on concurrent uploads. The bounds above are per request: an
authenticated client can keep several uploads spooling at once, each up to
max_body_bytesfor up tobody_read_timeout. No engine service declares
Uploadtoday, so no pod spools anything; the first service that adopts
Uploadadds a concurrent-upload limit (a spool semaphore, with its own coded
refusal) sized together with itsemptyDirsizeLimit.
br-service-engine 0.3.2
0.3.2 - 2026-09-23
A patch: migrate upgrades a database that the 0.2.0 engine migrated, and every
failure the engine answers reaches the client with a code.
Adopter note: nothing to do. Bump the pin. A database migrated by engine
0.2.0 (svc-runners 0.1.0 on the dev stage, svc-accounts) now upgrades on the next
migrate; before this release migrate on 0.3.0 or 0.3.1 exited 1 there with
VersionMismatch(9113000023) (sqlx migrate info: installed (different checksum)).
Fixed
-
migrateadopts the checksum a released engine applied for a migration that
was later edited. v0.3.0 (e667eb9) removed the four-line header comment of
9113000023_mirror_stream_identity.sql, which v0.2.0 had applied; sqlx hashes
the whole file, so every database 0.2.0 migrated refused the 0.3.x engine set
(ignore_missingdoes not cover a changed checksum). Audit of every engine
migration acrossv0.1.0,v0.2.0,v0.3.0,v0.3.1andmain(and every
commit onmain):9113000023is the only file whose bytes changed —Migration v0.1.0 v0.2.0 v0.3.0 / v0.3.1 / main 9113000001–9113000022shipped same bytes same bytes 9113000023_mirror_stream_identity.sql— SHA-384 74808ac1…daae46fSHA-384 ea7d3984…d84720fd9113000024,9113000025— — added schema/released.rsholds a data table,EDITED_AFTER_RELEASE: &[(version, &[released SHA-384])], with the one entry9113000023 → [v0.2.0's checksum].schema::migratetakes the sqlx migrator's own advisory
lock on one connection, rewrites in one transaction every stored checksum that
the table registers for its version to the embedded file's checksum, logs each
atinfowith the version and the old and new checksums, then runs the
migrator on the same locked connection. An unknown checksum is never touched
and still fails asEngineError::Migrate(VersionMismatch(v)), exactly as
before. Only the versions in the table — all inside the engine's reserved band —
are read or written, so library and service rows on the shared
_sqlx_migrationsledger are never touched. A failed run closes its connection
instead of pooling one that still holds the session advisory lock. -
Every failure reaches the client with a code (constitution principle 22).
A mutation whose transaction did not begin or commit answered an uncoded
mutation failed: database; a query, a subscription attach or a page that
failed answered an uncoded message too. Now everything the client cannot fix —
a database or NATS fault, a begin or commit failure, an engine wiring fault, a
handler error whoseMutationFault::reason()isNone— isINTERNAL
(extensions.code), with the fixed messageinternal errorand no database
detail. The few failures a client can act on keep a specific code:NOT_FOUND
for a page on a session the caller does not hold or a window it never attached,
CONFLICTfor an attach under a session id a live session holds,
UNAUTHENTICATEDfor an attach whose principal no longer exists. A failure to
load the principal's facts is a500with the sameINTERNALbody, no longer a
401carrying the error text. Refusals that carry a reason are unchanged: same
code, samemutation refused: …message. -
The engine's logs keep the whole cause chain.
EngineError::Dbdisplayed
database,Migratemigrations,Serviceservice(andCronError::{Db, Job},RelayError::{Db, Relay}likewise), so every%errorlog and every
adopter that logged an engine error saw the label, not the cause. These pure
wrappers are now#[error(transparent)]: theirDisplayis the wrapped error's
and the chain does not repeat. Every engine log site that logged an error by
Displaynow logs its wholesource()chain, as do the conversions to text the
engine stores or forwards (dispatch errors and dead-letter reasons, object
storage and NATS details). Where the engine converts a failure for the client it
logs the chain aterrorfirst: an engine fault in the mutation pipeline with
its context, a handler fault without a reason with the mutation name (and the
fault's own chain when it returnsSome(self)from the new
MutationFault::as_error), a query, attach or page fault with its context.
Added
graphql::{INTERNAL_CODE, INTERNAL_MESSAGE, NOT_FOUND_CODE, CONFLICT_CODE, UNAUTHENTICATED_CODE},graphql::internal_error(context, &cause)(logs the
chain, answersINTERNAL) andgraphql::internal_fault(detail)for a fault
with no error value behind it.MutationFault::as_error, a provided method (defaultNone): a fault that is a
std::error::ErrorreturnsSome(self)so the engine logs its chain.error::describe(&dyn Error) -> String, thetop: cause: rootrendering the
engine logs, for a service that logs an engine error or keeps it as text.- CI guard
.github/scripts/check-released-migrations.sh, run in the
fmt-clippy-testjob (checkout nowfetch-depth: 0): it exports the engine
migrations of everyv*tag and fails when a released file is gone from HEAD or
differs from it without its released SHA-384 registered in
EDITED_AFTER_RELEASE. Every tag is checked, not only the last, so a registered
checksum cannot be dropped silently. s241(conformance, in-crate, real Postgres): a store migrated by the real
v0.2.0 engine files plus a library and a service set upgrades through the
migration chain, its engine ledger then equals a fresh store's and its library
and service rows are untouched, and a secondmigratechanges nothing; a forged
unknown checksum still fails withVersionMismatchand is left as it was; a
fresh store migrates as before. The first scenario fails with
Migrate(VersionMismatch(9113000023))on 0.3.1.
Changed
- The
DisplayofEngineError::{Db, Migrate, Service}is now the wrapped
error's message, database text included. A service that forwards an engine
error's text to its clients should answerINTERNALinstead; both platform
adopters already do (svc-runners renders its code, svc-accounts' reasonless
faults now reach the client asINTERNAL). example-service'sAppFaultis the reference shape:NotFoundcarries the
reasonNOT_FOUND(a refusal the client can act on, no longer an uncoded
failure),Storekeeps the engine error's whole chain as text (rendered with
error::describe, so the type keeps its auto traits) andas_errorreturns
Some(self); itsReactionFaultkeeps the chain the same way.- The engine now owns one lowercase-hex encoder (
hex::lower); the four copies in
blob signing, blob digests, seal hashes and NATS subject tokens use it, and the
unwrapin the subject-token copy is gone.
br-service-engine 0.3.1
0.3.1 - 2026-09-23
A patch over 0.3.0: a mirror accepts the key grammar a published language really
uses, and migrate asserts the owner posture the ops contract already required.
Both are additive for an adopter on 0.3.0 — every /-terminated consumption and
offer keeps its behaviour and its manifest key — and the second only refuses a
deployment that was already outside the contract.
Fixed
-
A mirror consumes a
.-terminated prefix and a single exact key. 0.3.0
refused at registration everyConsumed::PREFIXthat did not end in/
(invalid configuration: mirror … consumes prefix … which must end with '/' so its manifest key sits outside the data prefix), so a service consuming a producer
whose keys use.as the segment separator, or one singleton configuration key,
could not boot. The rule protected the manifest placement:manifest_key(prefix)
strips the trailing separator and appends_manifest, and under the mirror's
starts_withmatching a prefix that ends mid-segment (catalog/items) would take
its own manifest (catalog/items_manifest) and any sibling sharing its
characters as data. 0.3.1 parsesConsumed::PREFIXonce into one of two forms
(mirror/scope.rs, crate-privateKeyScope):- a prefix ends in a segment separator,
/or.(catalog/items/,
catalog.item.); the mirror reads every key that starts with it, and its
manifest key is the sibling{prefix minus the separator}_manifest
(catalog/items_manifest,catalog.item_manifest), which never starts with
the prefix, so a manifest is never read as data and a data key is never judged
as a manifest; - any other string is one exact key (
catalog.settings), read with one GET at
scan and matched by equality at watch — a key beneath it or sharing its
characters is not read. A single key is judged against no manifest: only an
engine offer writes a manifest, and an offer publishes a family under a prefix,
so no engine producer ever writes one beside a single key. A non-engine
producer's single key is versioned throughwire_version(N9), as before.
Mirror::validaterefuses, asEngineError::Confignaming the mirror and the
string, only the forms that are ambiguous or malformed: a string ending in_or
-(a prefix cut mid-segment — it would otherwise silently read as a key nobody
publishes), a single key ending in_manifest(it would read an offer manifest as
data), a leading separator or two separators in a row (an empty segment), and a
character outside[A-Za-z0-9_./-].require_key::<C>(key)accepts the consumed
single key itself; a key beneath a single key isRequiredKeyOutsidePrefix. - a prefix ends in a segment separator,
-
An offer publishes under a
.-terminated prefix too.register_offerapplies
the same grammar throughmanifest_key, so a producer and its consumers compute
one manifest key from one rule; an offer still refuses a single key. Becausex/
andx.share the manifest keyx_manifest, an engine refuses a second offer
whose manifest key another registered offer already claims; across services the
manifest carries itsprefix, so such a collision reads as
ManifestMismatch::Offerand dead-letters instead of passing silently. -
manifest_key(prefix)(public) accepts a.-terminated prefix and refuses a
single key or a malformed prefix with a message that names the form.
Added
migrateasserts the owner posture. Constitution principle 30 and the ops
contract require the owner role to carryBYPASSRLS: a data migration run by a
role subject to row-level security touches no row of aFORCE ROW LEVEL SECURITY
table and still reports success. 0.3.0'smigratenever checked it, so an adopter
had to add its own guard.engine::boot::assert_owner_posture(&PgPool)(public)
passes a superuser or aBYPASSRLSrole and refuses any other with the new
EngineError::OwnerSubjectToRls { role };apply_migration_chaincalls it right
after the pure library validation and before the first migration, somigrate
(the chart's init container) logs the refusal and exits non-zero with nothing
applied. The engine's own black-box and example harnesses now declare their owner
rolesBYPASSRLS, as production does.- Conformance:
s238(a.prefix and a single key read exactly their keys at scan
and at watch, with sibling, child, near-miss and manifest keys published beside
them),s239(a.prefix is judged against its sibling manifest, a single key
against none),s240(the owner posture matrix, and the chain refusing before it
applies anything),bb15(the real binary'smigraterefuses an owner without
BYPASSRLSand exits non-zero). Unit tests pin the grammar, the manifest
placement for every accepted form, the refused forms, and the offer-side claim.
Adopter migration
- mirror (additive): a consumption of a
.-separated producer or of one
configuration key now registers as written — declareConsumed::PREFIXas the
prefix with its trailing.or/, or as the exact key. Nothing changes for a
/-terminated prefix. A string without a trailing separator is now one exact key:
if you meant a prefix, add the separator (0.3.0 refused that string, so no 0.3.0
adopter reads differently). - migrate (fail-loud): declare the owner role
BYPASSRLS(or runmigrateas a
superuser) — the ops contract already required it, and amigratethat now exits
withOwnerSubjectToRlsnames the role to fix in GitOps. A service-owned owner
posture guard inmainis redundant and can be deleted. EngineErrorgainsOwnerSubjectToRls(the enum is#[non_exhaustive]).
br-service-engine 0.3.0
0.3.0 - 2026-09-22
lane: reset
2. Reset paths — the reported race was not reproduced
The reported "lost delta" race — a queued delta not yet read, then a projector
reset that hands back a stale view — was not reproduced on 0.2 (s196 green on
0.2). render/deliver.rs writes every rendered upsert/remove into last_sent
before discarding it under a reset, so a reset always carries the latest rendered
view; the projector-reset path re-renders the dirtied members from the store, so
the Reset carries the committed view, never the stale one the unread delta
held. The three reset paths (deliver.rs incremental, repair.rs::resnapshot,
connect.rs attach) were left as is — collapsing them onto resnapshot was
not rated worth the churn on the delivery hot path this release. Every reset
still costs a full window render — watch service_engine_resets_total. No API
change; the write-set check this item once carried is dropped in favour of the
change-detecting mirror kit (see N3., lane: mirror).
12. Dead-letter at the render choke point
A projection failure is now dead-lettered at every render entry point, not
only during a normal render pass. The Renderer records the poison document
(DeadLetterSource::Render, keyed on projector plus key, deduped) the moment
project fails, so the attach snapshot (runtime/connect.rs), the page render
(runtime/paging.rs) and the repair re-snapshot (render/repair.rs) all land
the ops row that only the pass path recorded before — those three built a
dead-letter-less renderer. render/pass.rs::dead_letter_projection is gone; the
choke point is the Renderer itself. No API change.
lane: write
3. Transition-aware policies, delete policies
Aggregate now requires Clone, and the pipeline retains the loaded image of
every aggregate it loads. A post-save policy takes Saved<'_, A> { prior, next, events } instead of the bare saved aggregate: next is what was just saved,
prior is the stored image (None on a create), events its pending events, and
Saved::transitioned(|a| …) reports whether a projection of the two differs. The
same shape drives a new post-delete policy: register_post_delete_policy::<A>
/ require_post_delete_policy::<A>, run before a hard delete, refusing or allowing
it exactly like a save policy. One PostSave context serves both hooks; the seams
are declared and boot-checked together (EngineError::UnhonouredSeam).
N5. Persistence::delete and cx.delete
Persistence gains delete(conn, key), defaulting to a refusal
(EngineError::DeleteUnsupported) so a store that never deletes writes nothing.
cx.delete(&aggregate) runs the delete policy, issues Persistence::delete, then
stages the aggregate's offer and blob reconciliation; a refusal or a delete failure
rolls the transaction back like a save. A raw DELETE outside cx.delete is a
lock-domain violation (see N6), never an engine path.
6. Offer trigger types
OfferTrigger<O>: Aggregate with key_from(&self) -> Result<KvKey, EngineError>
(the offer's KvKey) and row_key(&self) -> Result<…::Key, EngineError> (the offer
row's typed store key the drain loads), registered by
Engine::register_offer_trigger::<O, T>() (refused when T is the offer's own
row). A change to a trigger aggregate stages the offer's dirty key, so the offer
re-publishes when a fact it derives from — not its own row — changes; because
row_key is typed as the offer row's store key, a trigger on a different
aggregate can no longer silently retract the offer by staging its own key. The
trigger must live in the offer's slice. Offer gains const VERSION: u16 = 1.
Retires the hand-written dirty_service class. Additive.
15. Engine-owned integration outbox
OUTBOX_TABLE is now service_engine.integration_outbox, created by the engine
migration 9113000024 and owned like every service_engine.* table; the six SQL
sites read the constant. A migrating pod adopts any legacy public.integration_outbox
rows through adopt_legacy_outbox, an idempotent post-migration step (not folded
into the migration: the fresh-database order would leave an orphan public table),
which drains the rows and drops the legacy table. The whole adoption runs in one
transaction holding a constant pg_advisory_xact_lock, so concurrent migrate init
containers on a multi-replica roll serialize: the second pod waits for the first to
commit, then sees no legacy table and returns without a double-DROP crash. The
sample and example outbox DDL is gone.
N2. cx.create refuses an existing key
cx.create now loads under the aggregate advisory lock and refuses an already-held
key with EngineError::KeyReused, surfaced as the stable code KEY_REUSED, instead
of overwriting or hitting a raw unique violation. Two concurrent creates leave exactly
one winner. Removes the principle-5 guard every service hand-wrote.
N6. One lock domain; Persistence::row_lock
Persistence::row_lock(conn, table, key) is a defaulted helper issuing the common
SELECT … FOR UPDATE on the aggregate row — the one-line lock implementation for a
store that wants a row lock beside the engine's advisory lock. Persistence::lock
keeps its no-op default (the advisory lock already serializes every pipeline load).
The one-lock-domain rule holds: every write to an aggregate's rows goes through
cx.load/cx.save/cx.delete, which take the per-key transaction advisory lock,
so two commands on one aggregate serialize even with the no-op lock.
lane: boot
4. The boot kit splits into three argv entry points behind one run_service /
BootPlan. engine/boot.rs becomes engine/boot/{mod,migrate,serve,env}.rs.
migrate runs under the owner role, from DATABASE_URL_OWNER only (no fallback
to DATABASE_URL), retries the owner connection with backoff up to
EngineConfig::migrate_connect_timeout (default five minutes), applies the engine
set then the service set on one shared _sqlx_migrations ledger — the kit sets
ignore_missing on both migrators — waits for the app role to exist, then
grant_engine_access + grant_app_access, and exits. serve runs under the app
role: it refuses to run — EngineError::MigrationsPending { engine, service }, logs
REASON_MIGRATIONS_PENDING and exits non-zero before it binds any port — while
either set is unapplied (checked through the app role against the shared ledger), then boots
the engine and serves. schema prints the SDL and touches no infra. run_service
dispatches on argv (migrate / serve / schema; no argv serves).
EngineConfig::from_env() reads the ops contract in one place and dispatches on the
same argv the boot kit does: serve reads the whole app-env group — ENGINE_CHANNEL,
HOSTNAME (replaces POD_ID), PORT/HOST (replace HTTP_ADDR), NATS_URL,
APP_ROLE, and the optional session/lease/beat timings; migrate reads only
APP_ROLE (placeholder channel/pod, default connect timeout) so the migrate init
container needs the owner Secret and APP_ROLE alone, never the serve env; schema
reads nothing. A service main reads no engine env var by hand; BootPlan.config is
from_env()? plus the service's own with_service / with_blob_storage /
with_session_ttl. BootPlan
loses nats_url and app_role (both now read by from_env into EngineConfig).
with_edge_observability is crate-private: serve is the one boot door, and no
observability helper is re-exported at the crate root.
N1. serve derives message_retention from the bound streams' max_age
(inbound::derive_message_retention, Nats::stream_max_age) after the reactions
are registered, taking the max over the subscription streams and refusing an
unlimited stream. A service no longer sets with_message_retention by hand, and the
pre-boot second NATS connection every adopter opened to read the stream max_age is
gone. An explicit with_message_retention still wins when larger. The existing
retention-mismatch boot check (s174) is unchanged.
N11. Shutdown is latching, so a stop can no longer be lost. The engine's internal
stop signals were bare tokio::sync::Notify: notify_waiters() wakes only the tasks
already parked on the signal and stores nothing, so a worker that had not yet reached
its first poll when shutdown was raised never saw it, and run then blocked forever
joining it. A pod shut down shortly after boot could hang until the kubelet's
terminationGracePeriod expired and SIGKILL landed; the scheduled-message loop was
the one this bit most often, because it is spawned last and joined first.
service_engine::stop::Stop replaces Notify on every internal stop path — an
atomic latch beside the notify, where stop() raises the latch before waking and
stopped() reads it on its first poll — so the signal is level-triggered and the
order of stop and first poll no longer matters. Public signatures that take a stop
handle move from Arc<Notify> to Arc<Stop>: AccumulatorRuntime::run,
Beat::run, MirrorSupervisor::start, SessionRuntime::run and graphql::serve.
Engine::shutdown_handle() is unchanged.
14. readiness re-exports Readiness / ReadinessHandle / readiness_route
from br-util-axum-readiness = "v1.3.0"; the engine holds no copy. Paths are kept,
and the shared crate's readiness: UP / readiness: DOWN tracing wording — the one
the black-box battery greps — is the wording the engine now emits.
lane: chart
- Library chart
br-engine-service, ops contract v1. The engine now
publishes the shared deployment topology as a Helm library chart
(charts/br-engine-service/,type: library) on its own version line starting
at1.0.0— the crate version appears nowhere in the chart. Named templates
render the topology from a thin chart's values:br-engine-service.deployment
(Recreate; init containermigrate= the service image with argvmigrate
a...
br-service-engine 0.2.0
0.2.0 - 2026-09-16
This release answers ws-cc-platform#126. The services-rewrite experiment and
the Runners adoption proved the engine composes across independently-authored
slices but leaks at the seams: every serious defect was cross-slice wiring a
slice was meant to call and never did. 0.2 makes those seams declarative —
a missing interlock is now a loud boot error, not silent nothing — closes the
contained render/parse/observability bugs, reworks the mirror so that empty is a
converged state, and proves the multi-pod fleet behaviour that was untested. The
entry is organised by the issue's items; every public API change is listed with
its one-line fix under Adopter migration at the end.
A1. Emission::Coalesced no longer drops the delta cause
The coalesced branch of the render plan hard-coded cause: None, so a coalesced
view's subscribers received Upsert/Remove deltas with no causal attribution
even when the impact carried a FactRef. A coalesced delta now carries the cause
of the last impact it folds (the latest in the frame that carries one);
causeless impacts contribute none, and a fold with no cause at all stays None.
This makes coalesced-with-cause — one delta per key carrying the causing fact —
expressible, which PerImpact could not stand in for. Bug fix only, no API
change.
A2. The boot gate parses the composed SDL with the real GraphQL parser
SchemaSlices verification walked the SDL line-by-line and was blind to """
block strings, so a root-field description that wrapped onto a line beginning
word ( or word: was read as a phantom root field and aborted boot with
EngineError::UndeclaredSchemaMember. The gate now parses the composed SDL with
the real GraphQL parser, so a wrapped doc-comment can no longer abort boot.
SchemaSlices::verify_root_fields(sdl) is renamed SchemaSlices::verify(sdl).
New EngineError::SchemaParse is raised if the composed SDL cannot be parsed
during the gate.
H0. Mirror rework — empty is a converged state
Per the amended doctrine (docs/service-engine/intent.md): a consumed prefix
that reads empty is a converged prefix with nothing in it — known_* follows
the source to empty, at boot or during a run — and a replaced bucket is
first-adoption, never a readiness failure and never an operator SQL. This
withdraws the 0.1.0 rule that an empty consumed prefix held readiness DOWN, which
made the first deploy of every consumer fail (every producer's bucket is empty
until it publishes). A mirror no longer judges the producer's content, no longer
validates the producer's bucket history (max_messages_per_subject), and drops
the ALLOW_EMPTY flag, the required marker, the stream-identity equality gate
and the history check.
- Per-bucket stream identity + boundary watermark. A mirror persists the
consumed stream's creation identity and the last sequenceSits read reached
beside the watermark, committing both with the projections in one transaction.
The boundary is taken from freshget_info()metadata before the first scan;
every scanned entry is applied and the watch resumes atS + 1. A standby
reports converged once the leader has committed the boundary the standby's own
boot read captured, with no broker round-trip per beat. Migration
9113000023_mirror_stream_identity.sqladds the nullable column; existing
watermark rows keep their revision and adopt an identity on their next read. - A bucket whose stream identity changed, or whose sequence is below the held
watermark, is the first-adoption case: full read, reconcile, adopt the new
identity and boundary. It never holds readiness DOWN. - Reconciling the persisted projection keys against the snapshot is now the one
behaviour of every scan, with no opt-in; on a watch event the engine keys the
change against the shadows both before and after it is applied, so a retract
whose projection key lived only in the retracted payload still reachesknown_*. - No NATS round-trip inside the leader transaction, and no
STREAM.INFOper beat
while a standby waits: a slow broker no longer pins the advisory lock and the
leader_slotrow. - A mirror resuming a bucket from sequence zero opens a future-only watch, then
re-reads the bucket's metadata after the subscription exists and reconciles if
the bucket gained a sequence — closing the window on the boot where every
consumer's bucket is still empty. - A standby no longer stalls a rolling upgrade on a leader watermark that carries
no identity (a pre-column row is compared by sequence alone and takes an
identity when this pod holds the lease), and a leader holding an identity the
standby never read is an adoption (re-read once and wait), not an
EngineError::Configthat would count against the supervisor's restart budget. - A single mirror watch that ends is logged and reopens the watches, instead of
being dropped silently byselect_alland leaving that bucket unwatched. - A shadow keeps the revision it holds and refuses an older one, so the overlap
between the scan and the watch can never write a stale value over a newer one.
NewShadows::put_at/remove_atcarry the revision guard (put/removekeep
their 0.1.0 unconditional signatures);KvBucket::entries_with_revisionsreads
the revision per key. /readyznames the mirror and its own failure after the fixed operator copy,
read from one sample of the health board rather than two.
H1. Boot kit — one call from main (subsumes A3)
run_service(BootPlan { .. }) (re-exported at the crate root) is the one call
from a service main. It installs structured JSON logging
(br-util-observability::init_logging), short-circuits a schema argv
subcommand by printing the composed SDL and exiting without touching infra, runs
the engine and service migration sets and grants the app role under the owner
role (DATABASE_URL_OWNER) before connecting the RLS-subject app pool
(DATABASE_URL, role APP_ROLE) via br-util-postgres, installs the
process-global Prometheus recorder (init_metrics), boots the engine and serves.
BootPlan carries component, app_role, the service Migrator,
EngineConfig, nats_url, the three GraphQL roots, declare_scopes and a
register closure.
graphql::with_edge_observability(app, sdl, metrics)(crate root) mounts
/livez(always 200),/metrics(Prometheus text of the engine's metric set)
and/sdl(the composed schema astext/plain) beside a service'sapp
router, and wraps the whole router in the HTTP metrics layer. This closes
engine defect A3: a deployed pod previously mounted only/graphql,
/graphql/wsand/readyzand installed no tracing subscriber, so it emitted
no logs and exposed no/livezor/metrics.- The engine gains
br-util-observabilityandbr-util-postgresdependencies
(both at the pinnedbr-rust-commonv1.3.0). example_service::db::migrator()is now public: the reference service hands
its migration set to the boot kit rather than running it itself.
H2. Fallible Projector::project
view::Projector::project and the low-level projector::Projector::project now
return Result<_, EngineError> (Result<Out, _> and Result<Option<View>, _>
respectively). A projector that hits a stored document it cannot render — a
nested blob that will not deserialize, a value the view type cannot represent —
returns Err instead of panicking. The render pass dead-letters that poison and
takes the existing repair-then-end-session path for the faulted sessions, so one
malformed row no longer takes the pod down.
EngineError::Projection { projector, key, source }reports the projector and
key that faulted; the erased projector wraps the implementor's error into it.DeadLetterSource::Render("render") andDeadLetters::record_renderland a
render-frame projection failure inservice_engine.dead_letterwith the
projector as the row's reaction and the failing key as its subject, deduped by
a name-based (UUIDv5) message id so a repeatedly failing key folds into one row.SessionRuntime::set_dead_lettersinstalls the store the render pass records
poison into; the engine wires the transport-backed store at boot.
H3. Capability fragments over one aggregate
SliceFragment::derive::<Query, Mutation, Subscription>(slice) reads a
capability's root fields and owned object types from its #[Object] /
#[SimpleObject] impls (through async-graphql's type registry), replacing the
hand-maintained root_fields / types tables a single-aggregate service used to
carry. A slice may register several capability fragments over one aggregate —
one #[Object] per capability file, all naming the same aggregate; capabilities
of one aggregate share its owned types, while two different aggregates claiming
one type name is a collision. The example card slice demonstrates this, split
into graphql/item.rs and graphql/board.rs.
SliceFragmentis now a derived, owned value: its fields are private, it is no
longerCopy, andSliceFragment::new(slice, &[..], &[..])is gone.
SliceFragment::from_claims(slice, root_fields, owned_types)is the explicit
primitivederiveis built on, for a synthetic fragment whose claims cannot be
read from a schema.- The composed-schema boot gate now gates object types, not only root fields:
EngineError::UndeclaredSchemaTypefails boot when the composed SDL exposes an
object type that no fragment owns and the engine does not inject. The
engine-injected object types (mutation ack, lane payloads) are derived from the
engine's own wrappers and allowed unclaimed. SchemaSlices::addtakes&SliceFragment(the fragment is no longerCopy).
Services on the standard boot path (Engine::run/run_with) need no change;
the engine calls verification internally.
H4. Typed consumption — the law and the kit
- **Raw `serde_jso...
br-service-engine 0.1.0
0.1.0 - unreleased
Prepared 2026-09-11 (UTC); unreleased — the v0.1.0 tag is cut when this lands
on main. This is the first functional engine release: service-engine ships
the reactive personalized delivery and process skeleton, and
conformance-service-engine its conformance battery in two modes — in-crate
against the real engine through a sample service, and black-box against the real
example-service binary. example-service (with example-contract and
example-twin) is the in-repo reference service the battery and the functional
spec run against. The sections below describe the shipped 0.1.0 surface; where a
choice is not obvious the rationale is stated.
Added
Engine facade and multi-pod delivery core. Engine<P> composes the render
runtime, transport, accumulators, housekeeping beat and mirror supervision:
boot(config, pg, nats, readiness) (the caller owns the ReadinessHandle, so a
boot that fails the posture or listener probe leaves it DOWN with the reason),
the fallible register_* seams (each returns Result and rejects a duplicate
name with a typed error, since a silent duplicate would hide a degraded
component's health), readiness, attach, push_chunk, seal, run /
run_with / run_with_listener. run supervises its render, beat, flush and
mirror workers: if one ends or panics before shutdown it flips readiness DOWN
(the dead worker in EngineError::WorkerStopped) and returns Err, never
serving readiness over a dead loop; a mirror step that panics self-heals through
the same restart-and-backoff path as an error. attach after shutdown began
returns AttachError::ShuttingDown.
RenderRegistry/SessionRuntime: the connect barrier, the snapshot, the
Reset/Upsert/Removewire over a contiguous per-sessionRevision,
the coalescing render pass,PassReportand GC.Population::{Keys, Ordered, Query}withInterestrouting; aQuerybuilt withwith_keysis
authoritative (its membership becomes exactlypopulate's result plus the keys
discovered this pass, so a key that leaves the result isRemoved and
per-session membership stays bounded), aQuerywithoutwith_keysis
discovery-only. Per-session fault isolation: a failed render or repair is
retried and the session ended afterEngineConfig::repair_attemptsfailures,
never served aResetrebuilt from its stale last-sent view; a session left
repair_pendingafter a failed reconnect resnapshot is retried by the beat.
Cohort keys are collision-free (an RLS render group keyed on the exact
PrincipalId, a declared cohort on the exact bytes of its parts, never a
64-bit hash), so two principals can never share one RLS render.- Impact transport over PostgreSQL
LISTEN/NOTIFY(PgListenNotify):
stage_inin the caller's transaction,schedule_in/fire_duefor
scheduled boundaries, a framed-group payload split that admits a frame only
strictly below Postgres's 8000-byteNOTIFYlimit and reassembles the whole,
a self-repairinglisten()stream surfacing every loss of continuity as
Reconnected(the loss detector that resets a pod's sessions), and
queue_usage(). The listening connection is drained by its own task into a
bounded in-process channel (listener_channel_capacity) that the render loop
consumes, so a slow render never stalls the drain; a channel overflow drops the
impacts and signals oneReconnected, re-snapshotting every session. Past
listener_queue_thresholdthe beat closes the listener throughset_brake,
which takes the pod DOWN, reconnects and resets its sessions.
service_engine_impacts_committed_totalcounts impacts of committed
transactions only, recorded after commit. - Boot posture assertion (
assert_posture: no superuser, norolbypassrls, no
ownership or membership of the engine schema/database) and the boot listener
probe that holds readiness DOWN behind a transaction-mode pooler. - Schema-version singleton (
service_engine.schema_version, one row): boot claims
it after the posture check under a per-service advisory lock, writing the engine
version and the configured service version. A pod that finds a different
version whose row is still live (heartbeat withinschema_version_liveness,
default 30s, refreshed by the beat) refuses to go UP —Engine::bootreturns
EngineError::SchemaVersionConflictand readiness stays DOWN with both versions
in the reason — so a rolling deploy configured by mistake fails loud instead of
two versions sharing one store; a stale row (heartbeat lapsed) never blocks. The
service version isEngineConfig::with_service_version. A running pod whose beat
heartbeat updates no row has been displaced by another version that claimed the
singleton, and lowers readiness to DOWN until it owns the row again, rather than
serving over a store it no longer owns. Timestamp, a newtype truncated to microseconds at construction, so every
instant crossing the PostgreSQLtimestamptzboundary round-trips equal.
Inbound NATS loop, poison and dead-letter. One shared durable consumer per
registered reaction, bound on the gitops-declared INTEGRATION_CMD /
INTEGRATION_EVT streams (bind-only, fail-loud, never created); the engine
creates or updates its own durable with a frozen work-loop contract (explicit
ack, deliver-all, instant replay, an exact filter subject from the reaction's
coordinates, max_deliver unlimited with the budget enforced in code,
ack_wait / max_ack_pending from InboundConfig). Every pod binds the same
durable name, so a message is owned by one pod at a time.
- Ack-after-durable: a message is acked only once its effect committed; a
retryable failureNaks with a growing capped backoff and frees the slot; a
redelivery after a crash-before-commit finds the idempotency claim and acks as
a no-op, so the effect lands exactly once. - A poison message is never lost: routing to the dead letter terminates the frame
only once the dead-letter row is durably written. If that write fails (Postgres
unreachable during a CNPG switchover), the frame is nak'ed with the capped
backoff, not terminated, so the broker keeps redelivering it (max_deliveris
unlimited) until the table can record it — then it is terminated once. The same
holds for a frame that cannot be identified at all. - A reaction handler that panics is caught at dispatch, rolled back and routed to
the dead-letter table as a terminal frame, so the consumer keeps serving instead
of redelivering the panicking frame everyack_waitforever with readiness UP.
cx.principalreturns a typedPrincipalUnresolved(terminal disposition) when
no sender principal has been resolved, rather than panicking — a handler
propagates it like any other reaction error, andcx.try_principalis the
fallible accessor a reaction that needs no sender uses instead. Disposition(Retry / Park / Terminal) routed against per-reaction budgets;
sqlx_is_terminalclassifies a Postgres integrity (SQLSTATE class 23) or data
(class 22) violation as terminal on the engine's own authority, ahead of the
handler's disposition, so a coarseStore(_) => Retrycannot nak a constraint
violation forever.- The
service_engine.dead_lettertable andDeadLettersstore (record,
record_work,listbyDeadLetterSource,discard,retry— retry
re-publishes with a fresh dedup token but the original logical id, so the claim
and sequence guard keep a replay on newer state a no-op). It is the one table
for every source of work: inbound reactions, scheduled messages, cron ticks,
a mirror stuck past its restart threshold and an outbox row that keeps failing
to publish against a reachable broker all record there
(DeadLetterSource::{Reaction, Scheduled, Cron, Mirror, Outbox}), each staging an
impact on the ops noun in the same transaction and incrementing the
service_engine_dead_letters_totalcounter labelled by source. - The per-(producer, reaction, key) sequence guard
(service_engine.sequence_guard) and the idempotency claim
(service_engine.message_claim), applied inside the effect transaction; the
claim is keyed(message_id, reaction), and the guard is scoped per reaction,
so two reactions consuming different facts of one producer under one key keep
independent watermarks and never drop each other's messages. The confirmation
the reaction emitted is stored on the claim row (message_claim.confirmations)
in the same transaction as the effect; a later command carrying an
already-claimed id re-emits that stored confirmation through the outbox instead
of acking a silent no-op, so a producer that lost its confirmation past the
broker's duplicate window is answered again without re-running the effect — the
aggregate does nothing, the engine replays. A command that reuses an
aggregate id under a fresh message id is a genuine duplicate the reaction
decides on (re-emit its confirmation, or a typed rejection), never a
dead-letter for a legitimate replay. - Inbound-consumer and lane-A ingress supervision: each runs under a supervisor
that restarts it with bounded backoff (one log per step, never a hot spin) and,
past a consecutive-failure threshold, lowers readiness (REASON_INBOUND_STOPPED,
aservice_engine_dependency_up{dependency="inbound"}gauge) so a dead consumer
never leaves the pod deaf while it reports ready; on shutdown a consumer naks its
pulled-but-unprocessed frames so they redeliver to a live pod. - Scheduled messages carry a per-row
attemptscount and publish isolated from
the rest of their batch, so one message that cannot publish against a reachable
broker is dead-lettered past its delivery budget while the rest of the batch and
the next beat still fire, rather than one poison row rolling back and re-blocking
the queue forever. It claims one row per transaction and ...
br-service-engine 0.0.0
0.0.0 - 2026-09-02
Added
- Repository scaffold: workspace root, governance files (LICENSE, CONTRIBUTING,
SECURITY, SUPPORT, PR template, issue-template config),.gitignore, and
deny.toml. - Two empty crates —
service-engine(the engine) and
conformance-service-engine(its black-box conformance battery) — carrying no
dependency and no code. Both ship with 0.1.0. - CI (
ci.yml): fmt + clippy + test, MSRV 1.88 build,cargo doc,cargo-deny,
cargo-machete,cargo semver-checks, changelog + README-pin check,
shellcheck, trufflehog secret scan, and the conformance battery against real
PostgreSQL 16 + NATS JetStream. - CD (
release-tags.yml): auto-tag and release the unified workspace version on
merge tomain.
No engine functionality.