Skip to content

Error Codes jobs

github-actions[bot] edited this page Nov 16, 2026 · 2 revisions

Error codes: @ultimat3/jobs

Part of Error codes: every code @ultimat3/jobs registers. x errors explain <CODE> --json prints the same row.

Code Means Typical cause Fix
X_ACTION_JOB_UNBRIDGED an action projection was registered as a job defineApi({ jobs: { x: someAction.job() } }). .job() returns a handle, not a job — action and jobs are both tier 3 so neither may import the other. It used to be skipped in silence alongside the helpers and constants a module namespace carries, so the job simply never ran and nothing failed anywhere register the real job(), or bridge it the way agentJob() in @ultimat3/ai does — it composes a job from the handle's idempotencyKey and invoke
X_BACKFILL_APPLIED the ledger already holds a completed pass a rerun asked for without --force x db backfill <name> --write --force --json — a rerun writes a NEW ledger row
X_BACKFILL_ENVIRONMENT the backfill is not declared for this environment environments: does not include the one ULTIMATE_ENV resolved run it where the declaration allows, or add this environment to environments
X_BACKFILL_MIGRATION_PENDING the migration this backfill requires is not applied requires: names an id x_migrations does not record x db migrate --json
X_BACKFILL_PENDING a declared backfill has never completed x g backfill, merged, deployed — and nothing ever enqueued it x db backfill <name> --write --json
X_BACKFILL_RUNNING a pass under this name is already live one live pass per name — the enqueue deduped onto it x jobs show <id> --json — its step trace names the batch it is on. A pass mid-step shows no new steps and is still healthy; x jobs cancel <id> --json only once the trace confirms it is stuck
X_BACKFILL_SOURCE_INVALID a backfill() declares both or neither of source and pages, or input with no scope backfill() declared source and pages together, neither of them, or an input schema with no scope(input) to key one pass per scope backfill({ name, tenant, pages, handle }) with exactly one of source or pages, and scope: (input) => input.accountId beside input
X_BACKFILL_STALLED the sweep ended with rows its own count still matches count() and source() select on different predicates make count() select on exactly what source() selects on
X_BACKFILL_UNKNOWN no declaration carries this backfill name a typo, or a sweep whose module was deleted x db backfill --pending --json lists every declared name
X_CONSUMER_INVALID a consumer() declaration cannot be run as declared consumer() declared without cursor.load and cursor.save, with a timeout, with a batch that is not a whole number, or with a rearm that is not whole minutes dividing an hour consumer({ name, tenant, keys, open, handle, cursor: { load, save } }) with no timeout
X_CONSUMER_SELF_ENQUEUE a consumer enqueued itself from inside its own run handle() or open() called the consumer own .enqueue, which dedupes onto the running row and arms nothing grep -rn ".enqueue(" apps/web --include=*.ts — delete the call inside the consumer: its .supervisor task re-arms every key
X_DRIVER_UNAVAILABLE the queue driver is unreachable the queue's backing service is down, or the URL is wrong x doctor --json; check the driver URL in app.config.ts
X_EXPORT_PART_TOO_LARGE one page encoded to more bytes than a part may hold a page whose encoded size passed maxPartBytes (default 8 MiB). The memory bound made mechanical: accumulating a result set before writing it is the failure exportRows() exists to prevent, and "we hold a page, not a dataset" is only true while a page is bounded — a row that is a megabyte of JSON turns a thousand-row page into a gigabyte in one buffer lower batch on exportRows("<name>") until a page fits (the default is 500 rows), or raise maxPartBytes if this deployment really can hold that much per part — the number is a heap bound, not a file-size preference
X_EXPORT_ROW_INVALID row() answered something the declared columns do not carry a column missing, a column nobody declared, or a value no format can carry. The extra-column half is the one worth the code: a row() returning a key columns omits is silently DROPPED by both encoders, so the rest of the file looks perfectly correct and the gap is found by whoever consumes it, later — and the reverse, row: (r) => ({ ...r }), is how PII leaves through an export nobody reviewed, picking up every column the entity gains from then on. cause names the COLUMN and never the value: a cell is user data by definition, and a cause reaches the log store as a field nothing can redact row: (source) => ({ …, <column>: <a string, a finite number, a boolean or null> }) on exportRows("<name>") — exactly the declared columns; format a date with an explicit timeZone and a Money with its own currency before it gets here
X_IDEMPOTENCY_REQUIRED the job has no idempotencyKey the field was omitted — normally a compile error, checked again at registration add idempotencyKey: (input) => '<stable>:' + input.id
X_IMPORT_HEADER_MISMATCH the csv header does not name the declared columns the first line of the file names columns importRows() does not declare, or omits some it does — the cause lists both sides, missing and undeclared, so a renamed spreadsheet column is visible before any row is read edit the file's header row to the declared columns, or edit columns on importRows("<name>") to match the file, then re-enqueue with a new key
X_IMPORT_RECORD_TOO_LARGE one record runs past the bytes a record may hold a record found no end within maxRecordBytes (default 64 KiB) — almost always an unclosed quote, which makes every later newline part of one cell. Refused rather than buffered: reading on until the quote closes is how one bad byte turns a bounded import into the whole file in memory repair the quote at the byte offset the cause names (x jobs show <id> --json prints it), or raise maxRecordBytes on importRows("<name>") if a record really is that large
X_IMPORT_TOO_MANY_ERRORS more rows were rejected than the import allows the rows the schema or the format refused passed maxRowErrors (default 1,000; 0 stops at the first bad batch). The batch that crossed the ceiling is not handed to handle, and every rejection so far is in the errors-*.ndjson parts under the run's error prefix — line, field and rule, never the rejected value x jobs show <id> --json names the error parts; read the errors-*.ndjson they list, repair those lines in the file, and re-enqueue with a new key
X_JOB_BACKLOG_FULL an enqueue past the job's declared maxPending backlog was refused the queue the job enqueues onto already holds maxPending rows waiting to run (ready or delayed) — workers are behind, or a producer is outrunning them x jobs list --queue <queue> --json — the queue depth; back off and retry the enqueue later (the refusal is a 429), add workers for the queue, or raise maxPending on the job() declaration
X_JOB_CLAIM_QUEUES_EMPTY a driver was asked to claim with queues: [] an empty list names no queue, and the two shipped drivers had two meanings for it — every queue on the memory driver, the default queue on the Postgres one — while ClaimOptions.queues documented neither. Each meaning is silently wrong in the other's deployment: one takes work this worker was never configured for, the other drains nothing and reads as an idle queue. Reachable only by an embedder, since jobWorker passes exactly one queue per pass driver.claim({ queues: ['default'], limit, visibilityTimeoutMs, workerId }) — the queue by name — or read the list from jobs.queues, which is what jobWorker walks one queue at a time
X_JOB_COMPENSATION_FAILED a compensator could not undo its step a job declaring compensates: true failed for good, and while unwinding a compensate callback exhausted its attempts, or met a once step whose outcome is unknown x jobs resolve-step <id> <step> --ran --output '<json>' # or --not-ran, for a once step; a failed compensator: repair it, then x jobs retry <id> — completed compensators replay
X_JOB_CONCURRENCY_UNENFORCEABLE job.concurrency is declared and cannot be enforced a jobs driver with no lease store — the cap then holds per process and the fleet runs concurrency × replicas. job.concurrency was documented, in the manifest, and enforced by nothing; the worker now refuses to start rather than run with the wrong number, which is what axiom 3 means by a convention that is not a build error not existing remove concurrency from the job cause names, or drop the jobs: field from apps/<app>/runtime.ts so the boot installs postgresJobDriver() over DATABASE_URL — the pg driver is the one with a lease store
X_JOB_DECLARATION_INVALID the job declaration is missing or mis-declares a field job() was called without one or more required fields (input, idempotencyKey, tenant, retry, run) — reachable from generated code and JS callers the type cannot guard — or declares a concurrency no worker can honour: a cap of 0, negative, fractional, NaN or Infinity (plain number or limit), a key that is not a function, a whenBusy other than 'wait' or 'fail'. Also raised at ENQUEUE when concurrency.key(input) answers an empty string, a non-string or more than 200 characters — an empty key is one lock shared by every run. Every missing field is named in one refusal at declaration retry: { attempts: 5, backoff: 'exponential' } on job('<name>') — the cause lists every missing field, each with its own edit; for a cap, concurrency: { key, limit: 1 } — a whole limit of 1 or more
X_JOB_DUPLICATE a name or a live key already has a job a second enqueue inside the dedupe window, or a second job/task registered under a name already taken pass onConflict: 'dedupe' or make the key narrower; for a name clash, rename one export
X_JOB_ENQUEUE_BATCH_TOO_LARGE enqueueMany was handed more inputs than one call may stage a caller passed a whole table or an unbounded list to handle.enqueueMany, past MAX_ENQUEUE_BATCH (10,000) inputs split the inputs into slices of MAX_ENQUEUE_BATCH and call handle.enqueueMany once per slice, or enqueue one job per page that fans out: backfill() from @ultimat3/jobs
X_JOB_EVENT_INVALID a waited event failed its schema the event found for step.waitForEvent fails the schema of that wait, on a hit or a replay make the publisher of the event send what the schema of the wait declares, or widen that schema; then x jobs retry <id>
X_JOB_INTROSPECTION_UNSUPPORTED the jobs driver has no operator surface the installed jobs driver ships no introspect (the redis driver, or a hand-rolled one), or its surface lacks resolveStep — nothing can list, trace, retry, pause or resolve its jobs drop the jobs: field from apps/<app>/runtime.ts so the boot installs postgresJobDriver() over DATABASE_URL — the pg and memory drivers implement introspect — then x jobs list --json
X_JOB_KEY_BUSY the concurrency key already holds its limit of runs a job declaring concurrency: { key, limit, whenBusy: 'fail' } was claimed while limit runs under the same key were still in flight. The run settles failed without its body running, is not retried and is not dead-lettered — the declared answer to a second run of one key, never a fault. whenBusy: 'wait' (the default) leaves the run queued instead x jobs list --name <job> --state running --json
X_JOB_LEASE_LOST the queue took this job back mid-run x jobs cancel wrote a terminal state, or this worker's visibility lease lapsed and another worker re-claimed the row. Its own code and not X_ABORTED because the response differs: X_ABORTED is "your deadline passed, make the work smaller", this is "somebody else owns this run now, stop writing to it" x jobs show <id> --json
X_JOB_MAX_ATTEMPTS the job exhausted its retries a step kept failing x jobs show <id> --json for the step trace, then x jobs retry <id>
X_JOB_NOT_CANCELLABLE the job cannot be cancelled x jobs cancel reached a job that already finished, or an id no queue holds — not a failure of the command, but never a silent success either: an operator cancelling a runaway pass has to know whether they stopped it or missed it. Also raised by a hand-rolled driver whose introspect has no cancel; the redis driver, which ships no introspect at all, is X_JOB_INTROSPECTION_UNSUPPORTED x jobs list --state running --json — for a driver that cannot cancel one job, drop the jobs: field from apps/<app>/runtime.ts so the boot installs postgresJobDriver() over DATABASE_URL — the pg and memory drivers implement introspect.cancel — then x jobs cancel <id> --json
X_JOB_NOT_FOUND the queue holds no job with this id requeue() — x jobs retry, or the admin retry action — was asked for an id the queue does not hold: a mistyped id, a job removed since, or an id from another environment. Both drivers answer it the same way, where the memory driver raised X_INVARIANT and the pg driver X_DRIVER_UNAVAILABLE x jobs list --json
X_JOB_NOT_PROMOTABLE the job is not waiting on its run time x jobs promote or promote() was asked for a job that is not waiting on a clock: an unknown id, a job already due, running or finished, or one suspended in a step.sleep — whose wake time is the step record, not the row x jobs show <id> --json
X_JOB_NOT_REMOVABLE the job is running and cannot be removed x jobs delete, remove() or removeMany() was asked for a job a worker holds. Deleting the row under a running body does not stop the body — its next step write and its settle would land on nothing — so the job is cancelled first and removed once it has stopped x jobs cancel <id> --json
X_JOB_NOT_REQUEUEABLE the job is still live and cannot be requeued x jobs retry / requeue was asked for a job that is ready, delayed or running. Requeueing a running job left claimed_by set with state ready, so a second worker claimed it and it ran twice. Only a dead, cancelled or done job may be requeued x jobs cancel <id> first, then x jobs retry <id>
X_JOB_ON_SETTLED_FAILED the onSettled hook failed after the run was settled a job declared onSettled and the hook threw on every one of its own attempts, after the row was already settled — completed, dead-lettered, dropped or refused. Logged as jobs.on-settled.failed and reported; never thrown to a caller and never a reason to settle the run again, which stays as the queue recorded it x jobs show <id> --json
X_JOB_PAGE_INVALID a job list page was asked for outside its bounds JobIntrospection.list() was asked for more than MAX_JOB_PAGE (200) rows in one page, or handed an after / before cursor no page produced — a hand-written one, or one from another driver. Not clamped: a dashboard that asked for 1,000 and silently got 200 would show a wrong total x jobs list --limit 200 --json — pass its data.nextCursor as --after while data.hasMore is true; in code, list({ limit: 200, after: jobCursor(lastRow) }), omit after for the first page and stop at a page shorter than its limit
X_JOB_REFRESH_VIEW_INVALID a refreshView declaration names no materialized view it can refresh refreshView() was given an entity that is not declared view: { sql, materialized: true }, or concurrently: true over a view with no unique index refreshView({ name, view, concurrently: false }) — or declare the view materialized: true with a .unique() column
X_JOB_ROLLUP_INVALID a rollup declaration cannot be run rollup() was declared without a finite lookback, without a bucket (a series unit and an IANA zone), with an empty key, or with a key no unique constraint on the into entity covers rollup({ name, into, table, key: ['orgId', 'postId', 'day'], lookback: '2d', bucket: { unit: 'day', timeZone: 'UTC' }, source, map, tenant }) — a lookback, a bucket, and a key a unique index on into covers
X_JOB_ROW_STATUS_UNKNOWN a queue row carries a status this build does not know a rolling deploy where a newer worker wrote a status this build has no name for. The decoders used to as-cast the column, which is worse than it sounds: steps.ts:263 returns a step's memoized output only when the status is completed, so a laundered value turned "this step already ran" into run it again drain the older workers, or roll forward — x jobs show <id> --json prints the row. The reverse direction cannot fire: a release that only adds a status still knows every older one
X_JOB_SLOT_LOST the fleet concurrency slot this run held was taken by another worker this worker stalled past the slot TTL and a second worker renewed slot 0 for itself, so job.concurrency was about to be exceeded. Its own code and not X_JOB_LEASE_LOST because the queue may still consider this worker the owner of the job — the two are different rows on different clocks (x_job_leases vs x_jobs.visible_at), and only one of them was lost. The run is aborted rather than left to double up x jobs show <id> --json — a run that keeps losing its slot is slower than its slot TTL, so raise the TTL or make the job shorter
X_JOB_TENANT_MISMATCH a job's payload names an org other than the tenant it runs under an enqueue whose tenant(input) names another org than its tenantId or the acting user's; a queued row refused at its run for the same reason; a webhook delivery's orgId, or the actor an agentJob resolved, outside the job's declared tenant <job>.enqueue(input) from an actor of the org the payload names, or inside crossTenant('<why>', () => <job>.enqueue(input)) by an actor holding tenancy:cross; for an agentJob, load the actor in its actor under the job's own tenant. A queued row that names the wrong org is dead: x jobs list --name <job> --state dead --json, re-enqueue it, never retry it
X_JOB_TENANT_REQUIRED the job declares no tenant the field was omitted — normally a compile error, checked again at declaration, and reached through backfill() too, which forwards its tenant to job() verbatim. A job body runs with no request behind it, and @ultimat3/entity's tenant guard derives from the ambient context, so a job that declared nothing ran with no tenant at all: a row naming another org was refused over HTTP as X_TENANCY_ACTOR_MISMATCH and accepted through the queue. A boot-supplied service actor would close the same hole with one identity for every job, which is the cross-tenant read the declaration exists to prevent. 'none' declares no org either way, but it reads differently on each side of the factory: on a job() it means the body touches no tenant-scoped table, because every scoped read then fails closed as X_TENANCY_ACTOR_ORG_REQUIRED; on a backfill() it is how a sweep says it spans every tenant, and the pass — never the author — opens the cross-tenant scope for it add tenant: (input) => input.orgId to the job; use tenant: 'none' for a job that touches no tenant-scoped table, or for a backfill() that sweeps every tenant
X_JOB_TIMEOUT a job exceeded its wall-clock limit one long step raise timeout, or split the work into more step.run() calls
X_LEDGER_ENTRY_INVALID a ledger entry is not a whole-minor Money amount on a named account an entry has a fractional or unsafe minor, a currency that is not an ISO 4217 code, a scale differing within one currency, an empty account, or the posting has fewer than two entries { account: 'cash', amount: { minor: 1299, currency: 'EUR' } } — in the enqueue of the ledger job: integer minor units (fromDecimal('12.99', 'EUR') from @ultimat3/money), never a float
X_LEDGER_POSTING_CONFLICT a ledger posting key is already stored with other entries the key was posted before and this posting differs in its line count or in a line, so writing the new lines beside the stored ones would leave the book unbalanced entries: [ … the posting as first stored … ] — in the enqueue of the ledger job — or a new key (e.g. ':adjust-1') for a correcting posting
X_LEDGER_UNBALANCED a ledger posting does not sum to zero in every currency the debits and credits of one posting differ in at least one currency entries: [{ account: 'cash', amount: { minor: -500, currency: 'EUR' } }, { account: 'revenue', amount: { minor: 500, currency: 'EUR' } }] — in the enqueue of the ledger job: every currency nets to zero
X_LOCK_LOST a lock withLock held expired before its work finished a renewal was refused or did not land within ttl:, so another holder may hold the key now; the signal handed to fn aborted withLock(key, fn, { ttl: '2m' }) — in the calling file: a ttl: longer than one renewal round-trip, and fn passes lock.signal to every write
X_LOCK_NOT_ACQUIRED withLock could not take the lock within its wait bound another holder kept the key past wait:, or wait: is shorter than the work the holder does under it withLock(key, fn, { ttl: '30s', wait: '10s' }) — in the calling file: raise wait:, or retry the caller later
X_LOCK_STORE_MISSING withLock found no lease store to hold the lock in no job driver is installed, or the installed driver has no leases store jobs: postgresJobDriver() — in apps//runtime.ts — or pass { store } to withLock in the calling file
X_OUTBOX_NO_TX enqueue outside a transaction <job>.enqueue called with no ambient tx ctx.tx(async (tx) => …) around the call — or enqueue with { outbox: false } deliberately
X_PERSONAL_DATA_UNERASED a subject erase would leave a personalData() column of a reached table defineSubject gives a table it reaches { anonymize } without naming every personalData() column, so the erase leaves the person's value there tables: { <table>: { anonymize: { <column>: null } } } — in defineSubject(, …): name every personalData() column, or 'delete', or { retain: '' }
X_PERSONAL_DATA_UNREACHED an entity carries personalData() columns no declared subject reaches no defineSubject() walk reaches the entity, so neither an export nor an erase would ever see its marked columns defineSubject(<entity>, { … }) — in the module declaring the subject: a subject of its own, or a reference / columns: entry so an existing subject reaches it
X_STEP_DUPLICATE two step.run calls share a name copy-paste inside run() rename one — step names are the replay key
X_STEP_NOT_RESOLVABLE the step is not an unknown once outcome an operator can resolve x jobs resolve-step named a job that is running, or a step with no started or failed record (it never ran, or it completed) x jobs show <id> --json # read the steps; wait for a running job to settle, then resolve a step whose status is started or failed
X_STEP_OUTCOME_UNKNOWN a once step started and never completed step.run(name, fn, { once: true }) recorded started before fn, and a crash or a throw left no completed record, so whether fn ran is unknown check the vendor, then x jobs resolve-step <id> <step> --ran --output <json> (or --not-ran), then x jobs retry <id>
X_STEP_OUTPUT_TOO_LARGE a step returned more bytes than a step output may hold step.run(name, fn) answered a value whose JSON runs past the job's declared stepOutputMax (undeclared, past 256 KiB it is logged under this code and stored — refused by default from the next major); every attempt loads every output of its run, so a page or a file in a step multiplies by steps and retries step.run(name, async () => { await storage.put(key, bytes); return key; }) — persist the payload and return its key or cursor, as export() and backfill() do — or raise job({ stepOutputMax }) when the output really is that size
X_STEP_OUTPUT_UNSERIALIZABLE a step returned a value JSON cannot hold step.run(name, fn) answered a bigint, a cycle, NaN, an infinity, an invalid Date, or text jsonb refuses (a NUL, a lone surrogate), so no attempt could replay it return plain JSON from the step: String(id) for a bigint, the ids rather than the object graph, null rather than NaN or an invalid Date, text with no NUL and no lone surrogate
X_SUBJECT_CYCLE the subject erase has no deletion order two tables the erase deletes reference each other (or one references itself) and no foreign key in the loop is nullable, so neither side can go first x db gen "nullable <column>" after making one key in the loop .nullable(): the erase clears a nullable key before deleting either side
X_SUBJECT_TABLE_UNDECLARED a table reaches the declared subject and defineSubject gives it no treatment an entity holds a foreign key into the subject (or into a table the subject erase deletes) and tables: in defineSubject names no treatment for it — usually a table added after the subject was declared. The cause prints the reference path tables: { <table>: 'delete' } in defineSubject — or { anonymize: { <column>: <value> } }, or { retain: '<why it is kept>' }
X_SUBJECT_TREATMENT_INVALID a defineSubject treatment cannot be carried out a blank retain reason, an anonymize naming a column the entity does not have (or its key), 'delete' on an append-only or soft-delete entity, a treatment for a table nothing reaches, or a kept row whose parent the erase deletes. The cause names the table and the field tables: { <table>: { retain: '<why it is kept>' } } in defineSubject — the cause names the field to change
X_WEBHOOK_DELIVERY_FAILED the endpoint did not accept the delivery a 5xx, a 408, a 425, a 429 that named no delay, or a connection that never opened. The one retryable code here, and the reason the split exists at all: this is the same request landing later. cause carries the status or never answered, plus origin and path x jobs list --json — no edit here: the job retries on its declared backoff; watch the endpoint and fix the receiver before it reaches disableAfter
X_WEBHOOK_DELIVERY_REJECTED the endpoint refused the delivery in a way a retry cannot change a 4xx that is not a throttle, or a redirect. Its own code rather than a per-instance flag on the row above: classifyThrown reads a per-instance terminal on an unregistered code as UNCLASSIFIED, so the distinction has to be a registered code or it does not exist x jobs show <id> --json — meta.status is the answer; then correct the url or the secret on this endpoint row in your own table. A 401 or 403 is a secret the two sides disagree about, a 404 is a path that moved
X_WEBHOOK_DELIVERY_THROTTLED the endpoint asked for the delivery later, and said when a receiver that answered a retryable status AND a delay — a 403 carrying a Retry-After is still X_WEBHOOK_DELIVERY_REJECTED, because the status is what says a retry can change something. Its own code because the answer is different work: classified retry-after, so statedDelayMs reads meta.retryAfterSeconds and the nack waits exactly that long — clamped by the policy's maxDelay — instead of spending a curve against a number it was already given retry: { maxDelay: '1h' } only for an endpoint that throttles constantly — or fan out fewer events to it; otherwise no edit: the next attempt waits the delay the receiver named
X_WEBHOOK_ENDPOINT_DISABLED the endpoint takes no deliveries the app disabled the row — refused before the ledger and before the socket, which is the whole point of disabling one — or this delivery is the one that reached disableAfter (default 10 consecutive failures, roughly a day of a retrying queue against a dead receiver), where cause prints both numbers. In the second case this code REPLACES the attempt's own: what an operator needs to see is that deliveries have stopped, not the status of the one that tipped it over. Terminal by construction — the next attempt reads the same row { disabled: false } on the endpoint's row in your own table, once the receiver is fixed — then enqueue the deliveries it missed; the framework never re-enables one on its own, because a dead endpoint that heals itself is a retry loop with no end
X_WEBHOOK_ENDPOINT_INVALID the endpoint cannot be delivered to as declared three conditions, one code, and cause names which: an empty secret, a url that is not a url, or a scheme that is not http: or https:. The scheme is refused by NAME rather than by a blocklist — a delivery opens an HTTP conversation, and file: or data: is a row in a table reaching the worker's filesystem. A delivery with no secret is an unsigned POST to wherever the row points { id, url: 'https://…', secret: '<non-empty>' } — the row webhook("<name>").endpoint returns
X_WEBHOOK_ENDPOINT_UNKNOWN no endpoint carries this id the endpoint seam answered nothing for an id a queued delivery names — usually an endpoint the app deleted between the enqueue and the attempt. Terminal: a queued delivery cannot invent a destination endpoint: ({ endpointId, orgId, ctx }) => <the app's row for endpointId> on webhook("<name>"), so the id cause names resolves — or stop enqueueing deliveries for endpoints your app has deleted
X_WEBHOOK_EVENT_ID_INVALID the event id a webhook receiver verified is not one it can key a job by verify answered an eventId that is empty, longer than 256 characters, or carries a character outside printable ASCII (a space, a control character). Refused before the enqueue, nothing queued, and the id is never echoed: the cause names its length only. Usually an id read from an unsigned header, or from the wrong field of the body return { eventId } from webhookReceiver("<name>").verify read out of the SIGNED bytes (verifyBodyHmac(...).text), 1-256 printable ASCII characters
X_WEBHOOK_EVENT_INVALID the event cannot be signed as declared an id or a topic that would make the canonical string ambiguous: outside 1–200 characters, or carrying a : or a control character. Refused BEFORE the mac is taken, because a mac over an ambiguous string is a valid signature for a delivery the sender did not write — over v1:<t>:<id>:<topic>:<body>, id evt:01HZ with topic orders.paid and id evt with topic 01HZ:orders.paid are the same bytes change how your app mints the field cause names — the separator can never appear in either, so the edit is at the mint site and not at the send
X_WEBHOOK_EVENT_UNKNOWN no event carries this id the event seam answered nothing for a queued id — the delivery was enqueued outside the transaction that writes the event row, so the read raced the commit ctx.tx(async (tx) => { /* write the event row, then enqueue its delivery */ }) — one transaction, so webhook("<name>").event can never be asked for a row that is not committed yet
X_WEBHOOK_RECEIVER_INVALID a webhookReceiver() declaration cannot be served the path is not absolute, sits under /_x or /api (the framework serves both), or a field the receiver needs is missing or not a function (verify, handle, a declared handshake), or maxBytes is not a whole number of at least 1. Refused at declaration, while modules load: no request is ever answered with it fix the field the cause names in webhookReceiver("<name>") — path: "/hooks/<vendor>", verify and handle as functions
X_WEBHOOK_RECEIVER_UNAVAILABLE an inbound webhook could not be queued, so it was not acknowledged the verified delivery reached the enqueue and the queue refused or did not answer (the database is down, the pool is exhausted, the driver is unreachable). 503 with Retry-After on purpose: the vendor redelivers, and an ack never precedes the durable row. The inner failure is named in the cause x doctor --json — then check DATABASE_URL on the web role; the vendor redelivers on its own once the queue answers

Notes

Backfills

Seven codes because each sends the reader somewhere different. Full walkthrough: Migrations and backfills.

Outbound webhooks

webhook() is a job factory, so a delivery ends on one of these eight — every one thrown inside a worker, where ROLE=worker opens no HTTP port, so none of them ever answers a request. The INBOUND pair is X_WEBHOOK_SIGNATURE_INVALID and X_WEBHOOK_SIGNATURE_STALE on @ultimat3/http, and both are 401.

The retry classification is what the split is for, and all eight are listed in registerErrorRetry rather than left to the default: classifyThrown reads an unregistered code as UNCLASSIFIED, which spends a delivery's whole retry policy re-proving that a 404 endpoint is still a 404.

Classification Codes
retryable X_WEBHOOK_DELIVERY_FAILED
retry-after X_WEBHOOK_DELIVERY_THROTTLED
terminal the other six

No secret and no signature reaches a cause, a fix or a meta, and a URL is redacted to origin + path — several senders in the wild put a token in the query string, and all three fields reach the log store and the dead-letter row.

Exports

exportRows() is a job factory too, and both its codes refuse the DECLARATION rather than the data — the same row() over the same page answers the same way on every attempt, so both are terminal.

Clone this wiki locally