v1.0.1

FlowForge v1.0.1

Patch release. Worker observability fix.

What changed

• The Worker <id> connected to the broker INFO line (added in v1.0.0, PR #53) was silently dropped in production because core.worker had no INFO config and fell through to the root WARNING level. INFO is now scoped to the core.worker logger only — the scheduler's own INFO call stays filtered, so a successful (re)connect is visible in the logs.
• Guarded by tests/unit/test_logging_config.py, which loads the real settings.LOGGING via logging.config.dictConfig and asserts the line is emitted at INFO while a core.scheduler INFO record is still not emitted.

Compatibility

• No API, schema, or security changes. Logging configuration only.

Links

• CI: https://github.com/Abolfazlrwm/flowforge/actions/runs/40219649041
• PR: #54
• Full diff: v1.0.0...v1.0.1

FlowForge v1.0.0

FlowForge v1.0.0 — release notes

Approved for release. The owner approved tagging and publishing v1.0.0 on
2026-10-25; that approval is the gate in owner decision 2026-10-22, recorded in
DECISIONS.md. The cut procedure is
RELEASE_CHECKLIST.md.

Summary

FlowForge is a distributed workflow orchestration engine for DAG-shaped jobs,
built on Django 5.2 LTS, PostgreSQL, RabbitMQ and Redis. It makes the hard parts
of distributed execution explicit — leases, fencing tokens, a transactional
outbox and at-least-once delivery — instead of hiding them behind a task
decorator.

v1.0.0 is the first stable release. It is the culmination of Phases 0–10
(shipped as v0.1.0 … v0.11.0) plus Phase 11, Production Hardening: the
operational surface that makes the engine deployable and operable — a security
review, safe migrations, Kubernetes, backups, high availability, an end-to-end
suite, and a signed, SBOM'd supply chain.

The 1.0 stability commitment was an owner decision, and it has been made.
Tagging v1.0.0 signals a SemVer stability promise for the REST API and the
public contract; the owner approved that promise on 2026-10-25.

What FlowForge gives you

• Describe, validate, publish, trigger a workflow over a versioned REST API,
  with a Python SDK (flowforge-client) and a flowforge CLI.
• Correct distributed execution. A scheduler promotes dependency-satisfied
  tasks into a transactional outbox; a publisher delivers to RabbitMQ with
  publisher confirms; a worker claims each task run with a fencing token, runs a
  trusted handler, writes the durable outcome, and only then acks.
• Reliability. Bounded retries with exponential backoff and full jitter, a
  terminal DEAD state with manual requeue, lease recovery that bumps the fencing
  token, and a failure model (docs/architecture/failure-model.md) that maps
  every failure mode to a test that proves the recovery.
• Multi-tenancy and security. Per-organization scoping by construction
  (ADR-014), RBAC, hashed/scoped/rotatable API keys, an append-only audit log
  protected by a database trigger, and secret redaction.
• Observability. Correlation ids and traces that follow a run across every
  process, per-process Prometheus metrics, and Grafana dashboards as code.
• Operability. Docker Compose and a Kubernetes Helm chart, a backup/restore
  runbook with a CI restore drill, high-availability guidance, and a signed,
  SBOM'd, digest-pinned container image.

New since v0.11.0 (Phase 11 — Production Hardening)

• Security review — gitleaks over the full history with push protection;
  Trivy image scanning with a CRITICAL gate; a STRIDE threat model; an
  overhauled SECURITY.md with a private disclosure process.
• Migration safety — a CI test that upgrades from the v0.9.0 schema with
  data present, and a per-migration zero-downtime classification
  (docs/deployment/migrations.md).
• Kubernetes — a Helm chart and rendered manifests, validated in CI with
  helm lint and kubeconform -strict, plus an on-demand kind smoke test.
• Backup and restore — a runbook and a CI drill that dumps, wipes, restores,
  checks row counts, and runs a workflow end to end.
• High availability — docs/deployment/ha.md, honest about what is proven
  and what is operator-validated.
• End-to-end suite — tests/e2e/ drives the REST API, SDK and CLI against
  the Compose stack in CI.
• Supply chain — a CycloneDX SBOM attached to each release, keyless cosign
  signing recorded in Rekor, build provenance, and a digest-pinned base image.
• Documentation site — the MkDocs site is published to GitHub Pages from
  main.

Install

git clone https://github.com/Abolfazlrwm/flowforge.git
cd flowforge
docker compose up --build -d
curl -s http://localhost:8000/health/ready/   # {"status": "ready", ...}

Or the Python SDK / CLI:

pip install ./sdk          # flowforge-client, no runtime dependencies
export FLOWFORGE_URL=http://localhost:8000
export FLOWFORGE_TOKEN=<token>
flowforge trigger my-workflow --version 1

Verify this release

The image is signed; verify without a key. CI publishes the image as main and
main-<commit-sha> (there is no per-version registry tag), so resolve the commit
the tag points at:

IMAGE="ghcr.io/abolfazlrwm/flowforge:main-$(git rev-list -n1 v1.0.0)"

cosign verify \
  --certificate-identity-regexp 'https://github.com/Abolfazlrwm/flowforge/.github/workflows/ci.yml' \
  --certificate-oidc-issuer-regexp 'https://token.actions.githubusercontent.com' \
  "$IMAGE"

For a reproducible deploy, pin the digest rather than the tag
(docker inspect --format='{{index .RepoDigests 0}}' "$IMAGE"). The signature
covers the digest, so a digest reference verifies the same way.

The release carries flowforge-sbom.cdx.json, a CycloneDX inventory of every
component in the image.

Honest scope

• At-least-once, never exactly-once. Duplicate suppression is best-effort
  (idempotency keys + fencing tokens + dispatch_id); two concurrent executions
  of the same key can both run.
• No arbitrary user code. Task handlers are a fixed, trusted set; a task
  definition is data, not code.
• Single writable PostgreSQL primary. The app has no read/write splitting;
  HA needs a standby to promote (see docs/deployment/ha.md).
• Full list: docs/development/PROGRESS.md → Known limitations.

Upgrade notes

Every migration is additive (no table rewrite), so a rolling upgrade is safe:
back up, run manage.py migrate as a one-shot job, then roll the new pods.
See docs/deployment/migrations.md. No action is required to upgrade from
v0.11.0 beyond the migration.

v0.11.0 — Phase 10: SDK, CLI and Documentation

v0.11.0 — Phase 10: SDK, CLI and Documentation

Added

flowforge-client — a dependency-free Python client, published as its own
distribution because the server package drags in Django and a client library must
not. It imports nothing from core or api; it talks HTTP.

• No runtime dependencies. urllib.request is enough for JSON over HTTP, so
  installing it cannot change a resolution anywhere else. CI asserts the list
  stays empty.
• Frozen dataclasses, so run.task_runs[0].state gets completion and a typo
  raises AttributeError.
• Typed errors from the envelope. The API's error.code is stable, so the
  client parses it once and you except NotFoundError, ConflictError,
  ThrottledError (with retry_after).
• trigger(..., idempotency_key=...) — a POST is not idempotent, so the
  client refuses to retry one without a key; with one, a retry replays.
• wait(run_id, timeout=...) polls with a growing interval and raises
  TimeoutError naming the last state, because a caller that ignored a
  still-running run would proceed on unfinished work.
• Retries are 429/5xx only, with exponential backoff, jitter and Retry-After.
  The transport refuses any scheme but http/https — urlopen accepts
  file:, and a base URL that could point at a local path is a
  credential-disclosure bug.

The flowforge CLI — one subcommand per client call, so the CLI cannot do
what the SDK cannot. Config from FLOWFORGE_URL/FLOWFORGE_TOKEN first, flags
second. Exit codes: 0 ok, 1 failed, 2 usage. trigger exits 1 when the run did
not succeed, which is what makes it usable as a CI step.

examples/ — four runnable programs: publish-and-run, watch-the-event-log,
requeue-failed (dry-run by default, because requeueing is a state change on live
work), and a CLI walkthrough that asserts an exit code at every step.

Documentation

• docs/sdk/index.md — the guide, leading with the design decisions and stating
  what the SDK deliberately does not do.
• docs/cli.md — configuration precedence, exit codes, one example per
  subcommand with real output.
• docs/api/reference.md — every endpoint with its request and response body,
  the error envelope, both state machines, rate limits and health probes.
• docs/api/overview.md — its endpoint table now lists the whole surface.

Verification

CI: https://github.com/Abolfazlrwm/flowforge/actions/runs/39968849829

• 52 SDK tests, all without a server — a FakeTransport answers from a
  table, which makes retry, pagination, error mapping and polling testable.
• New sdk job: own lock file, ruff, ruff format, mypy, pytest, an empty
  runtime-dependency assertion, and a wheel build that checks the console script
  is declared.
• Server: 681 unit tests on PostgreSQL, 29 integration tests, ruff, mypy
  (73 files), mkdocs build --strict.

Known limitations

• flowforge-client is not on PyPI — publishing is a release decision that
  has not been taken. Install with pip install ./sdk.
• No async client; no client-side definition validation; no cancellation call
  (the API has no endpoint for it); no publish subcommand.

v0.10.0 — Phase 9: Observability & Performance

v0.10.0 — Phase 9: Observability & Performance

One correlation id and one trace now follow a run from the HTTP request through
the outbox, the broker and the worker. Metrics are exposed per process, the
dashboards are code, and the first real benchmark is committed with its raw
output.

Added

Correlation ids. core.observability.context wraps structlog's contextvars.
The API binds the request id, the run stores it, the outbox row carries it, and
the worker rebinds it per message — clearing first, so one tenant's run cannot
leak into the next log line. Migration 0016 adds correlation_id and
traceparent to WorkflowRun and OutboxMessage.

Prometheus metrics, per process.

• GET /api/v1/metrics/ on the web process; --metrics-port on the scheduler,
  publisher and worker (0 or empty disables it; compose sets 9101/9102/9103).
• Process-local: flowforge_task_latency_seconds, flowforge_task_retries_total,
  flowforge_lease_expirations_total.
• Durable-state, computed at scrape time so they cannot drift:
  flowforge_workflow_runs, flowforge_task_runs, flowforge_outbox_pending,
  flowforge_outbox_lag_seconds, flowforge_dead_tasks,
  flowforge_scheduled_triggers_due.

OpenTelemetry tracing. run.trigger → task.dispatch → task.execute as
one trace across three processes. The W3C traceparent is persisted on the run
and the outbox row because the outbox crosses a process boundary with no live
span. The provider is module-local so a test can install an in-memory exporter.

Grafana dashboards as code + a compose profile.
docker compose --profile observability up adds Prometheus, Grafana and an OTel
collector beside the unchanged application stack. Everything under
deploy/observability/ is code; the dashboard is provisioned read-only.

Load test and measured benchmark. loadtests/locustfile.py drives the whole
pipeline and reports trigger and run end-to-end separately;
manage.py loadtest_user creates the fixture account. A loadtest CI job runs a
fixed 10-user/60-second shape and uploads the raw CSVs.

Measured (GitHub ubuntu-latest, 4 vCPU, AMD EPYC 9V74, 15 GiB, one worker,
Django runserver):

	5 s interval (default)	0.25 s interval
Runs completed	31 (124 task runs)	160 (640 task runs)
Failures	0	0
Throughput	0.36 runs/s	2.54 runs/s
POST /runs/ median / p95	62 / 150 ms	77 / 260 ms
Run end-to-end median	20 000 ms	3 500 ms

The 20-second figure is 4 dependency levels × 5 s — the poll cadence, not the
engine. Both configurations and the raw output are in
docs/development/BENCHMARKS.md and docs/development/benchmarks/.

Changed

Profiled bottleneck fixed. _record_task_run_event fetched the task run's
organization_id with a separate SELECT whenever the caller did not pass it,
and no caller ever did — although all five already held the row. On a 14-task
diamond that is 13 wasted round trips per run, every pass.

• 5.00 → 4.00 queries per transition (−20%)
• 1 672 → 1 542 queries over 140 scheduler passes (−7.8%)
• Reproduce: scripts/profile_scheduler.py --micro

Configurable, with the shipped defaults unchanged and asserted by a test:
THROTTLE_AUTH_RATE, THROTTLE_TRIGGER_RATE, THROTTLE_READ_RATE,
SCHEDULER_INTERVAL.

Documentation

• docs/deployment/observability.md — the correlation contract, the two kinds of
  metric and why they differ, the trace chain across the outbox, the profile, and
  a symptom-to-query table.
• docs/development/load-testing.md — the reproducible command, what each number
  means, and what the test does not tell you.
• docs/development/BENCHMARKS.md — the numbers with their hardware,
  configuration and limitations.

Verification

CI: https://github.com/Abolfazlrwm/flowforge/actions/runs/39962719864

• 681 unit tests on PostgreSQL and 29 integration tests green.
• lint (ruff, ruff format, mypy 73 files, bandit, pip-audit), schema, docs
  green.
• docker green, including the observability profile: flowforge targets up: 4,
  scheduler/publisher/worker metrics reachable on 9101/9102/9103, the dashboard
  provisioned under flowforge-overview, and the collector accepting an
  OTLP/HTTP export.
• loadtest green: the benchmark above.

Known limitations

• No alerting rules ship; thresholds are an operator decision.
• OTEL_EXPORTER_OTLP_ENDPOINT is empty by default, so traces are not exported
  unless set (an always-set endpoint would make the plain stack log a failed
  export every batch interval).
• The collector exports to its debug sink; point it at a real backend to keep
  spans.
• Histogram bucket boundaries are fixed in code.
• The benchmark is CI hardware with the database, broker, workers and load
  generator on one VM, and runserver rather than gunicorn — the figures are a
  floor.
• load_definition is still re-queried on every scheduler pass (~19% of
  schedule_once); caching it would remove a deliberate re-validation.
• SCHEDULER_INTERVAL sets a floor on run latency:
  dependency_depth × interval.

v0.9.1 — Phase 8 follow-ups

v0.9.1 — Phase 8 follow-ups

Added

• Membership management API. GET/POST /api/v1/members/ and
  PATCH/DELETE /api/v1/members/<uuid>/.
  • Only owner/admin may call them; an admin cannot modify an owner and cannot grant the owner role; the last owner cannot be removed or demoted (owner rows are locked while the count is taken).
  • Every real change writes an audit entry (member.added, member.role_changed, member.removed) in the same transaction; a no-op role change writes nothing.
  • Another tenant's membership is a 404.
• make test-pg runs the full unit suite against the compose PostgreSQL with REQUIRE_POSTGRES=1; make db-up starts the database alone.

Changed

• docs/development/PROGRESS.md rewritten against the code and CI: current branches/tags, state-machine edge counts, dependency versions, and only the limitations that still apply (worker reconnect, classic queue, sibling cancellation and Celery were resolved and removed).
• docs/development/setup.md documents the PostgreSQL test run and why SQLite is not the authority for locking, ordering or trigger semantics.

Verification

• CI: https://github.com/Abolfazlrwm/flowforge/actions/runs/39924920840 — 661 unit tests on PostgreSQL + 29 integration tests green; lint (bandit + pip-audit), schema, docs and docker green.
• Local: 661 unit tests (SQLite), ruff, mypy (68 files), mkdocs build --strict, bandit 0, pip-audit 0.

v0.9.0 — Security & Multi-Tenancy

Phase 8 — Security & Multi-Tenancy

FlowForge is now multi-tenant. A single deployment serves many organizations, and every tenant row is scoped by construction.

Highlights

• Organizations and memberships with owner > admin > member > viewer roles.
• Per-organization scoping: every tenant model carries an organization FK, and OrganizationScopedManager.for_organization is the read path. Another tenant's id is a 404, not a 403 (ADR-014).
• Per-tenant uniqueness: (organization, name, version) for definitions and (organization, user, key) for idempotency claims.
• Reversible migration: 0011 adds nullable columns, 0012 creates a default organization and backfills; the reverse detaches and removes it.
• API keys: ff_<hex-prefix>_<hex-secret>, stored only as a hash, with prefix lookup, scopes (read < write < admin), expiry, rotation, revocation, and a one-time plaintext response. A key is bound to one organization and never inherits the staff cross-tenant bypass.
• Append-only audit log, enforced by a database trigger that rejects UPDATE and DELETE; entries for publish, trigger, requeue and API-key lifecycle, with redacted metadata.
• Secret redaction by key name and by an explicit secret: true marker, applied to task config in API responses.
• Tenant-isolation suite: cross-tenant 404 matrix, role-permission matrix, and API-key lifecycle (expired, revoked, wrong scope, rotated, orphaned).
• STRIDE threat model and ADR-014.
• bandit + pip-audit as required CI checks.

Security fixes

pip-audit found real vulnerabilities in the locked dependency set and forced:

• djangorestframework 3.17.2 → 3.18.3 (CVE-2026-114136, CVE-2026-114137, CVE-2026-114138)
• certifi 2026.10.18 → 2026.10.19 (PYSEC-2026-4320)

Migrations

core.0011 (schema) → core.0012 (default-org backfill, reversible) → core.0013 (API keys) → core.0014 (audit log) → core.0015 (append-only trigger).

Verification

• CI run: https://github.com/Abolfazlrwm/flowforge/actions/runs/39921080486 — lint (incl. bandit + pip-audit), test (unit on PostgreSQL + integration on PostgreSQL/RabbitMQ), schema, docs, docker (compose e2e) all green.
• 645 unit tests locally (SQLite).
• bandit: 0 issues. pip-audit: no known vulnerabilities. mkdocs build --strict: clean.

Breaking change: authorization is per organization, not per user. A run is now visible to every member of its tenant; a caller from another tenant receives 404.

v0.8.1 — Phase 7 E2E gaps + transactional event log

Closes the two Phase 7 end-to-end gaps and makes the event log transactional.

End-to-end (compose docker job)

• Cron: the running scheduler fires a `ScheduledTrigger` and the triggered run reaches `SUCCEEDED` through the real outbox → publisher → RabbitMQ → worker path. Proven in CI.
• Saga: a task fails through the real worker and its compensator actually executes; the run settles at its final `FAILED` state with the compensator `SUCCEEDED`. Proven in CI. A new `fail` built-in handler produces the deterministic failure.

Event log is transactional (ADR-013)

`finish_task_run` and `cancel_task_run` now write the state `UPDATE` and the `TaskRunEvent` insert in one transaction, matching the scheduler paths and the outbox. A crash rolls back the state change and its event together — the audit trail no longer has a silent gap.

Full changelog: https://github.com/Abolfazlrwm/flowforge/blob/v0.8.1/CHANGELOG.md

v0.8.0 — Phase 7: Advanced Workflows

Phase 7 adds the workflow features that make FlowForge useful beyond a linear pipeline. Everything is proven on real PostgreSQL + RabbitMQ in CI.

Features

Conditional branching & fan-out/fan-in

Tasks may declare a when condition over the run's task results and a join policy — all (default) or any (one surviving path is enough). A false condition SKIPs the task, and the skip propagates to its dependants so an untaken branch cannot stall the run. The SKIPPED state is a new task-run state.

A sandboxed expression evaluator

when conditions are evaluated by a hand-written tokenizer + recursive-descent parser and evaluator (core/domain/expressions.py) — no eval, no attribute access, no calls, no indexing. A broken condition is a loud definition error, never a silent skip. Injection tests included.

Cron scheduling with no double-fire

ScheduledTrigger starts a run on a cron schedule. The scheduler fires each due trigger with SELECT FOR UPDATE SKIP LOCKED, so N scheduler instances cannot double-fire, and advances next_run_at before triggering so a crash is a missing run, not a duplicate. Proven by a 16-thread race on real PostgreSQL.

Cooperative cancellation

cancel_task_run / request_cancellation cancel a run and every unresolved task, including RUNNING siblings, and bump the fencing token so a handler still running has its late result discarded. A cancel-vs-completion race on real PostgreSQL proves exactly one of the two wins.

Execution history / event log

TaskRunEvent is an append-only record of every task-run transition (claim, finish, cancel, requeue, reclaim, promote, retry, skip). GET /api/v1/runs/<id>/events/ returns a run's events oldest first.

Manual re-run of a failed task

POST /api/v1/task-runs/<id>/requeue/ resets a FAILED task to a fresh PENDING attempt and reverts a FAILED run (never a SUCCEEDED/CANCELLED one). Two operators racing the same task re-run it exactly once.

Saga-style compensation

A task may declare compensate_for: <key>. It is held out of the forward DAG and runs only when the run fails, to undo the tasks that had succeeded — strictly one at a time in reverse dependency order, holding the run RUNNING until the rollback settles before it finishes FAILED.

New API

• GET /api/v1/runs/<id>/events/ — a run's task-run event log.
• POST /api/v1/task-runs/<id>/requeue/ — re-run a failed task.

Fixes

• advance_workflow_run now walks topological_order() so skip/promotion propagation completes in one scheduler pass on any backend (it previously depended on database row order, which differs between SQLite and PostgreSQL).
• The cancel-vs-completion test destructured the race helper's result tuple correctly.

Verification

All five CI checks green: lint, test, schema, docs, docker. The unit suite (581 tests) and the PostgreSQL/RabbitMQ integration suite — including the cron no-double-fire and cancel-vs-completion races — run and pass, not skip.
https://github.com/Abolfazlrwm/flowforge/actions/runs/39759366032

Full changelog: https://github.com/Abolfazlrwm/flowforge/blob/v0.8.0/CHANGELOG.md

v0.7.2 — Resilience verification: every process proven to survive restarts

v0.7.1 added broker and database reconnection, but its CI proof was a single
"restart db, run the sample" step with no per-process assertions. This release
closes that: the docker CI job now proves, per process, that the worker,
publisher and scheduler each survive a broker restart and a PostgreSQL
restart with no process restart.

The reconnection matrix

Process	Broker restart	PostgreSQL restart
Worker	docker step (reconnecting in) + pytest test_a_dropped_connection_redelivers_and_completes_exactly_once	docker step (post-restart run reaches SUCCEEDED)
Publisher	docker step (Publisher recovered)	docker step (Publisher recovered)
Scheduler	N/A — the scheduler never connects to the broker; container ID unchanged	docker step (Scheduler recovered)

Each step asserts the recovery log, diffs container IDs before/after (a manual
restart would change them), and requires a freshly queued run to reach
SUCCEEDED on its own.

Still UNVERIFIED (no named test)

• Clock skew across processes.
• Multi-node / quorum failover, disk loss — only single-node crash safety.
• A worker that is killed, not merely disconnected.

Verification

All five CI jobs green on PR #34:
https://github.com/Abolfazlrwm/flowforge/actions/runs/39670184431

The proof is CI-only — it needs real PostgreSQL and RabbitMQ, which only the
docker job restarts.

Full changelog: https://github.com/Abolfazlrwm/flowforge/blob/v0.7.2/CHANGELOG.md

v0.7.1 — Broker and database resilience

Closes the two gaps Phase 6 left open — and the tests that closed them found three more defects, all of the same class.

1. Reconnection without a restart

Worker.run() was a single start_consuming() call. A broker restart killed the process; the publisher failed its next sweep. Recovery needed a human.

run() is now a loop: exponential backoff with jitter (capped at 30s), then reconnect and re-establish the connection, channel, queue and consumer. The publisher retries its sweep on the same curve. Both waits are on a stop event, so SIGTERM during an outage exits promptly.

An ack that fails is not a success. If the channel dies between the durable write and the ack, the ack raises instead of being swallowed — the broker requeues the message, and the redelivery is deduplicated against the row that is already terminal. Swallowing it would have told the broker a message was finished when it was not.

2. Quorum dispatch queue

The queue is declared with x-queue-type: quorum. A classic durable queue can acknowledge a message it has not flushed; a quorum queue fsyncs through a Raft log before confirming.

A queue's type cannot be changed in place. declare_dispatch_queue translates PRECONDITION_FAILED into QueueTypeMismatchError carrying the delete-and-redeclare procedure, and execution-model.md documents the migration and the single-node caveat: crash-safe durability, not HA.

Three defects the tests found

Each one was found by CI, not by reading the code, and each was the same shape — an error class that the retry loop did not recognise:

Defect	Why it escaped
start_consuming() returns normally when the connection closes under it	The loop treated that as a clean stop and exited
DatabaseError is not an AMQPError	A DB restart escaped run() and killed the worker; the run then sat at running forever
A stopped container fails DNS (socket.gaierror, an OSError)	Not an AMQPError either — the worker died while reconnecting

The caught set is now RECONNECTABLE_ERRORS = (AMQPError, OSError, DatabaseError). The scheduler and publisher also discard their dead database connection (core.db.reset_connections) and back off instead of exiting — before that, restart db killed the scheduler and every run stayed pending.

Evidence, from CI run 39663403734

readiness=503 (attempt 1)
{"status": "degraded", "checks": {"postgres": "ok", "redis": "ok", "rabbitmq": "error"}}
queued run bf2cb12f… during the outage
readiness=200 (attempt 6)
worker-1  | Worker worker-5010e99b lost the broker ((320, "CONNECTION_FORCED …")); reconnecting in 0.49s (attempt 1)
scheduler-1  | Scheduler recovered after 1 failed pass(es).
publisher-1  | Publisher recovered after 4 failed sweep(s).
run bf2cb12f… state=succeeded (attempt 13)

The workflow is triggered while the broker is stopped, the container IDs are compared before and after (a manual restart would change them), and the run must complete. The explicit "restart consumers" step is gone.

A second step parks a dispatch with no consumer, waits until every outbox row is confirmed, kill -9s the broker, restarts it, and requires the run to complete.

failure-model.md

• Mode 14 (broker reconnection): known limitation → verified, with five named tests.
• Mode 17 (broker loses a confirmed message): UNVERIFIED → verified for abrupt termination on a single node. It still does not claim HA, disk-failure resilience, or multi-node failover.

Neither moved without a named test.

Verification

446 unit tests, 25 integration on real PostgreSQL + RabbitMQ; ruff, ruff format, mypy strict, mkdocs build --strict clean.