Repository navigation
v0.7.1 — Broker and database resilience
Closes the two gaps Phase 6 left open — and the tests that closed them found three more defects, all of the same class.
1. Reconnection without a restart
Worker.run() was a single start_consuming() call. A broker restart killed the process; the publisher failed its next sweep. Recovery needed a human.
run() is now a loop: exponential backoff with jitter (capped at 30s), then reconnect and re-establish the connection, channel, queue and consumer. The publisher retries its sweep on the same curve. Both waits are on a stop event, so SIGTERM during an outage exits promptly.
An ack that fails is not a success. If the channel dies between the durable write and the ack, the ack raises instead of being swallowed — the broker requeues the message, and the redelivery is deduplicated against the row that is already terminal. Swallowing it would have told the broker a message was finished when it was not.
2. Quorum dispatch queue
The queue is declared with x-queue-type: quorum. A classic durable queue can acknowledge a message it has not flushed; a quorum queue fsyncs through a Raft log before confirming.
A queue's type cannot be changed in place. declare_dispatch_queue translates PRECONDITION_FAILED into QueueTypeMismatchError carrying the delete-and-redeclare procedure, and execution-model.md documents the migration and the single-node caveat: crash-safe durability, not HA.
Three defects the tests found
Each one was found by CI, not by reading the code, and each was the same shape — an error class that the retry loop did not recognise:
| Defect | Why it escaped |
|---|---|
start_consuming() returns normally when the connection closes under it |
The loop treated that as a clean stop and exited |
DatabaseError is not an AMQPError |
A DB restart escaped run() and killed the worker; the run then sat at running forever |
A stopped container fails DNS (socket.gaierror, an OSError) |
Not an AMQPError either — the worker died while reconnecting |
The caught set is now RECONNECTABLE_ERRORS = (AMQPError, OSError, DatabaseError). The scheduler and publisher also discard their dead database connection (core.db.reset_connections) and back off instead of exiting — before that, restart db killed the scheduler and every run stayed pending.
Evidence, from CI run 39663403734
readiness=503 (attempt 1)
{"status": "degraded", "checks": {"postgres": "ok", "redis": "ok", "rabbitmq": "error"}}
queued run bf2cb12f… during the outage
readiness=200 (attempt 6)
worker-1 | Worker worker-5010e99b lost the broker ((320, "CONNECTION_FORCED …")); reconnecting in 0.49s (attempt 1)
scheduler-1 | Scheduler recovered after 1 failed pass(es).
publisher-1 | Publisher recovered after 4 failed sweep(s).
run bf2cb12f… state=succeeded (attempt 13)
The workflow is triggered while the broker is stopped, the container IDs are compared before and after (a manual restart would change them), and the run must complete. The explicit "restart consumers" step is gone.
A second step parks a dispatch with no consumer, waits until every outbox row is confirmed, kill -9s the broker, restarts it, and requires the run to complete.
failure-model.md
- Mode 14 (broker reconnection): known limitation → verified, with five named tests.
- Mode 17 (broker loses a confirmed message): UNVERIFIED → verified for abrupt termination on a single node. It still does not claim HA, disk-failure resilience, or multi-node failover.
Neither moved without a named test.
Verification
446 unit tests, 25 integration on real PostgreSQL + RabbitMQ; ruff, ruff format, mypy strict, mkdocs build --strict clean.