Releases · hops-ops/distributed

chore: update readme (by @patrickleet)
feat: durable SQLx lease lock for QueuedRepository (postgres + sqlite) (#50) (by @patrickleet)
- feat!: durable SQLx lease lock for QueuedRepository (postgres + sqlite)
Add PostgresLockManager / SqliteLockManager — durable, cross-process
AsyncLockManager implementations backed by an aggregate_locks lease
table — so a QueuedRepository can serialize per-aggregate access across
processes, not just within one. Drop-in via .queued_async_with(...).

Each per-key lock layers an in-process gate (InMemoryAsyncLock) over the
DB lease: same-process tasks serialize with true wakeups (no DB polling),
and only the local winner contends cross-process. Acquire is a single
atomic conditional upsert using the database clock (no cross-process
skew); release is owner-token scoped so it never frees a holder that
reclaimed an expired lease. It is a mutual-exclusion optimization, not a
fence — the event-store sequence PK remains the authoritative boundary.
v1 has no lease renewal; sweep_expired reclaims cold rows.

BREAKING CHANGE: AsyncLock::{try_lock,unlock} are now async (a durable
lock releases/acquires via I/O), which makes AsyncUnlockableRepository
and AsyncAggregateRepository::{abort,unlock} async as well. Callers must
.await these. The lock surface is now async-only.

Implements [[tasks/persistent-lock-sqlx]]

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
- fix(lock): address PR #50 review — cancellation safety, lazy ops, max_wait
- Cancellation-safe in-process gate: a GateGuard releases the gate from
  Drop, so a lease_lock/lease_try_lock/lease_unlock future dropped
  mid-await (cancellation/timeout) no longer wedges the key. Replaces the
  explicit-error-path-only gate release.
- max_wait now measured from entry and bounds the in-process gate wait too
  (was only applied to DB polling, after the gate was acquired).
- In-memory try_lock/unlock are now lazy async fn (side effect runs on
  poll, not at call time) — consistent with the I/O-backed locks; a dropped,
  never-awaited future is a no-op. unlock_core is pub(crate) for the guard.
- AsyncAggregateRepository::abort forwards to the repo's abort hook instead
  of unlock, so an AsyncUnlockableRepository overriding abort is honored.
- Tests: add cancellation-safety regression (cancelled acquire releases the
  gate) on both backends.
Migration-split suggestion intentionally not taken (owner decision): this crate
re-runs the full idempotent 0001_initial.sql on every migrate() — there is
no applied-migration tracking — so adding CREATE TABLE IF NOT EXISTS to the
baseline is the correct pattern here.

Refs [[tasks/persistent-lock-sqlx]]

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com

Co-authored-by: Claude Opus 4.8 (1M context) noreply@anthropic.com

See full diff: v1.0.0...v1.1.0

Contributors

patrickleet

Assets 2

30 May 04:20

github-actions

v1.0.0

f345b3f

v1.0.0

What's changed in v1.0.0

chore: use high level macros in most tests (#46)
- feat: use high level macros in most tests
- chore: db mapping
refactor: remove old pattern
feat(transport): async transport foundation in microsvc::transport

Establishes the shared async transport layer in three reviewed slices:
- Core contracts: TransportError (retryable/permanent), FailurePolicy/FailureAction,
  RunOptions/ConsumerDeliveryMode/InboxHook, TransportCapabilities, stable-id rules.
- Source runner: AsyncMessageSource/ReceivedMessage + run_source (ack after handler
  success, retryable->nack, permanent->failure policy, ack-and-ignore unhandled,
  graceful stop, no swallowed errors).
- Publisher/outbox bridge: AsyncMessagePublisher, OutboxMessage->Message mapping
  (reserved x-sourced- metadata namespace), OutboxDispatcher (dispatch_ids/
  dispatch_batch sharing claim->publish->complete), claim-by-id across
  HashMap/SQLite/Postgres, From for TransportError.
Not feature-gated; executor-agnostic (no tokio dependency). Verified with
cargo test --all-features (242 lib unit tests + integration/conformance suites),
fmt, clippy, and doc-link checks.

Implements [[tasks/async-transport-core-contracts]],
[[tasks/async-message-source-runner]], and
[[tasks/async-message-publisher-outbox]] under
[[tasks/async-transport-implementation]].

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
test(transport): reusable conformance harness + in-memory reference run

Adds tests/transport_conformance/mod.rs (adapter-neutral fakes: FakeSource,
FakeReceived, FakePublisher, recording service, plus source-runner and outbox
dispatcher contract fns) and the tests/transport_in_memory target that runs the
full contract against the in-memory reference. Concrete adapters reuse the
harness via #[path]. Adds OutboxDispatcher::publisher()/store() accessors.

Implements [[tasks/async-transport-conformance-tests]] under
[[tasks/async-transport-implementation]].

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
feat(transport): Postgres durable receive via OutboxSource

OutboxSource<S: AsyncOutboxStore> turns any outbox store into an
AsyncMessageSource (claim -> Message -> settle by row status:
ack=complete, nack=release-for-retry, dead-letter/park=fail).
OutboxSource is the Postgres starter transport
(outbox-backed mode, FOR UPDATE SKIP LOCKED + lease, no new table).

Adds tests/postgres_transport integration tests (verified against real
Postgres: drain, concurrent SKIP-LOCKED claim safety, retry, dead-letter),
wires postgres_transport into the Postgres CI job, and adds RabbitMQ/Kafka/
NATS services to compose.yaml for local integration testing.

sqlxmq was evaluated (owner suggestion) and not adopted: its push-based
JobRegistry conflicts with our pull-based AsyncMessageSource/run_source
boundary; patterns borrowed per the spec, not the crate.

Implements [[tasks/postgres-transport-adapter-first-pass]] under
[[tasks/async-transport-implementation]].

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
feat(transport): NATS JetStream adapter + integration tests + CI

NatsPublisher (publish ack threshold) and NatsJetStreamSource (durable pull
consumer; ack/nak/term settle) behind the nats feature, over the shared
AsyncMessagePublisher/AsyncMessageSource/run_source boundary. Stable id +
metadata ride as headers (Nats-Msg-Id is also the JetStream dedup key).

Adds tests/nats_transport (verified against nats:2.10 -js: round-trip +
metadata preservation), a nats CI job, and the nats compose service.

Implements [[tasks/nats-transport-adapter]] under
[[tasks/async-transport-implementation]].

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
feat(transport): RabbitMQ (AMQP) adapter + integration tests + CI

RabbitPublisher (publisher-confirm threshold) and RabbitSource (basic_get;
ack/nack-requeue/reject settle) behind the rabbitmq feature, over the shared
transport traits. Stable id via the AMQP message_id property, metadata+kind via
headers.

Adds tests/rabbitmq_transport (verified against rabbitmq:3.13), a rabbitmq CI
job (service container), and updates the module docs for the NATS/RabbitMQ
adapters.

Implements [[tasks/rabbitmq-transport-adapter]] under
[[tasks/async-transport-implementation]].

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
feat(transport): Knative CloudEvents HTTP ingress

cloud_events_router parses binary + structured CloudEvents into the canonical
Message and calls Service::dispatch_message (the same boundary as run_source).
HTTP response is the ack: 200 success, 503 retryable, 422 permanent, 400
malformed. knative_triggers() renders Trigger YAML from subscription_plan().
Retry/DLQ is platform-managed by Knative here (not this crate's FailurePolicy).

Adds tests/knative_cloudevents (6 in-process HTTP integration tests). Behind the
http feature.

Implements [[tasks/knative-cloudevents-ingress]] under
[[tasks/async-transport-implementation]].

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
feat(transport): Kafka adapter + integration tests + CI

KafkaPublisher (acks=all producer ack threshold) and KafkaSource (consumer
group, auto-commit off; ack=commit offset, nack=seek-back, dead-letter/park=
commit-skip) behind the kafka feature (rdkafka/librdkafka via cmake). recv rides
through transient broker-transport errors within a fetch-timeout budget.

Adds tests/kafka_transport (verified against apache/kafka:3.8.0 KRaft), a kafka
CI job (KRaft service container), and the kafka compose service.

Implements [[tasks/kafka-transport-adapter]] under
[[tasks/async-transport-implementation]].

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
ci: extract integration tests to reusable workflows; run on push-to-main

Moves the postgres/nats/rabbitmq/kafka integration jobs into reusable
workflow_call files (.github/workflows/integration-*.yaml) referenced via local
./ paths from both on-pr-quality and on-push-main-version-and-tag. The push-to-
main pipeline now runs all broker integration tests and gates version-and-tag on
them. Validated with actionlint.

Relates to [[tasks/async-transport-implementation]].

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
docs(transport): add async transports guide

docs/async-transports.md documents the transport layer: core contracts, the
two confirmation thresholds, the source runner, the publisher/outbox dispatcher,
all five adapters (in-memory, Postgres, NATS, RabbitMQ, Kafka, Knative), and how
to run the conformance + broker integration tests.

Progresses [[tasks/transport-docs-examples-cutover]] under
[[tasks/async-transport-implementation]].

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
feat(transport): add Bus/BusConsumer facade + InMemoryBus

Introduce the ergonomic servicebus-style surface over the async
transport traits: Bus (produce — send/publish/send_message/
publish_message) and BusConsumer (consume — listen/subscribe,
generic over the service data, deriving message names from the
service's command/event handlers and running through run_source).

Knative will implement only Bus (it consumes via generated Triggers
- the HTTP ingress); pull transports implement both.
InMemoryBus is the dev/test reference implementation: competing-
consumer queues back send/listen (point-to-point, each message popped
once) and retained per-subscriber-cursor logs back publish/subscribe
(fan-out — every subscriber sees every event), the in-memory shape of
the Postgres-as-log fan-out model. 5 unit tests cover both semantics
plus unknown-command-ignored and handler-error-via-failure-policy.

Implements [[tasks/build-transport-bus-facade]]

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
feat(transport): add NatsBus (send/listen + publish/subscribe)

NatsBus implements Bus + BusConsumer over one JetStream stream bound to
{namespace}.>:
- send → subject {ns}.cmd.{name} (command)
- publish → subject {ns}.evt.{name} (event)
- listen → durable pull consumer {group}_cmd filtered to the service's
  command subjects; replicas sharing a group share the durable, so
  JetStream load-balances — point-to-point / competing-consumer.
- subscribe → durable {group}_evt; each distinct group gets its own
  durable on the shared stream, so every group sees every event — fan-out.
Adds NatsJetStreamSource::with_strip_prefix (default off, backwards
compatible) so the dispatched message name is the bare name once the
{ns}.cmd./{ns}.evt. subject prefix is removed.

Two integration tests prove competing-across-a-group (each command
handled exactly once by concurrent replicas) and fan-out-across-groups
(every group sees every event) against a live JetStream server. Also
makes tests/nats_transport unique() process-unique so re-runs against a
persistent server don't collide with leftover stream/consumer state.

Implements [[tasks/build-transport-bus-facade]]

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
feat(transport): add PostgresBus (work queue + log/offset fan-out)

PostgresBus implements Bus + BusConsumer as a complete single-DB bus:
- send/listen (point-to-point): a bus_queue work table claimed
  FOR UPDATE SKIP LOCKED under a lease, so replicas sharing a group
  compete — each command handled once (ack=delete, nack=release,
  dead_letter/park=delete).
- publish/subscribe (fan-out): Postgres as a log — append-only bus_log
  (monotonic seq, retained) + per-consumer bus_offset (consumer →
  last_seq). publish appends; each group reads seq > last_s...

Assets 2

28 May 17:22

github-actions

v0.6.0

2c7c3fa

v0.6.0

What's changed in v0.6.0

feat: persist read models as relational rows (by @patrickleet)

Remove the generic document read-model store path and lower read-model write plans into declared relational tables for in-memory, SQLite, and Postgres repositories.

Update docs, migrations, tests, and the Bomberman example to use relational read-model schemas and handlers.

Verified with cargo fmt --all, cargo test --test bomberman --all-features, cargo test --all-features, and git diff --check.
feat: add async commit builder ergonomics (by @patrickleet)

Add async commit builder entrypoints for read-model, outbox, and aggregate staging so async SQL repositories can use repo.read_models(plan).commit(&mut aggregate).await.

Update SQLite/Postgres tests and docs to exercise the direct async read-model plus aggregate transaction shape.
test: add async distributed read model flow (by @patrickleet)

Implements [[tasks/async-distributed-read-model-tests]]
test: isolate postgres integration schemas (by @patrickleet)

Fixes [[postgres-read-model-schema-bootstrap-order]]
fix: guard bomberman spawn lookup (by @patrickleet)
docs: rebuild async read model plan example (by @patrickleet)
refactor: rename async commit builder starters (by @patrickleet)
ci: pin pr postgres test actions (by @patrickleet)
ci: pin main postgres test actions (by @patrickleet)
refactor: make async commit builder names primary (by @patrickleet)

See full diff: v0.5.0...v0.6.0

Contributors

patrickleet

Assets 2

27 May 21:23

github-actions

v0.5.0

7a21327

v0.5.0

What's changed in v0.5.0

feat: add microsvc handler message specs (by @patrickleet)

Implements the first slice of [[specs/async-microsvc-transports]].

Adds handler metadata, message envelopes, subscription planning, and projection handler envelope dispatch.
refactor: derive handler specs during registration (by @patrickleet)

Removes per-handler SPEC constants and has register_handlers! construct HandlerSpec values from COMMAND, EVENT, and EVENTS constants.
refactor: remove microsvc command registration aliases (by @patrickleet)

Drops the compatibility Service::command, Service::command_guarded, and Service::commands APIs and updates tests/docs to use HandlerSpec registration.
feat: add microsvc fluent handler registration (by @patrickleet)

Adds HandlerBuilder so command/event registration can use .handle(...) or .guarded(...), with envelope selection before registration.
refactor: expose messages on microsvc context (by @patrickleet)

Removes envelope input modes so handlers always get ctx.message() plus ctx.input::() for JSON payload decoding.
fix: key microsvc handlers by message kind (by @patrickleet)
fix: normalize microsvc message metadata (by @patrickleet)
fix: register inventory reserved saga handler as event (by @patrickleet)
fix: register order completed saga handler as event (by @patrickleet)
fix: register order created saga handler as event (by @patrickleet)
fix: register payment succeeded saga handler as event (by @patrickleet)
refactor: require explicit handler registration kind (by @patrickleet)

See full diff: v0.4.0...v0.5.0

Contributors

patrickleet

Assets 2

27 May 18:29

github-actions

v0.4.0

f6350a3

v0.4.0

What's changed in v0.4.0

chore: claude (by @patrickleet)
chore: less contrived test (by @patrickleet)
: (by @patrickleet)
: (by @patrickleet)
: (by @patrickleet)
: (by @patrickleet)
: (by @patrickleet)
: (by @patrickleet)
: (by @patrickleet)
chore: ai (by @patrickleet)
: (by @patrickleet)
: (by @patrickleet)
: (by @patrickleet)
: (by @patrickleet)
fix: address CodeRabbit review comments (by @patrickleet)

Implements [[address-coderabbit-review-comments]]
: (by @patrickleet)
test: compare snapshot hydration with full replay (by @patrickleet)
fix: address snapshot review comments (by @patrickleet)
feat: microsvc repo + readmodel context config (by @patrickleet)
chore: run ci for all prs (by @patrickleet)
docs: refresh read model write plan examples (by @patrickleet)
fix: refresh read model sync baselines (by @patrickleet)

See full diff: v0.3.0...v0.4.0

Contributors

patrickleet

Assets 2

24 May 18:01

github-actions

v0.3.0

80c90b4

v0.3.0

What's changed in v0.3.0

feat: unify read-model ORM write plans (by @patrickleet)

Implements [[specs/read-model-orm-unification]].

Covers [[tasks/read-model-orm-01-inventory]], [[tasks/read-model-orm-02-metadata]], [[tasks/read-model-orm-03-session-write-plan]], [[tasks/read-model-orm-04-commit-builder-bridge]], [[tasks/read-model-orm-05-compat-conformance]], [[tasks/read-model-orm-06-distributed-idempotency]], [[tasks/read-model-orm-07-schema-bootstrap]], and [[tasks/read-model-orm-08-test-migration-docs]].
fix: harden read-model derive metadata parsing (by @patrickleet)
fix: avoid relational row key fingerprint collisions (by @patrickleet)
fix: validate read-model relationship foreign keys (by @patrickleet)
feat: add read-model helper attributes (by @patrickleet)

Adds direct ReadModel helper attributes for collection, table, column, id, field indexes, unique indexes, and struct-level compound indexes. Updates distributed, metadata, session, schema, and docs coverage.

Implements [[tasks/read-model-orm-09-compound-indexes]].
test: organize distributed read model services (by @patrickleet)

Moves the distributed read-model integration test into account_service, projections_service, and query_service modules while preserving existing behavior.
feat: add tracked read model relationship includes (by @patrickleet)
fix: address read model review feedback (by @patrickleet)
fix: guard primary keys in row patches (by @patrickleet)
test: assert failed sparse insert leaves no row (by @patrickleet)
feat: delete removed has_many children on save_changes (by @patrickleet)

Make save_changes reconcile included collections to the struct: an owned
has_many child dropped from the loaded Vec is deleted, lowering to an explicit
DeleteRow with the loaded expected version. belongs_to clear-to-None stays a
no-op on the target. Safe because has_many includes load the complete owned set.

Replaces the prior "removal does not delete by default" behavior, which was
asymmetric (auto-persisted adds/edits but silently dropped removals).

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
test: distributed relational read-model examples with fulfillment saga (by @patrickleet)

Rework tests/distributed_read_model into a Catalog + Order CQRS slice over
normalized relational read models (ProductView, OrderView has_many
OrderLineView belongs_to ProductView, JSONB columns), and add a kanban
Board + Cards example. Add an order-fulfillment saga (inventory, payment,
saga orchestrator) driving confirm/cancel with a compensation path, projected
into an OrderFulfillmentStepView has_many child for a multi-include query.

Conventions: each write service is a microsvc::Service with service.rs +
handlers/ (one file per message) + models/ (aggregate); the projection service
is one dispatcher organized into handler modules; published domain events are
lowercase dot-namespaced. Services publish via the outbox and subscribe via
microsvc::subscribe — no bespoke transport.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com
: (by @patrickleet)
: (by @patrickleet)
fix: store binary read-model rows as bytes (by @patrickleet)
fix: reject duplicate read-model relationship attrs (by @patrickleet)
fix: fail document plans on unsupported mutations (by @patrickleet)
fix: reject duplicate read-model index names (by @patrickleet)
fix: fingerprint document read-model keys (by @patrickleet)
fix: release queued read-model locks once (by @patrickleet)
fix: fail board projection on malformed event versions (by @patrickleet)
fix: reject non-positive product creation prices (by @patrickleet)
fix: validate distributed inventory quantities (by @patrickleet)
fix: guard distributed order line edits (by @patrickleet)
fix: fail order projection on malformed event versions (by @patrickleet)
test: assert idempotency not-found variants (by @patrickleet)