Skip to content

v0.2.0

Choose a tag to compare

@thatsme thatsme released this 16 Sep 13:31
· 11 commits to main since this release

Operational release: a projection can now say how far behind it is, be
tested without a pipeline, be inspected after it dead-letters, be rebuilt,
and survive a rolling deploy. Upgrading from 0.1.x means one migration —
Scriba.Migrations.up(from: 1) — and nothing else breaks.

Changed

  • Documented the assumption :position carries: a global, monotonic,
    numeric ordinal over every event a projection consumes. Dedup, cursor
    monotonicity, the watermark and :start_from all rest on it, and
    Commanded's event_number satisfies it — which is also why the cursor
    carries over from commanded_ecto_projections. A store whose position is
    not a single increasing integer (commit/prepare pairs, a vector clock) is
    a design question rather than an adapter detail, and is left open rather
    than guessed at.

  • bench/ gained a suite of experiments that need a real event store rather
    than a real database: acknowledgement loss under a straggler, standby
    takeover, the watermark end to end, subscription contention, and the
    broadway_dashboard integration. Each one refutes or confirms a claim the
    documentation makes, which is why they are kept rather than deleted once
    the question was answered. The contention experiment is the reason standby
    exists: it disproved a predicted crash loop and found the real gap, which
    was that nothing retried.

  • Documented that the target is the transaction boundary — read-model rows,
    cursor advances and dead letters commit together or not at all — and that
    cursor and dead-letter storage will not be split into a separate
    behaviour. Splitting them would either leave them in the same transaction
    anyway or turn effectively-once delivery into at-least-once with a crash
    window.

    Other sources and targets are not being prepared for speculatively. An
    interface with one implementation behind it encodes that implementation's
    assumptions, which is how the acknowledgement defect fixed in 0.1.3
    survived a green test suite. The README and Scriba.Target now invite an
    issue instead: another adapter gets built with whoever needs it.

  • A producer that cannot get its subscription now stands by instead of
    failing.
    A persistent subscription admits one subscriber, so on a
    multi-node deployment every node but one is refused. Previously the loser
    raised during init, which meant DynamicSupervisor never adopted it and
    start_projection/1 returned an error — no crash loop, but no standby
    either: nothing retried, so a failover had nobody to fail over to.

    The producer now starts, retries in the background, and acquires the
    subscription when the holder releases it. The retry curve has three
    phases, because two different failures share the path: five attempts in
    milliseconds for the reap race after a deliberate producer death, then one
    a second for half a minute — a killed tree holds its names until its
    slowest in-flight handler returns — then about once a minute with jitter,
    so standbys that started together do not retry in lockstep.
    Configuration errors still raise; retrying those forever would hide them.

    Two new telemetry events make a takeover observable:
    [:scriba, :source, :standby] per failed attempt, and
    [:scriba, :source, :subscribed] when acquired. A standby's projection
    reports :running, because its pipeline is up — the telemetry is what
    distinguishes the node doing the work.

    This also covers a cutover from commanded_ecto_projections: Scriba can
    be started while the old projector still holds the subscription name, and
    picks up when it stops.

Added

  • REBUILDING.md and Scriba.reset/2. Rebuilding a read model is a
    procedure, not a function: declare the same name with a new version,
    start it from :origin with its own subscription name, watch :lag_ms
    fall, point reads at the new table, then retire the old version.
    reset/2 is the last step — it clears a version's cursors and watermark
    (and dead letters on request) so it can start over, refuses while the
    projection is running, and does not touch the read model, which Scriba
    does not know the shape of.

    The guide is explicit about what will bite: handler side effects replay
    over the whole history, and dead letters are not replayed — a rebuild
    that ends with the same error kinds fixed nothing. It also records what
    Scriba will not do: shadow targets and an atomic swap, which are the
    application's decision, and progress as a percentage, which the source
    cannot supply because Commanded exposes no head position.

  • Dead-letter inspection: Scriba.dead_letters/2 lists rows with
    filters, paging and ordering; Scriba.dead_letter_stats/2 summarises how
    many, of what kind, over what span. Scriba.DeadLetter.list/3, count/3
    and stats/3 are the underlying API.

    They read the table rather than a running process, and the module form
    takes the repo from the projection's own config, so a halted or stopped
    projection can be inspected — which is when anyone actually looks.

    The by_error_kind distribution is the diagnosis: one kind on one stream
    is a poison event, one kind across every stream is a schema problem that
    dead-lettering is papering over.

    No replay function. :event_data is stored serialized — __struct__
    becomes a string — so a row records what failed rather than a value that
    can be re-dispatched; replay has to read the event from the source by
    position, and needs an ordering policy this does not yet have.

  • README documents running broadway_dashboard against a projection. It
    works as-is: a projection is a Broadway topology, and the dashboard both
    discovers it through Broadway.all_running/0 and accepts Scriba's
    {:via, Registry, ...} pipeline names. Scriba will not ship a dashboard of
    its own.

  • :lag_interval set in use Scriba.Projection was accepted and then
    ignored — the macro validated it but never passed it on, so lag_interval: 0
    did not disable anything and the Coordinator always used the 5s default.
    Found by the release audit, between the option shipping and the release.

  • [:scriba, :projection, :lag] telemetry. The Coordinator emits it on a
    timer — :lag_interval, default 5s, 0 disables — carrying lag_ms and
    the watermark it was derived from, with the projection's status in the
    metadata.

    On a timer rather than on traffic, deliberately: a projection that has
    stopped receiving events emits nothing else, which is exactly when an
    operator wants to know how far behind it is. It stays silent until the
    projection has committed something, because a projection with no watermark
    reporting lag_ms: 0 would read as caught-up when it has not started.

    Throughput gets no event of its own. Broadway's batch telemetry and the
    per-event :stop events already carry the rate, and a second number could
    disagree with them.

  • scriba_watermarks: the contiguous global position a projection has
    reached
    , surfaced as :watermark and :lag_ms on Scriba.info/2. Every
    event at or below the watermark has been committed, skipped or
    dead-lettered, with no gap below it — so it is a position a replica could
    resume from, and the number that says how far a rebuild has got. Per-stream
    cursors answer neither question: a minimum across them ignores streams the
    projection never wrote to, a maximum counts work above an event still in
    flight.

    :lag_ms is measured from the event's own occurred_at, not from the
    event store's head — Commanded's adapter behaviour exposes no head
    position, so event-count lag is not obtainable, and time-lag is what
    operators alert on anyway.

    The source already computed this number to acknowledge safely (0.1.3); this
    persists it. Written outside the commit transaction and throttled to about
    one write a second while events are in flight, flushed immediately once the
    projection catches up — so it can trail what was applied but never runs
    ahead of it, which is the only direction that is recoverable.

  • Migration steps are idempotent (create_if_not_exists). Without that, the
    documented upgrade path broke fresh installs: an app's original migration
    calls up(), which means "latest", so a new database got the version 2
    table from it and then the upgrade migration collided. Found by running
    examples/bank end to end as part of the release gate.

  • Versioned migrations. Scriba.Migrations.up/1 takes :from and :to,
    so a schema change ships as a numbered step: version 1 is
    scriba_positions + scriba_dead_letters, version 2 adds
    scriba_watermarks. A fresh install still calls up(); an existing one
    adds a migration calling up(from: 1). Scriba tracks no migration state of
    its own — the version lives in the user's migration file, where Ecto
    already records what ran.

  • Scriba.Testing — run a projection's handlers in a test without a
    pipeline.
    project/3 applies events through the projection's configured
    target and commits them, so a test asserts on read-model rows; handle/3
    calls one clause with no database, for asserting the shape a handler
    returns. Events run in one transaction, in order, with per-stream cursors
    advancing as they do in production, and the result reports what committed,
    skipped, failed, or came back in a shape the target cannot apply.

    It deliberately stops at the handler and the commit: no retries, no
    dead-letter routing, no dedup, no telemetry. Those decisions live in the
    pipeline, and a second implementation of them here would be the copy that
    drifts.

Changed

  • The release gate (architecture §14) now requires the CHANGELOG entry to
    account for everything in the release, checked against
    git log <previous tag>..HEAD, and records that a published tag is not
    moved afterwards.