Skip to content

Consume content.replicate: permanent replication per RepSpec, emit replication_complete #29

Description

@gregoryfoster

Epic: CannObserv/archiver#137 · Step 5 — Archiver-issued replication
Blocked by: the co-core replicate contracts

Scope

Replicator gains its second command loop: consume content.replicate, copy the blob permanently per the RepSpec document, emit replication_complete (and a failure fact).

This is the "permanently replicate" half of Replicator's charter that the MVP deferred — the MVP proved fetch → temp-store → fingerprint → fact and explicitly left durable replication out.

Structurally it is the fetch loop again, and should reuse its machinery rather than grow a parallel one: one consumer group (competing consumers), dedupe on command_id with the same TTL posture, ack on success, claim_stale for crashed workers, dead_letter after N attempts, and a failure fact carrying terminal so the issuer can distinguish "still retrying" from "give up."

Providers

gcs, gdrive, ia — per the RepSpec provider sub-schemas Archiver already carries. The command supplies the RepSpec document inline (provider config, credentials_alias, path_template, required_fields), so Replicator holds no domain state and needs no database, exactly as with content.fetch. Resolve credentials_alias locally; a secret should never be on the wire.

MVP scoping is fair game — landing one provider end-to-end and stubbing the others beats three half-wired ones, provided the fact contract does not encode the assumption.

Boundary check

This does not change Replicator's charter (docs/contracts/replicator-boundaries.md): still no database, still no domain interpretation, still bytes in and bytes out. info_source_id and source_revision_id on the command are correlation carried for the issuer's benefit — echo them on the facts, do not branch on them. Same discipline as the info_source_id echo in step 2 of this epic.

Contract doc

The issuer contract (docs/contracts/content-fetch-issuer-contract.md) is written specifically for content.fetch. Two commands with near-identical issuer obligations argue for either a second document or a generalized one — the MUST-1 id-per-occasion trap, MUST-2 persist-before-publish, MUST-4 one-command-many-facts, and MUST-6 keep-a-reaper all apply verbatim to replication. Do not leave the second issuer to infer them. Archiver is that issuer and will get MUST-1 wrong without it, in the intermittent TTL-bounded way that document describes.

Interaction with blob TTL

A replicate command can only be served while the blob is alive (MUST-7, 7 days, clock from last fetch reference). Expired blob → terminal failure with a distinguishable reason so the issuer can decide whether to re-fetch. #7 (object-store backend) changes this and is worth sequencing against.

Done when

  • Second command loop running, sharing the fetch loop's dedupe/claim/DLQ machinery
  • At least one provider end-to-end with a real artifact and a returned public_url
  • Expired-blob path returns a distinguishable terminal reason
  • Issuer obligations documented for the replicate command, not left implicit

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions