Emit the enriched BlobAvailableEvent fields (cannobserv#271)

[gregoryfoster]

Tracking the producer side of CannObserv/cannobserv#271 (enriched
BlobAvailableEvent). Consumer: CannObserv/watcher#241 (Phase 4).

Why

src/worker/handler.py holds a full FetchResult at publish time and puts six of its
facts on the floor. Watcher needs them after Phase 4 because it no longer performs the
fetch itself:

• final_url — Watcher derives effective_url and domain_name (its rate-limiter
  key, and an FK) from the redirect chain. Post-cutover it sees only the requested URL.
  Also blocks watcher#157.
• content_type_raw — Watcher stores the verbatim Content-Type header
  (watcher#168). The normalized media_type drops charset and defaults absent headers to
  application/octet-stream, which is exactly the value Watcher's dispatch reads as
  "unknown, guess from the URL". Raw passthrough (None when absent) preserves the
  distinction.
• status_code, fetched_at — occurred_at is stamped at publish, which under
  retry or PEL parking is not when the bytes came off the wire.
• etag, last_modified — the conditional-GET seam. Returned here and replayed via
  ContentFetchCommand.headers (CannObserv/cannobserv#272), Replicator gets conditional
  GET while holding zero per-URL history — which is the point: this service has no
  database and should not grow one.

Scope

Populate the new optional fields in handler.py's BlobAvailableEvent construction from
the FetchResult. Notes:

• final_url — only when the client actually reports a landing URL; do not synthesize it
  by echoing command.url, or an issuer cannot distinguish "no redirect" from "unknown".
• content_type_raw — verbatim, None when the header was absent. The existing
  _media_type normalization is unchanged; this adds a channel beside it.
• last_modified — pass the string through unparsed; it gets replayed verbatim in an
  If-Modified-Since and a parse/re-serialize round trip can differ from what the origin
  sent.
• Log-line dedupe: duration_ms is currently called out in handler.py as "the one
  number nothing else keeps". With fetched_at on the fact that comment is stale — worth
  a look, though duration_ms itself is still journal-only.

Follow-on to flag, not fix here

A body-less 304 currently dead-letters. Conditional GET is only half-usable until that
becomes a first-class outcome — reason="not_modified" on the fetch_failed fact
(CannObserv/replicator, companion issue), or a dedicated content_unchanged fact. Decide
before an issuer sends its first validator header.

Contract doc

Update the BlobAvailableEvent field table in
docs/contracts/content-fetch-issuer-contract.md in the same change ("edit one, edit
both"), including the "unknown vs. absent" semantics for final_url and
content_type_raw — that distinction is the whole value of the fields and will be lost if
only the types are documented.

Refs

• co-core model: CannObserv/cannobserv#271
• Consumer: Phase 4: build the content.fetch producer against Replicator's issuer contract watcher#241, watcher#168, watcher#157

[gregoryfoster]

final_url is now unblocked — cannobserv v0.7.5 ships the producer source

The scope note here flagged the tension exactly: "final_url — only when the client actually reports a landing URL; do not synthesize it by echoing command.url." Until now there was no non-synthetic source — FetchResult didn't carry the landing URL, so handler.py had nothing to read short of reaching around the Fetcher seam with its own httpx.AsyncClient to inspect response.history (exactly the coupling the effect type exists to prevent).

CannObserv/cannobserv#279 (shipped in v0.7.5, tag v0.7.5) fixes that upstream:

• co_core.effects.fetch.FetchResult gains final_url: str | None (additive, None-defaulted).
• co_core_aio.fetch.AsyncFetchDriver.execute populates it from str(response.url) — the post-redirect landing URL httpx already held.

So handler.py reads it straight off the result:

final_url=result.fetch.final_url,   # via acquire: AcquisitionResult.fetch.final_url

The semantics match this issue's guidance without any client-side logic:

• None means "the driver did not report a landing URL", never "no redirect" — so an issuer keeps the "no redirect vs unknown" distinction this issue asked for. Do not coalesce None to command.url.
• The http driver sets it always — redirect or not (it's also the httpx-normalised request URL), and on non-2xx captures too (a 404 reached through two redirects carries a real final_url). So on the acquire path final_url is reliably populated; the None branch only matters if a future non-reporting driver appears.
• It's a str (matches how BlobAvailableEvent.final_url is typed), so it flows to the wire model verbatim.

To adopt: bump the cannobserv git-tag source to v0.7.5 (floor stays >=0.7,<0.8) and uv sync. The other five enriched fields (status_code / content_type_raw / etag / last_modified / fetched_at) are unaffected by this and can ship independently — this only closes the one field that had no producer.

Sequencing is safe either way: both FetchResult.final_url and BlobAvailableEvent.final_url are None-defaulted, so populating here before or after the tag bump never breaks a build.

[gregoryfoster]

Review commentary

Reviewed against src/worker/handler.py, the installed co-core, the contract doc, and the
upstream issues. The motivation holds on every field and the scope is right. Two corrections
and several additions before implementing.

The premise was half-true, and the missing half is now fixed

handler.py holds a full FetchResult at publish time and puts six of its facts on the floor.

True for four. final_url and fetched_at were never on FetchResult — it carried
{content, status_code, headers, duration_ms, fetcher_used}, and AsyncFetchDriver.execute
discarded response.url one layer below anything Replicator can see. So the field this issue
calls its strongest motivation (watcher#157, the domain_name FK / rate-limiter key) had no
producer at all.

Filed as CannObserv/cannobserv#279; shipped in co-core v0.7.5. FetchResult now
carries final_url: str | None, populated from str(response.url), always set regardless of
redirect history and on non-2xx captures too. All six fields are populatable in one pass.

fetched_at needs nothing upstream — stamped in handle immediately after _fetch returns,
which is "bytes off the wire" to within the handler's own overhead and needs no duration_ms
arithmetic.

Note for whoever implements: #279's closing comment refers to result.fetch.final_url.
That is the acquire workflow's shape (AcquisitionResult.fetch). Replicator drives the
driver directly, so in handler.py it is result.final_url.

Version floor is load-bearing, not tidiness

BlobAvailableEvent is extra="ignore". On co-core 0.7.2, BlobAvailableEvent(..., final_url=..., status_code=200, etag=...) constructs cleanly and silently discards all three — verified.
A version skew does not fail loudly; it publishes facts that look right and carry nothing.

So the pin moves from >=0.7,<0.8 to >=0.7.5,<0.8 on both co-core and co-core-aio,
turning that into a resolution error.

Per-field notes

Field	Source	Note
final_url	result.final_url	Pass through unchanged, None included. Never fall back to command.url — that is the unknown-vs-absent distinction the field exists for
status_code	result.status_code	Always 2xx on this fact. _raise_for_status closes every non-2xx as fetch_failed, so its value is 200 vs. 203 vs. 206. Stating this in the contract stops an issuer writing a branch that never fires
fetched_at	stamped after _fetch	tz-aware UTC — co-core's OccurredAt rejects naive fail-loud
content_type_raw	folded header lookup	Verbatim incl. charset; None when absent — it must not fall through to DEFAULT_MEDIA_TYPE, since that value is exactly what Watcher's dispatch reads as "unknown, guess from the URL"
etag / last_modified	folded header lookup	Verbatim incl. the ETag's W/ prefix and quotes — replayed unparsed in If-None-Match / If-Modified-Since

Additions to scope

• Bound the origin-controlled passthroughs. etag / last_modified / content_type_raw
  are strings an origin chooses, riding onto a broadcast stream nothing trims. Over a sane
  length, drop to None rather than truncate — a truncated ETag replayed in If-None-Match
  is worse than an absent one. Same reject-don't-truncate posture Honour ContentFetchCommand request options: headers, timeout_seconds (cannobserv#272) #11 has to adopt on the
  request side.
• Fold the header map once. _media_type builds a lower-cased map for a single lookup.
  Four reads now; fold once in handle and pass it down rather than rebuilding it three more
  times with the case-folding rationale copy-pasted.
• Contract doc, beyond the field table:
  • Reinforce MUST-5 where it already lives. These are per-fetch-occasion values on a
    fact whose idempotency key is content_fingerprint, so a consumer deduping its inbox on
    the fingerprint — already forbidden — now also pins etag / last_modified / final_url
    to the first emission for those bytes. co-core documents this as the validator-freshness
    caveat.
  • The co-core bump makes the ContentFetchCommand table stale too: headers and
    timeout_seconds are legal on the wire from v0.7.3 even though Honour ContentFetchCommand request options: headers, timeout_seconds (cannobserv#272) #11 has not taught
    Replicator to honour them. Add the rows marked accepted and currently ignored, or an
    issuer reads the table and concludes the fields do not exist.
  • Bump the stale co-core ≥ 0.7.2 markers.
• duration_ms log line — narrow the "the one number nothing else keeps" comment rather
  than drop the field. With fetched_at and size_bytes on the fact, duration is still the
  only journal-only datum, but the comment's claim no longer reads true.

On the flagged follow-on

The 304 companion issue does not exist yet. #10 and #11 together produce validators and
replay them, and the first issuer that does so gets its command dead-lettered by
_raise_for_status. Recommended shape when it is filed: reason="not_modified",
terminal=True on fetch_failed — not a new content_unchanged fact. The taxonomy row
already exists (http_status covers the body-less 304 today), issuers already branch on
terminal first, and a new event type costs every consumer a dispatch arm for an outcome that
is structurally "no blob is coming". Does not block this issue.

Sequencing

sync_wheelhouse.py → targeted relock of co-core / co-core-aio → floor bump → the six
fields + docs. #10 has standalone value for watcher#168 today; #11 stays inert until the 304
outcome is settled.

[gregoryfoster]

Shipped to main (merge 2edacd1; 4c1a1fa → 47c5656).

Implemented

• All six enriched fields populated in src/worker/handler.py's BlobAvailableEvent
  construction — final_url, status_code, fetched_at, content_type_raw, etag,
  last_modified. fetched_at is stamped where the fetch returns, not at publish:
  occurred_at under a reclaim is minutes after the bytes were on the wire.
• None means nobody said it, never a stand-in. final_url is never backfilled from
  command.url, content_type_raw never with DEFAULT_MEDIA_TYPE. Each has a test whose
  only job is to catch the tempting shortcut.
• New _passthrough helper: verbatim value, None for absent/blank, and dropped over
  MAX_HEADER_VALUE_LENGTH (1024) rather than truncated — these are origin-controlled
  strings on a broadcast stream nothing trims, and a truncated ETag replayed in an
  If-None-Match is a validator that can never match. _media_type reads the same bounded
  value so an absurd Content-Type cannot re-enter through the normalized channel; the
  case-folded header map is now built once per fetch rather than per lookup.
• docs/contracts/content-fetch-issuer-contract.md: the field table, the three semantics
  that are the whole value of the fields (unknown-vs-absent, status_code is always 2xx
  here, drop-not-truncate), a MUST-5 note that these are per-occasion values on a
  fingerprint-keyed fact, the ContentFetchCommand rows for headers / timeout_seconds
  marked accepted and currently ignored, and an explicit "do not attempt conditional GET
  yet" warning.
• scripts/seed_fetch.py --watch prints status= and final_url= on the blob_available
  line — the operator surface for verifying this change against a live fetch.

The blocker, and what it cost
The issue's premise — "handler.py holds a full FetchResult at publish time and puts six of
its facts on the floor" — held for four of them. FetchResult carried no landing URL at all:
AsyncFetchDriver.execute discarded response.url one layer below anything Replicator can
see, so final_url — the field motivating watcher#157 and the domain_name FK — had no
producer anywhere in the cluster. Filed as CannObserv/cannobserv#279, shipped in co-core
v0.7.5, and this branch consumes it.

Version floor
Raised to >=0.7.5,<0.8 on co-core / co-core-aio, and the patch level is load-bearing.
BlobAvailableEvent is extra="ignore": on an older wheel a construction carrying the new
kwargs succeeds and silently discards them, so a version skew would publish facts that look
right and carry nothing, with nothing anywhere to say so. The floor turns that into a
resolution error.

Review
Three rounds, 15 findings, all resolved (one stet). The substantive ones: a case-insensitivity
test whose membership assertion could not detect a field mis-mapping; the new --watch output
shipping untested because scripts/ sits outside the coverage gate; and tests/test_seed_fetch.py's
fact_fields helper claiming to be "the wire frame the worker's handler publishes" while
carrying eight of its fourteen fields.

Verification: 283 unit tests + 12 integration against the live VM Redis, coverage 99.68%
(src/worker/handler.py at 100%), ruff and format clean.

Follow-ups

• A body-less 304 dead-letters — make "not modified" a first-class outcome before conditional GET is usable #17 — a body-less 304 still dead-letters, so conditional GET is only half-usable. The
  contract's "do not attempt conditional GET yet" warning comes out when that lands.
• Honour ContentFetchCommand request options: headers, timeout_seconds (cannobserv#272) #11 — headers / timeout_seconds are legal on the wire from co-core 0.7.3 but not yet
  honoured; it is the replay half of the etag / last_modified seam.