Skip to content

Add the backfill census: changelog backfill inventory - #3764

Open
cotti wants to merge 1 commit into
feat/backfill-artifact-contractsfrom
backfill-inventory
Open

Add the backfill census: changelog backfill inventory#3764
cotti wants to merge 1 commit into
feat/backfill-artifact-contractsfrom
backfill-inventory

Conversation

@cotti

@cotti cotti commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Why

Backfill planning needs a full census before any selective run: config/products.yml cannot identify release-note surfaces on its own because the release-notes feature defaults to enabled, and only products with published history or native artifacts should produce backfill actions — while everything else must stay visible so an unresolved scope can never silently produce empty bundles.

Part of elastic/docs-eng-team#673 (sub-issue of the backfill epic elastic/docs-eng-team#656). Stacked on #3681 — the census writes the v1 inventory contract that PR introduces.

What

A new docs-builder changelog backfill inventory command (read-only apart from the local output file) backed by InventoryCensusService:

  • Enumerates every release-notes product from products.yml and merges a hand-maintained census seed (YAML) mapping products to their release-note sources — repository, ref, paths, target scheme, cutoff, substitutions, attributed repositories, adoption state, and classification.
  • Products the seed does not cover become source-unresolved entries with a warning; products can instead be deliberately deferred under unmapped, each with an auditable reason. source-unresolved is not seedable — it is the census's own conclusion, never an operator claim.
  • Applies the epic's defaults: stack-versioned semver sources get the 9.0.0 cutoff when none is given; unresolved products get a target scheme derived from their versioning system (serverless/project → date, ech → monthly), always paired with a note so a guess never reads as confirmed fact.
  • Checks attributed repositories against the local assembler.yml link allowlist (advisory — planning re-validates against the deployed scrubber allowlist identity, docs-eng-team#671 / Publish and resolve the deployed scrubber allowlist identity #3763).
  • Output goes through BackfillDocuments: enveloped, schema-versioned, validated, content-addressed. Same inputs produce the same hash.

One contract adjustment to the parent PR's InventoryDocument: SourceRepository/GitRef are now null-able only for source-unresolved entries — the census records products whose location is unknown, which the previously required fields could not express. Validation still demands both for every other classification.

Verified end-to-end against the real repo config: 87 release-notes products enumerated; with a sample seed, the elasticsearch source got the stack default cutoff and elastic/cloud correctly reported off-allowlist.

products.yml alone cannot say which products have release-note surfaces
because the release-notes feature defaults to enabled. The census
enumerates every release-notes product, merges the hand-maintained
source seed, and writes the versioned inventory document planning
consumes. Products the seed does not cover stay visible as
source-unresolved entries with a warning, so "we looked and decided no"
is always distinguishable from "we never looked" and an unresolved
scope can never silently produce empty bundles.

Stack-versioned semver sources default to the epic's 9.0.0 cutoff;
unresolved products get a scheme derived from their versioning system,
always paired with a note so a guess never reads as confirmed fact.
Attributed repositories are checked against the local assembler.yml
allowlist (planning re-validates against the deployed identity).

The inventory contract now allows a null source repository/ref for
source-unresolved entries only — the census records products whose
location is unknown, which the previous required fields could not
express.

Part of elastic/docs-eng-team#673

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@Mpdreamz Mpdreamz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes: census/inventory is the wrong shape for a one-off

Same pushback as on #3681. Stacking a seed→enveloped-inventory compiler on the six-family contracts still does not move bundles to S3, and it does not discover what is published on elastic.co/docs/release-notes.

For a temporary changelog migrate-from-web (fetch published notes → existing bundle YAML → create-only S3 upload → print report → delete the command), we do not need an inventory artifact family, census service, or content-addressed output. A product/cutoff list as plain config (or CLI flags) is enough.

Paper trail, if any: one run report on the issue/PR — not a persisted inventory schema. Please retarget with #3681 toward that one-off path instead of growing the backfill document platform.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants