feat(pipeline): corpus snapshots, page fetching, and gap-fed generation - #44
Merged
snowopsdev merged 23 commits intoAug 26, 2026
Merged
Conversation
Adds the crawl half of the information-gain corpus snapshot: - `SerpResearch.pages` (`SerpPage[]`) alongside the existing prompt-text `rankingPagesSummary`, so the snapshot builder has URLs as data. The mock client now renders summary and pages from one array, and its hosts match the hosts `corpus/mockPages.ts` has canned text for. - `corpus/fetchPage.ts`: a never-throwing fetcher with a 15s timeout, a 200KB body ceiling, a 24k-char text ceiling, and `extractReadableText` (linkedom + @mozilla/readability, falling back to `<body>` text). - `corpus/concurrency.ts`: `mapWithConcurrency`, order-preserving with a bounded number of in-flight calls. - `corpus/mockPages.ts`: canned per-host bodies about the same topic as the mock generate fixture, sharing five facet sentences verbatim. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds the corpus-snapshots collection the research stage will write (Task 10), plus four new research.* subfields (snapshot, queryCluster, facets, gaps) and revisionNotes/revisionCount on Articles. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… and facet clustering Adds the corpus-side prompts and the snapshot builder behind the information gain scorer: crawl the top-ranking pages for a keyword, extract atomic claims from each, pool in claims from our own published articles, and cluster the lot into the consensus facets and gaps a draft will be scored against. A snapshot is keyed by (keyword, country) and reused for 14 days, so the crawl and its one LLM call per baseline document are paid once per keyword rather than once per article. Internal-article claims are cached a second way: an earlier snapshot listing the same article at the same updatedAt is read back instead of re-extracted, with the array-subfield relationship filter wrapped in a try/catch so an adapter that refuses it degrades to a cache miss. Mock mode gets real fixture content for both calls, and mockPages.ts gains the one dial-in sentence the fixture excerpts were missing, so every excerpt is quoted verbatim by every canned host. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nd gaps into generation The research stage now captures a corpus snapshot alongside the SERP brief: the ranking pages are crawled, their claims extracted, and the pooled claims clustered into the consensus facets the baseline agrees on and the gaps it leaves. Snapshot id, query cluster, facets, and gaps are copied onto the article so a published draft stays explainable against the baseline it was actually written against, even after a fresher snapshot supersedes it. generatePrompt gains `gapsBlock`, which turns those facets and gaps into prompt sections ahead of `# Output`, together with the evidence rules that keep "cover this gap" from reading as licence to invent data — Datum has no first-party measurements — and any revision notes from a failed review. Articles with no snapshot yet generate exactly as before. Also logs which internal-corpus claims came from the snapshot cache, and warns when the adapter refuses the cache filter, since a silent refusal turns every internal document into a paid extraction call. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds docs/information-gain.md (PR2 half) covering corpus snapshots, baseline claims/facets, gap-fed generation, cost, and mock mode. Updates CLAUDE.md's research bullet, collections list, Articles fields, and cost tracking to match, and adds AHREFS_COUNTRY to the root .env.example. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The fetch-limits paragraph claimed FETCH_TIMEOUT_MS bounds only the request, not the body read. Reviewer reproduced against a dribbling local server: the shared AbortController signal aborts an in-flight reader.read() too, so the 15s deadline covers the request and the body read together; the 200KB cap is a separate, independent bound (mainly relevant to a fast server sending an oversized response). Also: "reused by every article that shares the keyword" now names both halves of the (keyword, country) reuse key, and the "Diagrams are updated in PR3" line gets a rule so it doesn't read as part of Mock mode. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The research.snapshot relationship had no maxDepth, so Payload populated the whole corpus-snapshots document — every crawled page's text, up to 10 x 24k chars, plus every baseline claim — onto each article returned by runPipeline's depth: 1, pagination: false query. No stage reads it: buildPrompt uses research.rankingPagesSummary/facets/gaps. A batch of 40 articles pulled megabytes of page text out of Postgres for nothing. maxDepth: 0 keeps it an id at any query depth; PR3 loads the snapshot explicitly when it needs the baseline. Verified against datum_ig: a depth-1 find now returns research.snapshot as a number while template stays populated, which the research stage depends on. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A snapshot is keyed by (keywordKey, country), deliberately not by template, so two articles on the same keyword share one crawl. But a facet's mustHave flag was derived once, at build time, from whichever article's template happened to trigger the build, and research.ts copied it onto the consuming article verbatim. A second article with a different template inherited the first one's required-section flags: gapsBlock told the writer a section was "required by template" when it wasn't, omitted the flag for sections its own template did require, and PR3's facetWeights would have floored the wrong facets' weights. applyTemplateHints (coverage.ts, so PR3 can reuse it) re-matches each facet against the consuming article's requiredSections headings using the same trim + lower-case comparison parseFacetClustering uses, against the facet's matchesHint or its label. Facet now stores matchesHint verbatim so the re-match has something to work with; it is optional, so snapshots captured before this fall back to the label. weight is left alone: it is a property of the corpus, not of the template. Verified against datum_ig: article 4 researched under How-To produced three mustHave facets, then re-researched under Listicle against the same reused snapshot produced none, while the snapshot row kept its build-time flags as the audit record. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
runPipeline had no try/catch around stage.run, so one throw propagated out of the stage and out of the run itself — killing every article queued behind it, including articles at completely different stages. An article waiting at drafted for QA never got processed because an unrelated article's page-3 claim extraction returned prose instead of JSON. Corpus snapshots raise the number of throw sites in a research run by roughly an order of magnitude (up to 16 LLM calls, a strict parseFacetClustering, a payload.create), and pagination: false means one bad article poisons an arbitrarily large batch. The per-article body is now wrapped: the error is logged in full, counted in a per-stage summary line, and the loop continues. The article keeps its status, so the next run retries it — the same convergent-rerun property the whole loop is built on. Verified against datum_ig with an injected throw: the failing article was logged and kept topic_selected, and the qa stage still advanced an unrelated article behind it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
docs/information-gain.md promised that parsePageClaims "drops claims whose excerpt can't be matched back into the fetched text". It never did: parsePageClaims is not given the page text and can only drop entries with a missing or blank excerpt, so a model that invents excerpts produced a corpus that looked fine. Dropping them is the wrong fix. A smaller baseline makes every draft scored against it look more novel, so over-dropping causes false passes, which is worse than a soft claim. Instead the claims are all kept and the unverifiable ones are counted: after parsing a document's claims, every excerpt is checked with excerptFoundIn against the text it came from, a line is logged per document when any fail, and the SERP total is stored as pages[].unverifiedExcerptCount. The internal-corpus path logs the same count; a claim reused from the extraction cache is not re-checked, because the article text is not fetched again. The doc now says all of this plainly and hands the weight-or-drop decision to PR3. Also folded in, same files: - M2: the CorpusSnapshots header comment said the collection "allows update" when access.update is () => false; it meant "no update-throwing hook". - M3: failedPageCount's description now says skipped pages count too. - M4: the internal-corpus find sorts -updatedAt, so past 200 published articles the window is the most recent content rather than an arbitrary slice; the ceiling is documented. - M7: the reuse lookup reads the 3 newest rows and takes the first reusable one (pickReusable), so a fresh empty row from a total crawl failure no longer shadows a good snapshot from days earlier. - M9: one paragraph on crawl politeness — no per-host throttle, three concurrent fetches, no 429 backoff. - Documents the maxDepth: 0 on research.snapshot and the per-article mustHave re-derivation from the two preceding commits. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…es honestly
M6: fetchPage followed redirects with no scheme check, and the fetched
body lands in Postgres and in an LLM prompt. A URL whose protocol is not
http:/https: is now skipped before the request (reason "unsupported
protocol"), and response.url is re-checked after redirects ("redirected
to unsupported protocol"), so a redirect cannot walk the crawler onto
file: or data:. This is a protocol guard only — no DNS resolution, no
private-address rejection — and the docs say so.
M8: USER_AGENT was built at module load from config.targetDomain, so an
unset TARGET_DOMAIN produced "DatumBot/1.0 (+https://)". It falls back to
a bare "DatumBot/1.0" rather than advertising a broken contact URL.
M5: gapsBlock rendered "(covered by N ranking pages)", but docCount
counts every baseline document, our own published articles included. It
now says "baseline sources".
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…hints applyTemplateHints re-derived mustHave per consuming article but kept the stored weight, so a reused snapshot's facet could carry weight = 1 — floored at build time because the *other* article's template flagged it — next to a re-derived mustHave = false. consensusCoverage reads facet.weight directly rather than recomputing it from mustHave, so PR3 would have graded coverage against a floor the consuming template never justified: a silently wrong governance input. Inert today (nothing calls consensusCoverage yet), but latent, and the "PR3 re-floors when it recomputes" comment was aspirational rather than enforced. applyTemplateHints now takes totalDocs and recomputes weight in the same pass, so the two can never disagree. The formula is not forked: the rule moved into a private weightOf(facet, usableTotal), and facetWeights maps over it — same degenerate-case handling (totalDocs <= 0 weights every facet at 1; a missing, negative, or non-finite docCount counts as 0), which the tests now assert against facetWeights' own output rather than against restated constants. research.ts passes the snapshot's baselineDocCount. docCount is still untouched: it is a property of the corpus, not of the template. Tests: 5 new cases in igCoverage.test.ts (losing mustHave drops the floor; gaining it applies the floor; an unchanged facet recomputes to exactly what facetWeights would give it; totalDocs = 0; a negative or NaN docCount), and the existing 8 updated for the new parameter. The MOCK_MODE run cannot show the difference — every mock facet has docCount 3 against baselineDocCount 3, so the weight is 1 with or without the floor. Confirmed the happy path is unchanged: article 4 reused snapshot 4 and advanced research -> generate -> qa_passed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Reuse paragraph said "`weight` is not recomputed: it's a property of the corpus, not of the template", which the preceding commit made false. It now explains why weight is re-derived alongside mustHave — the floor would otherwise outlive the flag that justified it, and consensusCoverage reads facet.weight directly — and that applyTemplateHints and facetWeights share one implementation. docCount is what's left alone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner
Author
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 09d8311a13
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
snapshotStatus depended only on fetch outcomes, so a build whose claimExtraction replies all parsed but yielded zero surviving claims was stored as `complete`. pickReusable then served that empty baseline for the whole 14-day reuse window: generation got no facets and PR3's scoring would grade every draft against nothing, so everything on that keyword would look wholly novel. snapshotStatus now takes the pooled baseline claim count and returns `empty` when it is zero, so isSnapshotReusable/pickReusable skip it. The two kinds of `empty` stay tellable apart on the stored row: a claimless build has baselineDocCount > 0 with failedPageCount counting only genuinely failed pages, where a failed crawl has baselineDocCount 0 and every page in failedPageCount. The builder also logs the claimless case explicitly, and both field descriptions say so in the admin. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The crawler checked only the URL scheme, and `redirect: 'follow'` meant an intermediate hop to a private or link-local address was issued before the post-redirect check could object. SERP URLs come from Ahrefs and the fetched body is stored in Postgres and fed to an LLM, so a ranking page that redirects to cloud metadata at 169.254.169.254 or an internal HTTP service turned a pipeline run into SSRF. fetchPage now follows redirects itself (`redirect: 'manual'`, at most MAX_REDIRECTS = 5 hops) and guards every hop before requesting it: the scheme must be http(s), and the hostname must resolve to addresses that are none of loopback, private, link-local, unique-local, unspecified, CGNAT, or multicast/reserved — including the IPv4-mapped IPv6 forms. Reasons distinguish `private address`, `unsupported protocol`, and `too many redirects` (each prefixed `redirected to ` after a hop); a host that will not resolve stays a `failed` fetch. fetchPage still never throws, and the mock branch touches neither DNS nor the network. The address predicate lives in a new pure module, corpus/addressGuard.ts, so the ranges are unit-tested exhaustively without network or DNS, and fetchPage takes an injectable `lookupImpl` alongside `fetchImpl` so its own tests stay hermetic. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
runPipeline caught per-article throws so one bad article could not strand the batch, but it then resolved normally and index.ts reached its unconditional process.exit(0). Scheduled and scripted runs therefore reported success while articles sat stuck, suppressing failure alerting and retry policies. The whole batch still runs and the per-article log line is unchanged. runPipeline now returns a PipelineRunSummary — per-stage `total`/`failed` plus a grand total — and index.ts prints the per-stage counts and exits 1 when anything failed. runPipeline also takes the stage list as an optional second argument, defaulting to the real pipeline, so the loop can be tested against stub stages and a fake Payload without booting one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
snowopsdev
marked this pull request as ready for review
August 26, 2026 13:08
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5272240f6d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…shots # Conflicts: # cms/src/payload-types.ts # cms/src/payload.config.ts # pipeline/src/stages.ts
…he article A build that crawls nothing, or reads its pages and extracts no claims, was persisted as `empty` so it would never be reused — but it was still returned, so the article that paid for it moved to `researched` with no facets and no baseline and went on to be generated ungoverned. Persist the row (it is the audit record of the attempt) and then throw. The per-article catch in `runPipeline` logs it, keeps the article at `topic_selected`, lets the rest of the batch run, and exits non-zero. `emptySnapshotMessage` is pure and names the keyword, the snapshot id, pages read vs crawled, internal docs, and the claim count, so the two kinds of empty stay distinguishable in the log. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`guardTarget` resolved each hop's hostname and then `fetch` resolved it again when it opened the socket, so a host that answered publicly for the guard and privately a moment later (DNS rebinding) could pass the check and still be connected to a private address. Each hop now gets its own undici `Agent` whose `connect.lookup` is `pinnedLookup(host, addresses)`: it serves only the addresses the guard just cleared, re-applies `isBlockedAddress` to them, and errors on any other hostname or an empty set rather than falling back to real DNS. `connect.lookup` alone decides the socket's destination, so there is no second resolution left to rebind. Pinning replaces resolution only — TLS still uses the real hostname for SNI and certificate checking. `guardTarget` returns the addresses it cleared instead of just a refusal. undici is imported dynamically and only when the real `fetch` is used, so an injected `fetchImpl` never reaches it and the tests stay hermetic. It is now a declared dependency rather than one hoisted in from payload. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The HTTP-error and non-HTML branches returned without consuming or cancelling `response.body`. A `fetch` response whose body is neither read nor cancelled holds its connection open, so a server streaming such a body indefinitely kept a socket alive after `fetchPage` had already reported, and repeated snapshots could accumulate them. Both branches now cancel first, via a `cancelBody` helper that also covers the redirect paths. It never throws, so a body that is absent, already read, or locked cannot turn into the reported failure reason. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`createLlmClient('mock')` called `completeJSONMock` without `request.fixtureKey`,
so a `claimExtraction` request got the whole `{ page, facets }` fixture object
instead of the sub-fixture it asked for and `parsePageClaims` rejected it.
`completeJSONLogged` routes every call through this client, so in mock mode the
research stage failed on the first crawled page of every snapshot build.
Pre-existing, from the jobs/pipeline-runs work merged in from main; it only
became visible here because an empty snapshot is no longer allowed to pass
silently.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Its fake `serpResearch` returned no pages, so the run built an empty corpus snapshot and the assertion that the article reached `qa_passed` only held because an empty snapshot used to be handed back rather than rejected. It now returns the same three mock hosts `MockAhrefsClient` does, and asserts the snapshot it builds is not `empty` and that no article failed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds the versioned reference corpus the information-gain gate scores against, and feeds it back into generation (PR2 of 3). Stacked on #42 — review that first; this PR's base is
feature/informational-gain, so its diff shows only PR2's 13 commits.What's here
pipeline/src/corpus/fetchPage.ts) — the repo's first crawler: nativefetch+ Readability/linkedom, one 15sAbortControllerdeadline covering the request and the body read, a 200KB byte cap and a 24k-char text cap,http(s)only before and after redirects. It never throws: a page that fails or is skipped is recorded on the snapshot with its reason, and a partial baseline is still a baseline.corpus-snapshotscollection +pipeline/src/corpus/snapshot.ts) — for each keyword: the top-10 SERP pages plus up to 5 of our own published articles (selected by non-stopword keyword-token overlap), each decomposed into atomic claims by aclaimExtractioncall at concurrency 3, then clustered into consensus facets and information gaps by one more call that takes the template'srequiredSectionsas must-have hints. Snapshots are hashed, keep the readable page text for audit, and are reused for 14 days per (keyword, country); internal-article claims are cached onupdatedAt, so unchanged articles cost nothing on later snapshots.researchnow writesresearch.{snapshot,queryCluster,facets,gaps}, andbuildPromptgrows four blocks: consensus facets to cover, information gaps to aim at, evidence rules (the handoff spec's anti-synthetic-novelty rule: name a public source or label it an inference; never claim first-party measurements), and revision notes for the regeneration loop PR3 adds.docs/information-gain.md(first half), CLAUDE.md and.env.exampleupdates.Why
Scoring a draft for information gain needs something to score against, and the spec is explicit that it must be a versioned, auditable corpus rather than a live lookup. Doing it in
research(which previously made no LLM calls) means the same claims that will judge the draft also steer the prompt that writes it — the single biggest lever against a model inventing novelty to pass a novelty gate.Two deliberate choices worth a reviewer's attention
mustHave/weightare re-derived per article. A reused snapshot's template hints came from whichever article built it, soresearch.tsrecomputes both against the consuming article's own template; the snapshot row keeps the build-time flags as its audit record.Behaviour change beyond the feature:
runPipelinenow catches per-article stage errors and continues instead of aborting the whole run. PR2 multiplies the number of LLM calls (and therefore throw sites) in a run by roughly an order of magnitude, and the pipeline's documented convergent-rerun property only holds if one bad article doesn't strand every article queued behind it.Deploy note: adds the
corpus_snapshotstables and new article columns; the same "no migrations directory" caveat as #42 applies —payload migrate:createbefore any real deploy.Verified: 287 pipeline tests, 92 CMS integration tests, both typechecks and lint clean; a mock-mode
pipeline:runtakes a fresh articletopic_selected → qa_passed, builds a snapshot, and reuses it on the next run (both cache branches exercised).🤖 Generated with Claude Code