Feature: client-backfill by philcunliffe · Pull Request #66 · hyparam/hypaware

philcunliffe · 2026-05-28T16:29:54Z

Feature: client-backfill

Bead 1: Core backfill registry + CLI skeleton + AI gateway materializer extraction — This is bead 1 of the client-backfill feature. Read the full plan at .ai/plans/hypaware-first-class-client-backfill.md. This bead covers rollout phases 1 and 2.
Bead 2: Claude backfill provider — This is bead 2 of the client-backfill feature. Read the full plan at .ai/plans/hypaware-first-class-client-backfill.md. This bead covers rollout phase 3. Bead 1 added the backfill registry, CLI, telemetry, and AI gateway materializer — those are now available.
Bead 3: Claude onboarding integration — This is bead 3 of the client-backfill feature. Read the full plan at .ai/plans/hypaware-first-class-client-backfill.md. This bead covers rollout phase 4. Bead 2 added the Claude backfill provider — it's now available.
Bead 4: Codex backfill provider — This is bead 4 of the client-backfill feature. Read the full plan at .ai/plans/hypaware-first-class-client-backfill.md. This bead covers rollout phase 5. Beads 1-2 added the registry and AI gateway materializer — those are available.
Bead 5: Codex onboarding integration — This is bead 5 of the client-backfill feature. Read the full plan at .ai/plans/hypaware-first-class-client-backfill.md. This bead covers rollout phase 6. Bead 4 added the Codex backfill provider — it's now available. Bead 3 added the Claude onboarding integration — use the same pattern.
Bead 6: Smoke coverage + idempotency tightening — This is bead 6 of the client-backfill feature. Read the full plan at .ai/plans/hypaware-first-class-client-backfill.md. This bead covers rollout phases 7 and 8. All providers and onboarding integrations are now available.

Auto-generated by /feature-launch. The ship formula will replace this with a diff-aware summary before marking ready-for-review.

…al) (#67) Bead 1 of the client-backfill feature (rollout phases 1-2). Phase 1 — core: - Add BackfillRegistry and BackfillMaterializerRegistry (register/get/list, duplicate + shape validation) in src/core/registry/backfills.js. - Expose ctx.backfills / ctx.backfillMaterializers during plugin activation and on CommandRunContext; wire both through the kernel runtime and dispatcher. - Add `hyp backfill [provider...]`, `hyp backfill list`, `hyp backfill plan`: provider selection (config-filtered by default, explicit names override), retention/date-window resolution, dry-run (scan-only), and JSON output. - Emit the backfill.* telemetry envelope (start/plan/provider_start/scan/ materialize/write/flush/provider_finish/finish) with dev_run_id, provider, dataset, and row counts. Writes route through storage.appendRows so rows land in the dataset's per-source partitions exactly like live capture. Phase 2 — AI gateway materializer: - Extract aiGatewayRowsFromProjectedExchange() as the single row-expansion path shared by live capture and backfill; createAiGatewayMessageProjector delegates to it with a persistent per-listener conversation state, so live behavior is unchanged. - Register the ai_gateway.projected_exchange materializer (dataset ai_gateway_messages) converting projected exchanges into canonical rows, stamping backfill provenance (hashed source path, native id). Tests: registry register/list/duplicate/validation; CLI parsing, provider selection, retention resolution; dry-run writes nothing; materializer parity with the live projector (native + fallback identity). npm test (600 pass) and npm run typecheck both green. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…7) (#68) Register a `claude` backfill provider in @hypaware/claude that imports local Claude Code JSONL transcripts into ai_gateway_messages through the ai-gateway `ai_gateway.projected_exchange` materializer, so backfilled and live-captured rows are identical for the same conversation. - Discover ~/.claude/projects/**/*.jsonl, group by sessionId, and join session-context.jsonl for cwd/git_branch. - Preserve native DAG identity (uuid -> message_id/provider_uuid, parentUuid -> previous_message_id/parent_uuid) plus role/content, tool calls/results, permission mode, sidechain, and compact metadata. - Store only a minimized native frame in raw_frame (never the full transcript); tag conversation_source/client_name = 'claude'. - Deterministic reruns: identity and timestamps come from the immutable transcript and the materializer is pure. Reuse: surface role/content on TranscriptEntry and export walkTranscriptFiles/loadTranscriptFile from transcripts.js. Tests: fixture -> projected exchange -> canonical rows, rerun determinism, session grouping, native DAG identity, minimized raw_frame, and since-window filtering. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

#69) Bead 3 of client-backfill. When the onboarding picker selects a client with a registered backfill provider (claude in V1), import its local history into the query cache right after the config write and before the daemon (re)start that resumes live capture. - Add runBackfillProvider() to commands/backfill.js: runs one named provider end-to-end (scan -> materialize -> write -> flush) via the shared runProvider path, so finale-imported rows land in the same per-source tables as `hyp backfill <provider>` and live capture. - Thread a PickerBackfillRunner hook + finale backfill step through the walkthrough. Behavior by mode: interactive defaults to enabled with an opt-out confirm; --yes / --dry-run run automatically; --dry-run scans without writing; --no-daemon still backfills (local file import). - Bound the import by the selected retention window and an `until` cutoff captured at finale entry so it never overlaps live gateway capture. - Surface per-provider results in the finale summary (provider, dryRun, ok, scanned, rowsWritten, skipped) and emit a walkthrough.backfill span. Flush happens inside runProvider so `hyp query` sees rows immediately. Tests: onboarding runs backfill when claude is picked; --dry-run plan without writes; --yes auto-runs bounded; finale summary stats; interactive consent yes/no; unavailable provider skipped; thrown runner contained; runBackfillProvider unit tests. npm test (621) + typecheck + lint green. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…8cow) (#70) Register a `codex` backfill provider in @hypaware/codex that imports local Codex history into ai_gateway_messages via `hyp backfill codex`. - Discovers ~/.codex/sessions/** rollout files (CODEX_HOME aware), parsing both the modern line-delimited {timestamp,type,payload} format and the legacy single-document {session,items} format, best-effort and version-defensively. - Groups one projected exchange per session, preserving workspace, git origin/commit/dirty, sandbox, entrypoint, client version, and sidechain/subagent signals. - Preserves native message ids when present; otherwise leaves identity to the gateway's deterministic fallback. Stamps attributes.codex.identity_source ('native' | 'gateway_fallback') plus codex provenance mirroring the live exchange projector. - Emits projected OpenAI exchanges (provider 'openai', conversation_source 'codex') that the @hypaware/ai-gateway materializer expands into the same canonical rows as live capture. - Detects but never parses ChatGPT/Codex app+browser storage (emits an `unsupported_location` event); treats history.*/log as diagnostic-only; excludes the gateway cache. Tested end-to-end through the real ai-gateway materializer; verified against a 60-session sample of real local rollouts (zero parse warnings, balanced tool_call/tool_result pairing). Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Bead 5 of client-backfill (rollout phase 6): wire the onboarding picker finale to run the Codex backfill provider when `codex` is selected, the same way Claude was wired in bead 3 (#69). No production change was required. Bead 3 built the picker finale backfill step generically (`runFinaleBackfill` in src/core/cli/walkthrough.js): it intersects `clientsPicked` (already `('claude'|'codex')[]`) with the registered providers exposed by `buildPickerBackfillRunner` = `ctx.backfills.list()`, runs each, pushes a per-provider entry onto the finale summary (`provider: string`), and guards every provider in a try/catch so one failure neither aborts siblings nor the daemon restart. Bead 4 (#70) registered the `codex` provider in @hypaware/codex's `activate()`. @hypaware/codex is in the bundled set and `hyp init` boots with `bootProfile=all-available`, so codex activates, registers its provider, and appears in `ctx.backfills.list()` — the picker then runs it. The "claude in V1" scope of bead 3 was only because no codex provider existed yet; nothing in the finale was claude-specific. Verified end-to-end against the real CLI (hermetic HOME/HYP_HOME): hyp init --yes --source codex --dry-run --no-daemon → (dry-run) backfill codex: ok (scanned 0, wrote 0, skipped 0) hyp init --yes --source claude --source codex --dry-run --no-daemon → backfill claude: ok / backfill codex: ok `--no-daemon` still backfills (local file import) and `--dry-run` scans without writing, matching the bead contract. This change locks that behavior in with tests: - walkthrough-backfill.test.js: codex selected runs the step and records stats; both claude+codex run both providers in deterministic order; interactive codex pick prompts consent and runs on yes; a failing provider (claude) does not abort the sibling (codex). Reworded the stale "only claude is registered" comment on the empty-intersection skip test (codex is a real provider now; the test drives skip-logic via the mock's `available` set). - boot-installed.test.js: the real all-available boot (the `hyp init` profile) registers both `claude` and `codex` backfill providers — the real-wiring counterpart to the mocked-runner walkthrough tests. npm test (637) + npm run typecheck + npm run lint all green. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…hy-xbrk5) (#72) Add a narrow PRE-WRITE dedupe to the ai-gateway backfill materializer: before a batch is written, rows whose part_id already exists in a committed ai_gateway_messages partition are skipped, so rerunning `hyp backfill <provider>` writes zero new rows. Falls back to message_id + part_index for transitional rows that predate part_id. The scan is feature-detected and degrades gracefully; cache-maintenance compaction stays a backup dedupe layer, not the primary guarantee. Add hermetic smokes backfill_claude_fixture, backfill_codex_fixture, and walkthrough_backfill_client_history. Each asserts both the user-visible ai_gateway_messages query result and the internal backfill telemetry (dev_run_id, provider, row counts); the two provider smokes also prove an idempotent rerun adds no duplicate rows. Traditional tests cover rerun determinism, partial-interrupt rerun, the part_id fallback key, in-run accumulation, and graceful degradation. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

philcunliffe · 2026-05-29T05:06:23Z

Dual-agent review — `request_changes`

Verdict: request_changes
Risk class: medium
Auto-merge advisory: 👎 thumbs down - verdict is request_changes; changes still need human-gated follow-up
Blast-radius: computed inline (sub-formula dispatch had stalled; impact surface analyzed directly) — see risk.md

Auto-merge advisory

👎 thumbs down - verdict is request_changes; changes still need human-gated follow-up

Advisory only: no merge was attempted.

Risk capstone

Cross-reference: reviewer findings that intersect high-risk surfaces

Source	Finding (severity, evidence)	Intersects
Codex	Error Handling & Resilience (major) — silent skip+success on `materializer_missing`/`dataset_mismatch`/`dataset_not_registered`; `backfill.js:398,409,457,562`	Targets (backfill.js) + Risks #1 (silent data loss)
Codex	Contract & Interface Fidelity (major) — invalid `--since`/`--until` accepted & propagated; `backfill.js:23,681,838`	Targets (backfill.js) + Risks #2 (unvalidated time-window flags)
Codex	Release Safety (major) — `PromptCancelledError` may escape cancel flow; `walkthrough.js:656,1164`, `tui/runtime.js:109`	Targets (walkthrough.js) + Risks #4 (onboarding cancellation)
Claude	`@typedef` in `.js` should be `interface` in `.d.ts` (convention); `backfill.js:310-322`	Targets (backfill.js); low-risk (style/convention) surface
Claude	inline `import('...')` type should be hoisted `@import`; `test/core/walkthrough-backfill.test.js:16-18`	test-only surface; no runtime blast-radius intersection

Blast radius

Silent data loss masked as success — missing materializer /
dataset_mismatch / dataset_not_registered are treated as skip+ok, so
a provider run can drop all rows while reporting ok
(src/core/commands/backfill.js:398,409,457,562). Highest-impact risk.
Unvalidated time-window flags — invalid --since/--until are
accepted and propagated, potentially widening scope to a full-history
import (src/core/commands/backfill.js:23,681,838).
Dedupe miss windows — interrupted-run / pre-flush spool rows are
invisible to the next run's committed-partition scan; relies on
downstream compaction to collapse duplicates (dataset.js dedupe).
Onboarding cancellation path — PromptCancelledError from the
finale backfill consent may escape the established cancel flow/exit code
(src/core/cli/walkthrough.js:656,1164; src/core/cli/tui/runtime.js:109).
Shared projection refactor — message_projector.js removed 81 lines
while extracting projection helpers; regression risk on any retained
shared projection behavior, though current callers trace to the backfill path.

Codex review

Fix Validations

Backfill reruns duplicating already-committed rows

Status: correct
Evidence: hypaware-core/plugins-workspace/ai-gateway/src/dataset.js:313, hypaware-core/plugins-workspace/ai-gateway/src/dataset.js:321, hypaware-core/plugins-workspace/ai-gateway/src/dataset.js:358, src/core/commands/backfill.js:452, src/core/commands/backfill.js:618
Assessment: The materializer now pre-dedupes by part_id against committed partitions per run, and the runner flushes touched datasets, so normal reruns avoid re-appending committed rows.

Picker onboarding not running client-history backfill

Status: correct
Evidence: src/core/cli/walkthrough.js:1091, src/core/cli/walkthrough.js:1158, src/core/cli/walkthrough.js:1194
Assessment: Finale now intersects picked clients with registered providers and executes each provider with explicit summary accounting.

Findings

Contract & Interface Fidelity

Severity: major
Confidence: high
Evidence: src/core/commands/backfill.js:23, src/core/commands/backfill.js:838, src/core/commands/backfill.js:681
Why it matters: CLI advertises ISO-bounded flags, but invalid --since/--until values are accepted and propagated, which can silently change scope (potentially broad full-history imports).
Suggested fix: Validate --since/--until in parser (Date.parse + since <= until) and fail fast with exit code 2 on invalid input.

Error Handling & Resilience

Severity: major
Confidence: high
Evidence: src/core/commands/backfill.js:398, src/core/commands/backfill.js:409, src/core/commands/backfill.js:562, src/core/commands/backfill.js:457
Why it matters: Missing materializer/dataset contract violations are treated as “skip + success,” so integrations can silently drop all rows while reporting ok.
Suggested fix: Promote materializer_missing, dataset_mismatch, and dataset_not_registered to provider failure (status=failed) and return non-zero for that provider run.

Release Safety

Severity: major
Confidence: medium
Evidence: src/core/cli/tui/runtime.js:109, src/core/cli/walkthrough.js:656, src/core/cli/walkthrough.js:1164
Why it matters: Backfill-consent cancellation can throw PromptCancelledError after config/finale side effects have started, without being translated into the established cancel flow/exit code.
Suggested fix: Catch PromptCancelledError around finale backfill consent and map to the same cancel path used by initial picker prompts.

No Finding

Behavioral Correctness
Change Impact / Blast Radius
Concurrency, Ordering & State Safety
Security Surface
Resource Lifecycle & Cleanup
Test Evidence Quality
Architectural Consistency
Debuggability & Operability

Evidence Bundle

Changed hot paths: src/core/commands/backfill.js, src/core/cli/walkthrough.js, hypaware-core/plugins-workspace/ai-gateway/src/dataset.js
Impacted callers: src/core/cli/walkthrough.js:735, src/core/cli/walkthrough.js:1091, src/core/commands/backfill.js:284, src/core/commands/backfill.js:342, hypaware-core/plugins-workspace/ai-gateway/src/dataset.js:257
Impacted tests: test/core/backfill-command.test.js:1, test/core/walkthrough-backfill.test.js:1, test/plugins/ai-gateway-backfill-materializer.test.js:1, test/plugins/claude-backfill.test.js:1, test/plugins/codex-backfill.test.js:1
Unresolved uncertainty: Did not execute test suite in this review; could not verify runtime behavior of invalid ISO flags across all providers beyond command-layer propagation.
tokens used
132,343

Claude review

Code review

Found 2 issues:

BackfillProviderResult is declared with a JSDoc @typedef inside a .js file; it should be an interface in a .d.ts and @imported (CLAUDE.md says "Do not use @typedef in JSDoc. Define shared types as interfaces in .d.ts files and import them via @import.")

hypaware/src/core/commands/backfill.js

Lines 310 to 322 in 891155a

    
           /** 
        
            * @typedef {{ 
        
            *   provider: string, 
        
            *   plugin: string, 
        
            *   datasets: string[], 
        
            *   items_seen: number, 
        
            *   rows_written: number, 
        
            *   rows_skipped: number, 
        
            *   sessions_seen: number, 
        
            *   status: 'ok' | 'failed', 
        
            *   error?: string, 
        
            * }} BackfillProviderResult 
        
            */

The makeBackfill JSDoc uses an inline import('...') type annotation; the type import should be hoisted to a top-of-file @import and referenced by bare name (CLAUDE.md says "Never use inline import('...') types. Declare type imports at the top of the file with @import JSDoc comments, then reference the bare names.")

hypaware/test/core/walkthrough-backfill.test.js

Lines 16 to 18 in 891155a

    
            * @param {string[]} available 
        
            * @param {Record<string, import('../../src/core/cli/types.d.ts').BackfillFinaleResult>} [entries] 
        
            */

🤖 Generated with Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}

Reports: /Users/phil/testcity/.gc/pr-pipeline/reviews/pr-66 · Bead: hy-0tyaf

…le cancel (#73) Addresses PR #66 dual-review (request_changes) on client-backfill: materializer_missing/dataset_mismatch/dataset_not_registered now set provider status=failed and make runBackfill exit non-zero (no silent skip+success data loss); --since/--until validated via Date.parse + since<=until with fail-fast exit 2; finale backfill-consent PromptCancelledError mapped to the cancel flow; BackfillProviderResult moved to a .d.ts interface; inline import() type hoisted to @import. Implemented via the pair CLI (codex driver + opus reviewer). npm test + 3 backfill smokes green. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…xit 2) (#74) Implements the M2 finding the prior fix (#73) claimed but never landed: parseRunArgv now rejects unparseable --since/--until and since>until with exit code 2. Adds tests for invalid --since, invalid --until, and reversed range. The original fix had no M2 test, so a green suite masked the gap; this re-review (run manually) caught it. Implemented via the pair CLI (codex driver + opus reviewer). npm test + 3 backfill smokes green. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feature-launch and others added 7 commits May 28, 2026 09:29

chore: seed integration/client-backfill for feature flow

3c9a545

philcunliffe mentioned this pull request May 29, 2026

Fix review findings: integration/client-backfill (hy-44y6q) #73

Merged

philcunliffe mentioned this pull request May 29, 2026

fix(backfill): validate --since/--until (M2 follow-up) #74

Merged

philcunliffe marked this pull request as ready for review May 29, 2026 15:37

philcunliffe merged commit 747ab14 into master May 29, 2026
6 checks passed

philcunliffe deleted the integration/client-backfill branch May 29, 2026 15:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature: client-backfill#66

Feature: client-backfill#66
philcunliffe merged 9 commits into
masterfrom
integration/client-backfill

philcunliffe commented May 28, 2026

Uh oh!

philcunliffe commented May 29, 2026

Fix Validations

Backfill reruns duplicating already-committed rows

Picker onboarding not running client-history backfill

Findings

Contract & Interface Fidelity

Error Handling & Resilience

Release Safety

No Finding

Evidence Bundle

Code review

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

philcunliffe commented May 28, 2026