Feature client-backfill: Bead 6: Smoke coverage + idempotency tightening (hy-xbrk5)#72
Merged
Merged
Conversation
…hy-xbrk5) Add a narrow PRE-WRITE dedupe to the ai-gateway backfill materializer: before a batch is written, rows whose part_id already exists in a committed ai_gateway_messages partition are skipped, so rerunning `hyp backfill <provider>` writes zero new rows. Falls back to message_id + part_index for transitional rows that predate part_id. The scan is feature-detected and degrades gracefully; cache-maintenance compaction stays a backup dedupe layer, not the primary guarantee. Add hermetic smokes backfill_claude_fixture, backfill_codex_fixture, and walkthrough_backfill_client_history. Each asserts both the user-visible ai_gateway_messages query result and the internal backfill telemetry (dev_run_id, provider, row counts); the two provider smokes also prove an idempotent rerun adds no duplicate rows. Traditional tests cover rerun determinism, partial-interrupt rerun, the part_id fallback key, in-run accumulation, and graceful degradation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
philcunliffe
added a commit
that referenced
this pull request
May 29, 2026
* chore: seed integration/client-backfill for feature flow * feat(backfill): core registry + CLI + AI gateway materializer (hy-8wfal) (#67) Bead 1 of the client-backfill feature (rollout phases 1-2). Phase 1 — core: - Add BackfillRegistry and BackfillMaterializerRegistry (register/get/list, duplicate + shape validation) in src/core/registry/backfills.js. - Expose ctx.backfills / ctx.backfillMaterializers during plugin activation and on CommandRunContext; wire both through the kernel runtime and dispatcher. - Add `hyp backfill [provider...]`, `hyp backfill list`, `hyp backfill plan`: provider selection (config-filtered by default, explicit names override), retention/date-window resolution, dry-run (scan-only), and JSON output. - Emit the backfill.* telemetry envelope (start/plan/provider_start/scan/ materialize/write/flush/provider_finish/finish) with dev_run_id, provider, dataset, and row counts. Writes route through storage.appendRows so rows land in the dataset's per-source partitions exactly like live capture. Phase 2 — AI gateway materializer: - Extract aiGatewayRowsFromProjectedExchange() as the single row-expansion path shared by live capture and backfill; createAiGatewayMessageProjector delegates to it with a persistent per-listener conversation state, so live behavior is unchanged. - Register the ai_gateway.projected_exchange materializer (dataset ai_gateway_messages) converting projected exchanges into canonical rows, stamping backfill provenance (hashed source path, native id). Tests: registry register/list/duplicate/validation; CLI parsing, provider selection, retention resolution; dry-run writes nothing; materializer parity with the live projector (native + fallback identity). npm test (600 pass) and npm run typecheck both green. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(claude): backfill provider for local Claude transcripts (hy-eqtd7) (#68) Register a `claude` backfill provider in @hypaware/claude that imports local Claude Code JSONL transcripts into ai_gateway_messages through the ai-gateway `ai_gateway.projected_exchange` materializer, so backfilled and live-captured rows are identical for the same conversation. - Discover ~/.claude/projects/**/*.jsonl, group by sessionId, and join session-context.jsonl for cwd/git_branch. - Preserve native DAG identity (uuid -> message_id/provider_uuid, parentUuid -> previous_message_id/parent_uuid) plus role/content, tool calls/results, permission mode, sidechain, and compact metadata. - Store only a minimized native frame in raw_frame (never the full transcript); tag conversation_source/client_name = 'claude'. - Deterministic reruns: identity and timestamps come from the immutable transcript and the materializer is pure. Reuse: surface role/content on TranscriptEntry and export walkTranscriptFiles/loadTranscriptFile from transcripts.js. Tests: fixture -> projected exchange -> canonical rows, rerun determinism, session grouping, native DAG identity, minimized raw_frame, and since-window filtering. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(onboarding): claude backfill step in the picker finale (hy-tlyll) (#69) Bead 3 of client-backfill. When the onboarding picker selects a client with a registered backfill provider (claude in V1), import its local history into the query cache right after the config write and before the daemon (re)start that resumes live capture. - Add runBackfillProvider() to commands/backfill.js: runs one named provider end-to-end (scan -> materialize -> write -> flush) via the shared runProvider path, so finale-imported rows land in the same per-source tables as `hyp backfill <provider>` and live capture. - Thread a PickerBackfillRunner hook + finale backfill step through the walkthrough. Behavior by mode: interactive defaults to enabled with an opt-out confirm; --yes / --dry-run run automatically; --dry-run scans without writing; --no-daemon still backfills (local file import). - Bound the import by the selected retention window and an `until` cutoff captured at finale entry so it never overlaps live gateway capture. - Surface per-provider results in the finale summary (provider, dryRun, ok, scanned, rowsWritten, skipped) and emit a walkthrough.backfill span. Flush happens inside runProvider so `hyp query` sees rows immediately. Tests: onboarding runs backfill when claude is picked; --dry-run plan without writes; --yes auto-runs bounded; finale summary stats; interactive consent yes/no; unavailable provider skipped; thrown runner contained; runBackfillProvider unit tests. npm test (621) + typecheck + lint green. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(codex): backfill provider for local Codex session rollouts (hy-m8cow) (#70) Register a `codex` backfill provider in @hypaware/codex that imports local Codex history into ai_gateway_messages via `hyp backfill codex`. - Discovers ~/.codex/sessions/** rollout files (CODEX_HOME aware), parsing both the modern line-delimited {timestamp,type,payload} format and the legacy single-document {session,items} format, best-effort and version-defensively. - Groups one projected exchange per session, preserving workspace, git origin/commit/dirty, sandbox, entrypoint, client version, and sidechain/subagent signals. - Preserves native message ids when present; otherwise leaves identity to the gateway's deterministic fallback. Stamps attributes.codex.identity_source ('native' | 'gateway_fallback') plus codex provenance mirroring the live exchange projector. - Emits projected OpenAI exchanges (provider 'openai', conversation_source 'codex') that the @hypaware/ai-gateway materializer expands into the same canonical rows as live capture. - Detects but never parses ChatGPT/Codex app+browser storage (emits an `unsupported_location` event); treats history.*/log as diagnostic-only; excludes the gateway cache. Tested end-to-end through the real ai-gateway materializer; verified against a 60-session sample of real local rollouts (zero parse warnings, balanced tool_call/tool_result pairing). Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(onboarding): codex backfill in the picker finale (hy-8nhpc) (#71) Bead 5 of client-backfill (rollout phase 6): wire the onboarding picker finale to run the Codex backfill provider when `codex` is selected, the same way Claude was wired in bead 3 (#69). No production change was required. Bead 3 built the picker finale backfill step generically (`runFinaleBackfill` in src/core/cli/walkthrough.js): it intersects `clientsPicked` (already `('claude'|'codex')[]`) with the registered providers exposed by `buildPickerBackfillRunner` = `ctx.backfills.list()`, runs each, pushes a per-provider entry onto the finale summary (`provider: string`), and guards every provider in a try/catch so one failure neither aborts siblings nor the daemon restart. Bead 4 (#70) registered the `codex` provider in @hypaware/codex's `activate()`. @hypaware/codex is in the bundled set and `hyp init` boots with `bootProfile=all-available`, so codex activates, registers its provider, and appears in `ctx.backfills.list()` — the picker then runs it. The "claude in V1" scope of bead 3 was only because no codex provider existed yet; nothing in the finale was claude-specific. Verified end-to-end against the real CLI (hermetic HOME/HYP_HOME): hyp init --yes --source codex --dry-run --no-daemon → (dry-run) backfill codex: ok (scanned 0, wrote 0, skipped 0) hyp init --yes --source claude --source codex --dry-run --no-daemon → backfill claude: ok / backfill codex: ok `--no-daemon` still backfills (local file import) and `--dry-run` scans without writing, matching the bead contract. This change locks that behavior in with tests: - walkthrough-backfill.test.js: codex selected runs the step and records stats; both claude+codex run both providers in deterministic order; interactive codex pick prompts consent and runs on yes; a failing provider (claude) does not abort the sibling (codex). Reworded the stale "only claude is registered" comment on the empty-intersection skip test (codex is a real provider now; the test drives skip-logic via the mock's `available` set). - boot-installed.test.js: the real all-available boot (the `hyp init` profile) registers both `claude` and `codex` backfill providers — the real-wiring counterpart to the mocked-runner walkthrough tests. npm test (637) + npm run typecheck + npm run lint all green. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(backfill): pre-write part_id dedupe + hermetic backfill smokes (hy-xbrk5) (#72) Add a narrow PRE-WRITE dedupe to the ai-gateway backfill materializer: before a batch is written, rows whose part_id already exists in a committed ai_gateway_messages partition are skipped, so rerunning `hyp backfill <provider>` writes zero new rows. Falls back to message_id + part_index for transitional rows that predate part_id. The scan is feature-detected and degrades gracefully; cache-maintenance compaction stays a backup dedupe layer, not the primary guarantee. Add hermetic smokes backfill_claude_fixture, backfill_codex_fixture, and walkthrough_backfill_client_history. Each asserts both the user-visible ai_gateway_messages query result and the internal backfill telemetry (dev_run_id, provider, row counts); the two provider smokes also prove an idempotent rerun adds no duplicate rows. Traditional tests cover rerun determinism, partial-interrupt rerun, the part_id fallback key, in-run accumulation, and graceful degradation. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backfill): fail on skip-conditions, validate date range, map finale cancel (#73) Addresses PR #66 dual-review (request_changes) on client-backfill: materializer_missing/dataset_mismatch/dataset_not_registered now set provider status=failed and make runBackfill exit non-zero (no silent skip+success data loss); --since/--until validated via Date.parse + since<=until with fail-fast exit 2; finale backfill-consent PromptCancelledError mapped to the cancel flow; BackfillProviderResult moved to a .d.ts interface; inline import() type hoisted to @import. Implemented via the pair CLI (codex driver + opus reviewer). npm test + 3 backfill smokes green. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backfill): validate --since/--until (Date.parse + since<=until, exit 2) (#74) Implements the M2 finding the prior fix (#73) claimed but never landed: parseRunArgv now rejects unparseable --since/--until and since>until with exit code 2. Adds tests for invalid --since, invalid --until, and reversed range. The original fix had no M2 test, so a green suite masked the gap; this re-review (run manually) caught it. Implemented via the pair CLI (codex driver + opus reviewer). npm test + 3 backfill smokes green. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: feature-launch <feature-launch@gas.city> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Feature client-backfill: Bead 6: Smoke coverage + idempotency tightening
Implemented (phases 7+8). Phase 8: narrow pre-write dedupe in the ai-gateway backfill materializer (dataset.js) — scans committed ai_gateway_messages partitions for existing part_id and skips matches before write, with message_id+part_index fallback for transitional rows; feature-detected + graceful, per-run memo keyed by dev_run_id. Compaction stays the backup layer. Phase 7: three hermetic smokes (backfill_claude_fixture, backfill_codex_fixture, walkthrough_backfill_client_history) asserting both query rows and backfill telemetry (dev_run_id/provider/row counts); the two provider smokes also assert idempotent rerun = zero new rows. +6 traditional dedupe tests. Verified: npm test 643 pass, npm run typecheck clean, npm run lint clean, all 3 smokes ok. Branch polecat/hy-xbrk5.
Delivery
This PR is auto-merged into the integration branch once CI is green. Human
review for the full feature happens on the downstream PR
(
integration/client-backfill -> <final target>), which is opened separately when the featureis ready to ship.