Skip to content

feat(phase-61): admin backfill-index (61-01) - #51

Open
RichardHightower wants to merge 1 commit into
feature/phase-60-live-backend-isolationfrom
feature/phase-61-backfill-index
Open

feat(phase-61): admin backfill-index (61-01)#51
RichardHightower wants to merge 1 commit into
feature/phase-60-live-backend-isolationfrom
feature/phase-61-backfill-index

Conversation

@RichardHightower

Copy link
Copy Markdown
Contributor

Closes the pre-v3.1 empty text_preview gap (#41 / OPS-01). Stacked on #49 (needs the sequence-0 checkpoint fix).

What

memory-daemon stop
memory-daemon admin backfill-index --index bm25 --from-sequence 0
memory-daemon start --foreground
  • Default start resumes from the index checkpoint (second run = 0 new).
  • --from-sequence 0 replays the event log so stores whose outbox is already drained still get text into Tantivy (idempotent upsert by event_id).
  • --dry-run prints counts, writes nothing (checkpoint unchanged).
  • Refuses if the daemon PID is live; the error names memory-daemon stop.
  • Progress n/total on stderr every batch. max_batches=1 in tests is the SIGINT-after-batch-1 equivalent.

This is not admin rebuild-indexes (TOC/grips) and not admin rebuild-bm25 (a prune). Vector backfill of raw events is still a no-op without grips; use --index bm25 for this gap. Import.rs was not cherry-picked — it is not on this line.

A v3.0 RocksDB dump at 68ab122 is not in-tree (impractical). The e2e seeds 50 events with real OutboxEntry JSON, never runs the indexer, then backfills.

Verify (targeted; same bar as #49 / #50)

cargo fmt --check
cargo clippy -p memory-storage -p memory-indexing -p memory-daemon --all-targets -- -D warnings
cargo test -p memory-storage --lib -- test_get_events_page test_count_outbox_from
cargo test -p memory-indexing --lib -- backfill
cargo test -p memory-daemon --lib -- backfill refuse
cargo test -p e2e-tests --test backfill_test

All of the above are green locally.

Honest caveats

  • --index all / --index vector loads Candle. The OPS-01 e2e uses --index bm25.
  • Event-log fallback assigns synthetic sequences 0..N-1 and pins the checkpoint to max(last, outbox_head-1) so a later live drain does not re-process.

Stopped-daemon CLI that replays events into BM25 so pre-v3.1 empty
text_preview documents can be filled. Default start resumes from the
index checkpoint; --from-sequence 0 walks the event log (outbox may
already be drained). --dry-run counts and writes nothing. A second
run reports 0 new. Refuses to run if the daemon PID is live and names
memory-daemon stop.

Not rebuild-indexes (TOC/grips) and not rebuild-bm25 (a prune).
Vector backfill of raw events is still a no-op without grips.

OPS-01 / #41.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants