feat(index): resolve each session's source repo at index time#86
Merged
Conversation
Add a `repo` column — the owner/name of the session's checkout git remotes (space-joined; any remote, so a fork's upstream is included; empty when unresolvable). Resolved once per session from the transcript's recorded cwd via `git -C <cwd> remote`, cached per cwd, and stamped by the indexer. Appended last (the add_columns slot, after `harness`). Attribution now happens when the checkout is present and fresh, and is stored — so it survives the checkout later being deleted or moved, and a reader (curation) just reads the column instead of poking git at read time. src/repo.rs holds the resolver (cwd extraction, url normalization), shared with the backfill migration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds the `repo` column to a local store via Lance add_columns, resolving each session's value from its transcript cwd with the same src/repo.rs resolver the indexer uses. Additive — no rewrite, no re-embed, no reindex; vectors and indexes untouched. Skips a store already carrying `repo`. Kept as an `[[example]]` (run: `cargo run --example backfill_repo`); delete once every store carries `repo`. The remote backfill follows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`hf_dataset::add_column` runs Lance add_columns through the capture-store and lands the new column's per-fragment files in one head-guarded commit — the same pattern as append/reindex/rename_column; data, vectors, and indexes untouched. The `backfill_remote_repo` example resolves each session's repo from the local transcripts (the remote mirrors this machine) and applies it. Single guarded attempt (a backfill runs with no concurrent writers). Kept as an `[[example]]`; delete with backfill_repo and add_column once every store carries `repo`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR moves session-to-repository attribution from read time to index time by resolving git remotes for a session’s recorded working directory and persisting the result in a new repo column. This makes curation attribution robust even if the original checkout is later moved/deleted and avoids repeated live git lookups during reads.
Changes:
- Add
src/repo.rsresolver to extract transcriptcwdand normalizegit remote -vURLs intoowner/nameidentities. - Extend the Lance schema and indexing pipeline to stamp each chunk with a per-session
repovalue (cached percwdduring indexing). - Add one-off backfill examples for local and Hub stores, plus a remote
hf_dataset::add_columnhelper to landadd_columnsoutput in a single guarded commit.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/repo.rs | Adds transcript cwd extraction + git remote normalization/resolution + unit tests. |
| src/lib.rs | Exposes the new repo module. |
| src/index.rs | Adds repo column to schema/batches and stamps chunk repo at index time with a per-cwd cache. |
| src/hf_dataset.rs | Adds add_column() to commit remote add_columns outputs in one guarded commit. |
| src/hello.rs | Updates test/fixture chunk construction for the new repo field. |
| src/chunk.rs | Adds repo field to Chunk and wires it through chunk (de)serialization paths. |
| scripts/backfill_repo.rs | Local-store one-off backfill example using Lance add_columns. |
| scripts/backfill_remote_repo.rs | Hub-store one-off backfill example using remote add_column commit pattern. |
| Cargo.toml | Registers the two new backfill examples. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
repo.rs resolves a repo from a cwd — its module and fn docs shouldn't
narrate the indexer, the `repo` column, curation, or the backfill (those
are callers). Same for the Chunk field doc ("stamped by the indexer") and
add_column's "a backfill runs with no concurrent writers". Trimmed each to
what the code in front of you does.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`identity()` trimmed `.git` before the trailing `/`, so a `…/name.git/` remote kept `.git` (stored as `owner/name.git`). Trim slashes first, then strip a `.git` suffix. Added the trailing-slash test case. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Resolve the source repository a session belongs to (
owner/nameof its checkout's git remotes) once at index time and store it in a newrepocolumn, rather than resolving it live when a reader needs it.Why
This prepares the introduction of curated memories associated to git repos.
The curation process will attribute a session to a publication by its repo. Doing that resolution at read time — reading the transcript's cwd and running
gitagainst it — is fragile: it only works if the checkout still exists on this machine, at that path, right now. A deleted or moved checkout, a removed worktree, a temp dir, or a session synced from another machine all come back unattributable. It also re-pokesgiton every read.Resolving at index time captures the attribution when the checkout is present and fresh, and stores it — so it survives the checkout later being deleted or moved, and a reader just reads a column.
How
src/repo.rs— the resolver: the transcript's recorded cwd (top-levelcwdfor Claude/pi,payload.cwdfor Codex) →git -C <cwd> remote→ each remote URL normalized toowner/name, space-joined (any remote, so a fork'supstreamis included). Empty when unresolvable.repocolumn — appended last, theadd_columnsslotharnessalready uses (the column-order tripwire test pins it). The indexer resolves each unit once, cached per cwd, and stamps its chunks.[[example]]s, kept until every store carriesrepo) — additive via Lanceadd_columns: no data rewrite, no re-embed, no reindex.backfill_repo— local~/.funes/store.backfill_remote_repo— a Hub store, resolving from the local transcripts (a personal mirror mirrors this machine);hf_dataset::add_columnlands the new column's files in one head-guarded commit via the capture store, the same pattern asappend/reindex/rename_column.repoand verify the column at the end.Notes
repo. Disable session-end auto-indexing, backfill, then resume.curate's discovery switches from livegitto a plain read of this column.Tests
Resolver unit tests (URL normalization across ssh/https/HF, multi-harness cwd extraction, all-remotes listing), the column-order tripwire updated, full suite green.
🤖 Generated with Claude Code