Skip to content

feat(index): resolve each session's source repo at index time#86

Merged
dacorvo merged 5 commits into
mainfrom
feat/repo-column
Jul 16, 2026
Merged

feat(index): resolve each session's source repo at index time#86
dacorvo merged 5 commits into
mainfrom
feat/repo-column

Conversation

@dacorvo

@dacorvo dacorvo commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

What

Resolve the source repository a session belongs to (owner/name of its checkout's git remotes) once at index time and store it in a new repo column, rather than resolving it live when a reader needs it.

Why

This prepares the introduction of curated memories associated to git repos.

The curation process will attribute a session to a publication by its repo. Doing that resolution at read time — reading the transcript's cwd and running git against it — is fragile: it only works if the checkout still exists on this machine, at that path, right now. A deleted or moved checkout, a removed worktree, a temp dir, or a session synced from another machine all come back unattributable. It also re-pokes git on every read.

Resolving at index time captures the attribution when the checkout is present and fresh, and stores it — so it survives the checkout later being deleted or moved, and a reader just reads a column.

How

  • src/repo.rs — the resolver: the transcript's recorded cwd (top-level cwd for Claude/pi, payload.cwd for Codex) → git -C <cwd> remote → each remote URL normalized to owner/name, space-joined (any remote, so a fork's upstream is included). Empty when unresolvable.
  • repo column — appended last, the add_columns slot harness already uses (the column-order tripwire test pins it). The indexer resolves each unit once, cached per cwd, and stamps its chunks.
  • Backfill (disposable [[example]]s, kept until every store carries repo) — additive via Lance add_columns: no data rewrite, no re-embed, no reindex.
    • backfill_repo — local ~/.funes/store.
    • backfill_remote_repo — a Hub store, resolving from the local transcripts (a personal mirror mirrors this machine); hf_dataset::add_column lands the new column's files in one head-guarded commit via the capture store, the same pattern as append/reindex/rename_column.
    • Both skip a store that already has repo and verify the column at the end.

Notes

  • A store must be backfilled (or freshly indexed) before the new binary appends to it — the schema now has repo. Disable session-end auto-indexing, backfill, then resume.
  • Follow-up (separate branch): curate's discovery switches from live git to a plain read of this column.

Tests

Resolver unit tests (URL normalization across ssh/https/HF, multi-harness cwd extraction, all-remotes listing), the column-order tripwire updated, full suite green.

🤖 Generated with Claude Code

dacorvo and others added 3 commits July 16, 2026 21:27
Add a `repo` column — the owner/name of the session's checkout git remotes
(space-joined; any remote, so a fork's upstream is included; empty when
unresolvable). Resolved once per session from the transcript's recorded cwd
via `git -C <cwd> remote`, cached per cwd, and stamped by the indexer.
Appended last (the add_columns slot, after `harness`).

Attribution now happens when the checkout is present and fresh, and is
stored — so it survives the checkout later being deleted or moved, and a
reader (curation) just reads the column instead of poking git at read time.
src/repo.rs holds the resolver (cwd extraction, url normalization), shared
with the backfill migration.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds the `repo` column to a local store via Lance add_columns, resolving
each session's value from its transcript cwd with the same src/repo.rs
resolver the indexer uses. Additive — no rewrite, no re-embed, no reindex;
vectors and indexes untouched. Skips a store already carrying `repo`.

Kept as an `[[example]]` (run: `cargo run --example backfill_repo`); delete
once every store carries `repo`. The remote backfill follows.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`hf_dataset::add_column` runs Lance add_columns through the capture-store
and lands the new column's per-fragment files in one head-guarded commit —
the same pattern as append/reindex/rename_column; data, vectors, and indexes
untouched. The `backfill_remote_repo` example resolves each session's repo
from the local transcripts (the remote mirrors this machine) and applies it.

Single guarded attempt (a backfill runs with no concurrent writers). Kept as
an `[[example]]`; delete with backfill_repo and add_column once every store
carries `repo`.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR moves session-to-repository attribution from read time to index time by resolving git remotes for a session’s recorded working directory and persisting the result in a new repo column. This makes curation attribution robust even if the original checkout is later moved/deleted and avoids repeated live git lookups during reads.

Changes:

  • Add src/repo.rs resolver to extract transcript cwd and normalize git remote -v URLs into owner/name identities.
  • Extend the Lance schema and indexing pipeline to stamp each chunk with a per-session repo value (cached per cwd during indexing).
  • Add one-off backfill examples for local and Hub stores, plus a remote hf_dataset::add_column helper to land add_columns output in a single guarded commit.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/repo.rs Adds transcript cwd extraction + git remote normalization/resolution + unit tests.
src/lib.rs Exposes the new repo module.
src/index.rs Adds repo column to schema/batches and stamps chunk repo at index time with a per-cwd cache.
src/hf_dataset.rs Adds add_column() to commit remote add_columns outputs in one guarded commit.
src/hello.rs Updates test/fixture chunk construction for the new repo field.
src/chunk.rs Adds repo field to Chunk and wires it through chunk (de)serialization paths.
scripts/backfill_repo.rs Local-store one-off backfill example using Lance add_columns.
scripts/backfill_remote_repo.rs Hub-store one-off backfill example using remote add_column commit pattern.
Cargo.toml Registers the two new backfill examples.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/repo.rs Outdated
Comment thread src/repo.rs
dacorvo and others added 2 commits July 16, 2026 22:39
repo.rs resolves a repo from a cwd — its module and fn docs shouldn't
narrate the indexer, the `repo` column, curation, or the backfill (those
are callers). Same for the Chunk field doc ("stamped by the indexer") and
add_column's "a backfill runs with no concurrent writers". Trimmed each to
what the code in front of you does.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`identity()` trimmed `.git` before the trailing `/`, so a `…/name.git/`
remote kept `.git` (stored as `owner/name.git`). Trim slashes first, then
strip a `.git` suffix. Added the trailing-slash test case.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@dacorvo
dacorvo merged commit 751f7ae into main Jul 16, 2026
8 checks passed
@dacorvo
dacorvo deleted the feat/repo-column branch July 16, 2026 20:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants