Add incremental branch indexing for fast-moving branches - #1
Closed
simianhacker wants to merge 2 commits into
Closed
Add incremental branch indexing for fast-moving branches#1simianhacker wants to merge 2 commits into
simianhacker wants to merge 2 commits into
Conversation
Introduce an isolated `update: incremental` mode for branch selectors that maintains a single mutable, ref-addressed view of a moving branch in new `sourcerer-v2-*` indices, re-indexing only the files changed since the last run instead of re-snapshotting the whole repository on every move. The default `snapshot` path and all existing `sourcerer-v1-*` behavior are unchanged. - config: `Selector.update_mode` (`snapshot` default / `incremental`); rejects incremental on non-branch selectors and with `since`/`retain`; selection raises on a branch selected in both modes and dedupes same-mode overlap. - indices/templates: `sourcerer-v2-files`, `sourcerer-v2-lines`, and a `index.mode: lookup` single-shard `sourcerer-v2-refs`; case-sensitive (no-normalizer) `git.ref_key`/`ref`/`org`/`repo` fields. - documents: ref-addressed v2 file/line builders (no commit in the id) and a targeted `index_paths_v2` ingest that indexes a supplied path iterable. - git: pure NUL-safe `plan_changes` diff planner (add/modify/delete/type-change/ copy/rename) with an explicit `base_missing` fallback. - markers: one mutable v2 refs doc per branch with indexing/ready/failure writes that only advance the completed commit after success; synchronous, ref-scoped `delete_by_query` helpers (`wait_for_completion`, `conflicts=proceed`). - orchestration: incremental branch units bypass the v1 skip/retention path; initial/no-op/targeted-update/retry/missing-base-rebuild flows with failure isolation per unit. - agent builder: `refs.list` returns both schemas with `update_mode`; content tools take an exact `git_ref_key` and use mutually-exclusive snapshot/ref-key scoping plus a `LOOKUP JOIN sourcerer-v2-refs` to attach the completed commit for citations; ref-resolution/citations skills document the flow. - setup: creates the v2 refs lookup index and empty schema-anchor indices so the dual-schema ES|QL resolves `git.commit`/`git.ref_key` on any cluster. - docs: README, AGENTS.md, repos.example.yml cover config, the mixed-revision consistency window, the v1/v2 boundary, and a local evaluation procedure.
- refs.list: tolerate null git.commit so an incremental branch mid- (or failed) first index is still listed (was silently filtered by null LIKE "*"); + test. - AGENTS.md: document that setup must be re-run after upgrade before the tools work (setup creates the v2 lookup + schema-anchor indices before deploying the Agent Builder tools, so a single setup keeps the ordering correct).
Owner
Author
|
Superseded by elastic#1 (filed against upstream). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds an opt-in
update: incrementalmode for branch selectors. Instead of writing afull commit-addressed snapshot on every branch move, an incremental branch keeps one
mutable, ref-addressed view in new
sourcerer-v2-*indices and re-indexes only the fileschanged since the last run (typically 10–20), which keeps refresh cost proportional to the
diff rather than the whole repo. The default snapshot path and all existing
sourcerer-v1-*behavior are unchanged.
How to Verify
Against an Elasticsearch/ES|QL cluster that supports
index.mode: lookupandLOOKUP JOIN(9.x):
uv run pytest tests/— 327 tests pass (uv buildalso exits 0).sourcerer setup— loads the v1+v2 templates and creates the v2 refs lookup index plusthe schema-anchor indices.
sourcerer index --config repos.yml. Confirmsourcerer-v2-refshas exactly one doc forthe branch with
status: ready,update_mode: incremental, and the completed commit.only the changed paths were processed, deleted/rename-source paths leave no docs, and an
unchanged file's
_seq_nois unchanged.sourcerer.refs.listshows the incremental ref; pass its exactgit_ref_keytosourcerer.code.grep/sourcerer.files.catand confirm rows carry thecompleted commit (attached by the
LOOKUP JOIN).An end-to-end run of steps 2–5 was executed against a live ES 9.5 cluster and passed,
including the mid-first-index visibility case in
sourcerer.refs.list.Breaking Changes
None for existing configs (omitting
updatekeeps the v1 snapshot behavior). Operationalnote: the content and refs tools now query both schemas and
LOOKUP JOIN sourcerer-v2-refs,so re-run
sourcerer setupafter upgrading before relying on the tools — a singlesetupcreates the required indices before it (re)deploys the Agent Builder tools.