diffctx over-selection: PPR/edge relevance filter pulls in unrelated files (whole-tree over-dump + co-located k8s manifests)

[nikolay-e]

Observed

diffctx . --diff aebae3a..f176e10 on a commit range whose actual diff is 13 files / 637 lines (git diff aebae3a..f176e10 --stat) produced ~130 files / 8895 lines of fragments — including many files the range never touched (e.g. packages/core/, tools/vocab-tools/tests/, backend src, generated api-client) .

Expected

Output scoped to the files in the diff range (plus, at most, tight related context), not a near-full-tree dump.

Repro

diffctx . --diff <from>..<to>   # where the range touches ~13 files
# → emits ~130 file fragments / ~8900 lines
git diff <from>..<to> --stat    # ground truth: 13 files

Env

• diffctx 1.8.1
• monorepo (npm workspaces + python), range = a single feature commit + its parent

Impact

Over-dump (~88k tokens here) makes --diff unusable for focused review; consumers must fall back to git diff. Likely the range filter isn't applied to fragment selection.

[nikolay-e]

Still reproduces at 1.9.2 (HEAD 6e0aab7), confirmed during a QA pass by dogfooding on this repo's own history.

diffctx . --diff 152ba111..6e0aab7      # range touches 22 files (git diff --name-only)
# → fragment_count: 816, ~101k tokens, fragments drawn from 192 distinct source files

The pulled-in set includes files the range never touched, e.g. tests/cases/diff/gap_076_ruby_import_chain_music_playlist.yaml (verified absent from git diff 152ba111..6e0aab7 --name-only).

The budget-tuning commit 6e0aab7 (lower auto_max 124k→48k, per-fragment core cap) reduced but did not eliminate the over-dump.

Likely mechanism (same family as #59): the range includes .github/workflows/*.yml edits, and YAML↔YAML structural/sibling edges link those to every other YAML doc in the tree — including the tests/cases/diff/*.yaml fixtures — so PPR expansion drags the fixture corpus in as 'context'. A single-commit range over Rust-only changes (776dd1f..6e0aab7) stays tight (65 fragments / 9.5k tokens), which is consistent with the YAML-edge hypothesis rather than a global range-filter bug.

Suggested direction: down-weight or gate cross-file structural edges between co-located config/YAML siblings (mirrors the #59 k8s-manifest precision gap), validated against the yaml_cases benchmark to avoid recall regression.

[nikolay-e]

Still reproduces on diffctx 1.9.2 — and it is systemic, not monorepo-specific

QA pass dogfooded diffctx . --diff on this repo's own session range (12 changed
files, ~226 lines, mostly docs/config):

• changed_files is correctly scoped to the 12 changed files (so the range
  filter IS applied to diff detection — the 1.8.1 "ignores range" framing is
  too narrow).
• But fragment selection over-expands: 542 fragments / 163 distinct files
  / 67,798 tokens for an 18-changed-fragment diff. 150 of 163 files (92%)
  are pulled-in context the range never touched — the entire benchmarks/
  tree, .github/workflows/{cd,automerge,sensitivity-check}.yml,
  .pre-commit-config.yaml, CHANGELOG.md, dozens of tests/** and
  diffctx/src/**.

Root cause (hypothesis, code-read)

Over-expansion, not range-ignoring. Two contributors:

1. Prose↔code lexical edges — src/diffctx/edges/similarity/lexical.rs
   tokenizes prose fragments the same as code; a changed Markdown file that
   mentions cli, tau, budget, pipeline (and cites ~150 file:line
   paths, as PRODUCT.md/README do) gets lexical-similarity edges to every code
   file sharing those tokens, which PPR then expands into.
2. The precision filter that suppresses unrelated fragments for code cases is
   not holding in general.

Scope is much larger than this one repro

Running the full yaml_cases benchmark locally (CI caps at
DIFFCTX_YAML_CASES_LIMIT=20, hiding it): 2260 passed, 465 failed, and
425 of the 465 failures are forbidden_rate=100% (selection pulled in every
fragment the case marks "must NOT select"); only 8 are recall misses. So this is
the dominant precision bug across the whole corpus, spanning every language —
not a monorepo edge case. #59 (k8s manifests) is one slice of the same root cause.

Why not fixed in this QA pass

The fix is a benchmark-validated recalibration of the core selection
precision/recall operating point, which is coupled to the research paper
(docs/parameter-strategy.md is "the contract between implementation and the
paper") and is already being iterated commit-by-commit
(fix(budget): … curb over-selection). A half-validated edge-weight change
pushed without the full sweep would risk regressing recall. Tracking with this
quantified evidence; the unblock is a deliberate calibration sweep, not access.

[nikolay-e]

Update: diffctx 1.10.0 shipped two precision fixes that touch this issue's root cause but do not resolve it:

• Graph::add_edge now rejects self-edges (src == dst).
• the test→source import-edge regex is line-anchored (no longer matches import in comments/strings).

Both cut a class of spurious edges, but the dominant over-selection (prose↔code lexical expansion + general PPR over-reach; 425/465 full-suite failures are forbidden_rate=100%) is unchanged. Keeping open as the central over-selection tracker.

[nikolay-e]

Still reproduces on diffctx 1.10.0 (QA pass, fresh numbers)

Dogfooded diffctx . --diff 80d450b0~1..f6de4ad on this repo (range = 1.10.0 fix + release + qa-docs + dep bumps):

• Ground truth: 12 changed files / 168 insertions (git diff --stat).
• Output: 440 fragments / 185 distinct files / 47,279 tokens. Only ~18 fragments carry a non-context role; ~96% (422/440) are pulled-in context, spanning the whole benchmarks/, tests/**, diffctx/src/**, and workflow trees the range never touched.

Discriminator (per QA.md Diff-Mode Self-Eat): 185 output files vs 12 changed = 15× file expansion, context-dominated → over-selection, not a big diff. Consistent with the YAML/prose-edge hypothesis already recorded here.

Real-repo sweep this pass for contrast (same build): numpy 10-file diff → 67 files / 211 frags (~6.7×), react-native 1-file → 32 files / 53 frags. Docs/config-heavy ranges blow up worst, matching the prose↔code lexical-edge theory.

No blind edge-weight edit mid-QA (regression risk across the 2725-case suite; current baseline 2260 passed / 465 failed, unchanged this pass). Tracking here for the benchmark-validated recalibration.

[nikolay-e]

Fresh reproduction (QA pass, range 62c1ac6..HEAD)

Dogfooded diffctx . --diff 62c1ac6e..HEAD (7-commit range: 1.10.0 release + qa-docs + the #70 wall-clock fix + 3 dependency bumps) via the pipx binary.

• Ground truth: 3 changed files (QA.md, diffctx/Cargo.lock, diffctx/src/main.rs).
• Output: 49 fragments / 21 distinct files, only 1 fragment carries role: changed (8 total role-tagged fragments).

The 18 extra pulled-in files are unrelated to the actual change (--timeout wall-clock enforcement + doc updates): pipeline.rs, ppr.rs, tokenizer.rs, test_harness.rs, README.md, .github/workflows/ci.yml, .github/workflows/bench-sweep.yml, benchmarks/calibrate.py, benchmarks/loo_swebench.py, tests/test_benchmark_runner.py, plus 3 unrelated tests/cases/diff/*.yaml fixtures (powershell, kubernetes, java) and a stray k8s/ml/training-job.yaml / src/main/java/.../Dog.java snippet from those fixtures. All appear to be lexically pulled in by the literal word "timeout" appearing across the codebase (matches the prose↔code lexical-edge theory already tracked here) — same root cause, another data point, not a new bug class.

No edge-weight changes made mid-QA per the standing guidance on this issue.

[nikolay-e]

Merged with #59 — same root cause, consolidating into one tracker

Closing #59 as duplicate; both are the same underlying defect (the relevance/precision filter that should exclude unrelated fragments doesn't hold), just surfacing through two different edge families:

1. Prose↔code lexical edges (this issue's original repro): a changed Markdown/README/QA.md file that mentions common tokens (cli, tau, budget, pipeline, or cites file:line paths) gets lexical-similarity edges to every code file sharing those tokens, which PPR then expands into. Confirmed on every re-run this and prior QA passes (numpy 10→67 files, react-native 1→32 files, this repo's own history 3→21 files, 12→185 files).
2. Co-located structural/sibling edges for k8s manifests (7 kubernetes diff cases score 0 (forbidden_rate=100%) — pull unrelated manifests #59's repro): manifests in the same directory, all kind:-bearing YAML "definitions", get linked via structural/sibling (and possibly same-name) edges regardless of whether they reference the same resources — PPR expansion pulls in every sibling manifest even when the diff only touches one.

Consolidated evidence

• Full yaml_cases suite (2725 cases, CI only runs the first 20 via DIFFCTX_YAML_CASES_LIMIT=20 so this is invisible in CI): 2260 passed, 465 failed, and 425 of the 465 failures are forbidden_rate=100% (selection pulls in every fragment marked "must not select"); only 8 are recall misses. This baseline has been stable and unchanged across at least three QA passes (1.9.2 → 1.10.0 → current) — no regression, but also no progress, since the fix requires deliberate benchmark-validated recalibration rather than a drive-by edit.
• Dogfooding this repo's own history at 1.10.0: 62c1ac6e..HEAD (7-commit range) — ground truth 3 changed files, output 21 distinct files / 49 fragments, only 1 fragment carries role: "changed".
• 1.10.0 shipped two precision fixes that touch the root cause but do not resolve it: Graph::add_edge rejects self-edges; the test→source import-edge regex is now line-anchored. Both cut a class of spurious edges without moving the 2260/465 baseline.

#59's specific acceptance criteria (kept as the k8s regression-test slice)

These 7 cases are the concrete, minimal reproduction of edge family #2 above — use them to validate any future edge-weight recalibration, in addition to the full-suite score:

case	recall	forbidden_rate
kubernetes_005_configmap_reference	100%	100%
kubernetes_007_volume_mount	100%	100%
kubernetes_008_resource_limits	100%	100%
kubernetes_012_sidecar_container	80%	100%
kubernetes_017_service_account	75%	100%
kubernetes_026_ingress_rules	100%	100%
kubernetes_038_service_topology	75%	100%

cd diffctx
cargo test --release --test yaml_cases -- kubernetes_005_configmap_reference
# score 0.0% < min 10.0% (recall=100%, forbidden_rate=100%)

Fix direction (unchanged, still not a QA-pass-sized change)

A real fix is a benchmark-validated relevance recalibration:

• Down-weight or condition structural/sibling edges among co-located config/YAML manifests so they don't bridge on directory-adjacency alone.
• Tighten (or separate) the lexical-similarity edge builder so prose fragments don't get full code-token lexical weight against arbitrary code files.
• Validate the change against the full 2725-case suite (not just these 7 or the CI-visible 20) plus the SWE-bench/contextbench harness in benchmarks/, since a change that only targets these 7 cases risks the exact overfitting trap already flagged in this issue's history.

Tracking centrally here going forward. No edge-weight changes made mid-QA per the standing guidance.

[nikolay-e]

Fresh minimal real-world repro (QA sweep 2026-07-03, diffctx 1.10.2 pipx): numpy @ 93d61add, diffctx . --diff HEAD~1 on a ONE-line change (actions/checkout version-comment bump in .github/workflows/scorecards.yml). Output: 13 fragments across 5 files — 1 changed + 12 context, including 6 fragments from .github/workflows/linux.yml (other checkout pin sites — defensible) and 3 clearly unrelated source fragments: numpy/_core/src/common/npy_longdouble.h (declaration npy_longdouble), numpy/_core/code_generators/generate_umath.py, numpy/_core/_add_newdocs_scalars.py. Small in absolute tokens (1,250) but 12:1 context:changed on a comment-only diff, with C-header declarations pulled in for a workflow pin bump — a clean, tiny fixture for the lexical-edge over-pull component of this issue.

[nikolay-e]

Another occurrence, same shape. diffctx . --diff 356dcde..HEAD in lingua-quiz produced 57 fragments / 12.5k tokens including files never touched by the range (apps/backend/src/api/v2/auth.py, apps/backend/src/main.py::openapi_with_auth_responses, tools/vocab-tools/vocab_tools/core/ssot.py::build, tools/vocab-tools/vocab_tools/cli/commands/track.py, several packages/core/tests/*.test.ts import lines, tools/codegen/generate-all.sh).

Ground truth:

git diff --stat 356dcde..HEAD -- . ':!package-lock.json'
 .github/workflows/README.md          |  22 ++--
 QA.md                                |   2 +-
 packages/api-client/package.json     |   2 +-
 packages/core/package.json           |   9 +-
 tools/vocab-tools/build_skeletons.py | 197 -----------------------------------
 tools/vocab-tools/chunk.py           |  27 -----

6 files, not the ~15+ distinct symbols/files diffctx surfaced. diffctx 1.8.1 again.

[nikolay-e]

Still reproduces on diffctx 1.10.2 (QA pass 2026-07-05, fresh numbers)

Dogfooded diffctx . --diff 3a918b55..ebb54e90 on this repo (range = dep bump

• PDF rebuild + one added LaTeX line — about as minimal a diff as exists).

• Ground truth: 3 changed files (diffctx/Cargo.lock, main.pdf, main.tex),
  3 insertions / 2 deletions (git diff --stat).
• Output: 6 fragments / 5 distinct files / 9,205 tokens. Only 1/6 fragments
  carries role: "changed" (the added main.tex line); the other 5 are
  pulled-in context from tests/cases/diff/javascript_007_describe_module.yaml,
  diffctx/src/config/importance.rs, benchmarks/aggregate_sweep.py (two
  separate functions), and benchmarks/baselines/aider_subprocess.py — none
  touched by the range.

Discriminator (per QA.md Diff-Mode Self-Eat): 5 output files vs 3 changed =
still over-selection even on a near-trivial diff with zero code changes in the
touched files. Consistent with the prose/lexical-edge hypothesis already
recorded here — main.tex prose apparently pulls in unrelated Python/Rust/YAML
fragments via lexical similarity.

No blind edge-weight edit mid-QA (regression risk, current full yaml_cases
baseline unchanged this pass). Adding as another data point for the
benchmark-validated recalibration.