feat!: replace ctx duplicates with MinHash structural near-duplicate detection#7
Merged
Merged
Conversation
…l search Rework ctx duplicates as a MinHash-based near-duplicate detector over normalized token shingles: - New src/fingerprint.rs: tree-sitter tokenization (identifiers -> ID, literals -> LIT, comments dropped), FNV-1a k=5 shingles, 128-perm MinHash (SplitMix64 seeds, LE blobs, deterministic across platforms), 16x8 LSH banding with exact-Jaccard verification of candidates. - ctx index now fingerprints function/method symbols incrementally per changed file into the new symbol_fingerprints table (cascade-deleted with symbols); serial/parallel/watch paths share one store funnel. - Index schema bumped to v2; v0/v1 databases now fail with the existing 'run ctx index --force' SchemaVersionMismatch message. - New CLI: --threshold <0.0-1.0 Jaccard, default 0.85, clamped at 0.5>, --min-tokens <N, default 50>, --against <REF>, --fail-on-found (exit 1 via Outcome::Findings), global --json envelope with SymbolRef pairs, similarity, token counts, skipped_languages. - Old line-based detector fully removed: find_duplicates, normalize_code, calculate_similarity, md5_hash, DuplicateResult, the --similarity/--min-lines/--output flags, and the stale shell completion entry. Solidity is skipped (solang-parser, no tree-sitter). BREAKING CHANGE: --threshold is a shingle-set Jaccard similarity, not a percentage of matching lines; existing indexes must be rebuilt with ctx index --force.
Update README, docs/, the docs website mirror, the agents guide, and the JSON output contract for the reworked ctx duplicates: new --threshold/--min-tokens/--against/--fail-on-found flags, the duplicates JSON envelope payload, the Solidity skip, and the breaking-change / reindex notes in the changelog.
# Conflicts: # CHANGELOG.md
# Conflicts: # docs/json-output.md # src/commands/mod.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR 4 of 7 in the Quality Intelligence Suite. Stacked on #2 (
feat/quality-foundation) — will be retargeted tomainonce #2 merges; review only the top 2 commits until then.Breaking: replaces the old O(n²) line-based
ctx duplicatesin place — same command name, entirely new semantics. Old--similarity/--min-linesflags and the line-based detector are deleted (no hidden fallback, nodupesalias). Existing indexes must be rebuilt (ctx index --force, schema v2).symbol_fingerprintstable, populated incrementally duringctx index.--threshold 0.85(Jaccard, clamped ≥0.5 with a warning),--min-tokens 50,--against <REF>(diff-scoped),--fail-on-found(exit 1 = findings, for hooks/CI), global--json(includesskipped_languages: ["solidity"]— solang has no tree-sitter grammar).store_file_implfunnel (the parallel path had an inlined copy), and rejects pre-versioning v0 databases instead of silently stamping them (they'd yield silently-empty results).Test plan
--fail-on-foundexit codes; Solidity → zero rows.cargo test(201 green),clippy -D warnings,fmt --check,--no-default-featuresbuild — all clean.🤖 Generated with Claude Code