Skip to content

fix: fingerprint Solidity functions in ctx duplicates (AGE-2)#30

Merged
saldestechnology merged 1 commit into
mainfrom
johan/age-2-duplicates-solidity-functions-are-skipped-no-tree-sitter
Jul 11, 2026
Merged

fix: fingerprint Solidity functions in ctx duplicates (AGE-2)#30
saldestechnology merged 1 commit into
mainfrom
johan/age-2-duplicates-solidity-functions-are-skipped-no-tree-sitter

Conversation

@saldestechnology

Copy link
Copy Markdown
Collaborator

Summary

ctx duplicates silently skipped all Solidity functions. The MinHash near-duplicate detector normalizes each function into token shingles via tree-sitter, and Solidity has no tree-sitter grammar in this build (it's parsed with solang-parser), so .sol functions never got fingerprints at index time. On a Solidity-heavy repo this made the detector — and the perception of coverage — blind to the largest part of the codebase.

Approach

Rather than add tree-sitter-solidity (which conflicts with the pinned tree-sitter ~0.20.10 — the documented reason the project chose solang in the first place), tokenize Solidity with the solang-parser lexer already in the dependency tree.

  • New tokenize_solidity in src/fingerprint.rs: lexes with solang_parser::lexer::Lexer and maps tokens with the same normalization as the tree-sitter path (normalize_leaf): identifiers → ID; string/hex/address/number literals → LIT; keywords and punctuation kept verbatim; comments dropped (the lexer excludes them). Each token's 1-indexed line is derived from its byte offset so file_fingerprints slices by symbol line range exactly as for other languages.
  • tokenize short-circuits to this path for Language::Solidity; ts_language still returns None for it.
  • Removes the now-false "skipped" surfaces: skipped_languages in duplicates --json no longer lists solidity, the human-readable skip note is gone, and the duplicates/module docs are updated.

Scope note

ctx audit's Duplication score is a separate name-based heuristic that never consults MinHash, so this change fixes ctx duplicates but does not by itself move the audit score. Rewiring audit's duplication category onto fingerprints is a possible follow-up, not included here.

Tests

  • New test_tokenize_solidity_normalization (identifiers→ID, literals→LIT, renamed identifiers produce an identical stream).

  • Inverted test_solidity_files_produce_no_fingerprintstest_solidity_files_produce_fingerprints.

  • Updated tests/duplicates_cli.rs skip-list expectation and added an end-to-end test asserting two structurally-identical .sol functions are detected as a near-duplicate pair.

  • cargo fmt --all -- --check

  • cargo clippy --all-targets -- -D warnings ✅ (0 warnings)

  • cargo test ✅ (304 lib + all integration suites, 0 failures)

Refs AGE-2. Sibling of AGE-4 (modifier edges) and AGE-5 (qualified-call resolution).

ctx duplicates silently skipped all Solidity functions: the MinHash detector
tokenizes each function via tree-sitter, and Solidity has no tree-sitter
grammar in this build (it uses solang-parser), so .sol functions never got
fingerprints. On a Solidity-heavy repo that made `ctx duplicates` blind to
the largest part of the codebase.

Tokenize Solidity with the solang-parser lexer (already a dependency) in
`tokenize`, mapping identifiers -> ID, string/hex/address/number literals ->
LIT, and keeping keywords/punctuation verbatim -- the same normalization the
tree-sitter path uses. Solidity functions now get MinHash fingerprints at
index time and participate in near-duplicate detection. Removes the now-false
"skipped" messaging from `duplicates` output and help; inverts the index-level
test and adds tokenizer + end-to-end duplicate tests for .sol.

Note: `ctx audit`'s Duplication score is a separate name-based heuristic that
does not consult MinHash, so its score is unaffected by this change; rewiring
audit to use fingerprints is a possible follow-up.

Refs AGE-2
@saldestechnology
saldestechnology force-pushed the johan/age-2-duplicates-solidity-functions-are-skipped-no-tree-sitter branch from ee43493 to a906e78 Compare July 11, 2026 14:19
@saldestechnology
saldestechnology merged commit 92cee73 into main Jul 11, 2026
6 checks passed
@saldestechnology
saldestechnology deleted the johan/age-2-duplicates-solidity-functions-are-skipped-no-tree-sitter branch July 11, 2026 15:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant