Skip to content

Document rationale and trade-offs of --no-dedup flag #1848

Description

@Widthdom

Summary

--no-dedup (QueryCommandRunner.cs:2822-2823) disables result deduplication across overlapping chunks, but the help text and USER_GUIDE never explain when a user would want it. The flag's existence reveals an underlying design choice — chunks have 10-line overlap, so a single FTS hit can land in two adjacent chunks and surface as two results — but a user reading help has no way to know whether to use the flag for "more recall" or "performance" or "this is a debug toggle". The result: the flag is either ignored (most users) or used at random.

Where

  • src/CodeIndex/Cli/QueryCommandRunner.cs:2822-2823 (--no-dedup wiring)
  • src/CodeIndex/Database/DbSearchReader.cs:360, 484 (dedup application)

Suggested approach

(1) Expand the --no-dedup help text: explain that chunks overlap by 10 lines, dedup collapses duplicate hits across the overlap, and disabling it surfaces every raw hit (useful for: debugging chunk-boundary bugs, computing exact match densities, correlating with the chunks table directly). (2) Document the trade-off in USER_GUIDE under "Result deduplication". (3) Add a regression test that runs the same query with and without --no-dedup against a fixture with a known overlap-boundary hit and asserts the count delta matches expectation. (4) Consider a --debug-chunks flag that's a clearer-named alternative for the debug use case while keeping --no-dedup for backwards compat. (5) Cross-link with #1593 (find_in_file) — same chunk model. (6) Cross-link with the just-filed #1810 (tool examples).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions