Skip to content

Add recursive scanning (-r), plus --version, --min-size, --exclude #14

Description

@fahadsiddiqui

Status

Partly resolved by #17. Rewritten to describe what is actually left.

Done on main: flag parsing is wired up. --help works and exits 0, unknown flags exit 2 with a message naming the flag instead of stat --help: no such file or directory, a second positional argument is rejected rather than silently ignored, and --json / --quiet / --no-script exist.

Still open: recursive scanning, and the smaller flags below.

Recursive scanning (-r / --recursive)

The single most likely thing a user wants, and still impossible — scan() is non-recursive by design and only reads the top level of the given directory.

PR #1 implemented this against the old findDuplicates() and was closed when #17 replaced that function; the logic could not be rebased onto a function that no longer exists. The work is smaller now:

  • Thread a recursive bool through scanWithProgress() in scan.go.
  • Swap os.ReadDir for filepath.WalkDir (not filepath.WalkWalkDir avoids a per-entry Lstat).
  • Skip .git, node_modules, and vendor via fs.SkipDir, with an --exclude PATTERN flag to extend the list.

The exclusion list is not optional

Measured with PR #1's build, on a git repo containing exactly one real file:

files processed under -r:     27
files processed without -r:    1

Twenty-six of those are .git internals. Git objects are content-addressed, so this is both a large amount of wasted I/O and a source of meaningless duplicate groups.

Interaction with #7 — handle in the same change

Priority scoring only looks at filepath.Base, so recursion sharply increases equal-priority ties across directories. Measured with PR #1's build:

✗ DELETE: t3/pkg_b/__init__.py    (tied with pkg_a/__init__.py)
✗ DELETE: t3/vendor/lib/LICENSE   (tied with pkg_a/LICENSE)

Both are structurally required files, and the loser is picked by path order rather than by evidence that it is a copy. Recursion turns this from an edge case into the common case, so #7 should be settled first or in the same PR.

Cross-directory duplicates also want their own tie-break rule once #7 defines one — prefer the shallower path, or decline to recommend across directories at all.

Remaining smaller flags

flag purpose
--version version string, built from ldflags
--min-size N ignore files below N bytes (interacts with #6)
--exclude PATTERN skip matching names; needed by -r above

Tests to write first

  • -r finds duplicates across subdirectories; without it, subdirectories are untouched (there is already a TestScanSkipsSubdirectories guarding the default).
  • Excluded directories are not descended into — assert on a fixture containing a .git directory.
  • A path that begins with - is still handled, via --.
  • Cross-directory ties behave per whatever Equal-priority ties still nominate a file for deletion with no evidence it is a copy #7 decides.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions