feat: add --ignore-from to exclude paths with gitignore-style patterns - #358
Merged
Conversation
Reads gitignore-syntax patterns from one or more files and leaves everything they match out of the report, in both aggregate and interactive mode. This is the equivalent of rsync's --exclude-from and restic's --exclude-file, so the same pattern file can answer "how much of this would actually be backed up?". Matching is powered by gix-ignore, which is already a dependency, so negation, anchoring, `**` and directory-only patterns all behave exactly like Git. Excluded directories are pruned from the walk rather than only hidden, and excluded top-level paths are dropped before the walk so they are absent from the report instead of appearing as empty. Fixes Byron#277 Fixes Byron#312
Owner
|
Thanks a lot! This works! |
<!-- agent --> - [P2] Preserve all repeated ignore files across command levels — src/main.rs:284-288 When `--ignore-from` is supplied both before and after a subcommand, this merge discards every subcommand-level file whenever the global list is nonempty. For example, `dua --ignore-from first aggregate --ignore-from second` loads only `first`, so patterns in `second` never apply and the documented later-file precedence is violated; concatenate the lists in command-line order instead. - [P2] Keep the ignore base stable across interactive refreshes — src/common.rs:320-323 In interactive mode with multiple absolute roots outside the current directory, refreshing a subtree passes that subtree as a new traversal root, causing this fallback to rebase all patterns. An entry initially excluded by `nested/secret` therefore reappears after refreshing `nested` because it is now matched as `secret`; conversely, anchored patterns can begin excluding deeper entries. Preserve each original input root as the pattern base during refresh scans. Assisted-by: GPT 5.6 Co-authored-by: GPT 5.6 <codex@openai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #277
Fixes #312
Adds
--ignore-from FILE, which reads gitignore-style patterns and leaves everything they match out of the report — in both aggregate and interactive mode.The motivation from both issues is the same: people already keep an exclude list for their backup tool, and they want to point
duaat it to answer "how much of this is actually being backed up?".--ignore-dirscan't do that — it takes absolute directory paths, so there's no way to say**/node_modules/or*.log.The option is repeatable (later files win over earlier ones), works on the root command and both subcommands, and can be set via
DUA_IGNORE_FROM.On the design
In #312 you said you'd want this powered by
gix-glob, and that it should be able to exclude or include. So this usesgix::ignore::Search, which is already pulled in through theexcludesfeature — no new dependency. That also means the include case falls out for free through gitignore's own!negation rather than a separate flag, which keeps it to one option and one well-known syntax. Negation, anchoring,**, and directory-only patterns all behave exactly like Git, including the rule that you can't re-include something under an excluded directory.Matching is case-sensitive on every platform. Git flips this based on
core.ignorecase, but here that would mean the same pattern file producing different totals on Linux and macOS, which seemed worse than the inconsistency with Git.Two things that turned out to matter more than I expected:
What the patterns are matched against. My first attempt matched paths relative to each walk root, and the flag then did nothing for plain
dua. The reason isextract_paths_maybe_set_cwd— given a single directory,duachanges into it and walks each of its entries as a separate root, so every top-level directory is at depth 0 and atarget/pattern never sees a path to match. Patterns are now matched against the pathduareports an entry under, i.e. relative to the directory being looked at, which is the same thing a.gitignoreat the top of a repo matches. Roots outside the current directory have no such path and fall back to being relative to the root they were found under;pattern_relative_pathis the whole rule and is unit-tested directly.Excluded directories are pruned, not just hidden. Exclusion is checked in the
descendpredicate and in the event filter, so a matching directory is neither descended into nor emitted — otherwise its children would arrive without a parent inintegrate_traversal_event. Excluded top-level paths are dropped inextract_paths_maybe_set_cwdbefore the walk starts, because filtering them during the walk left them in the output as0 Brows, which reads as "this directory is empty" rather than "this directory was excluded".Cost when unused
The exclusion predicate short-circuits on an empty pattern set before touching a path, so an invocation without
--ignore-fromdoes one atomic-free bool check per entry and nothing else. When the option is used, directories are matched twice (once to decide descent, once to decide emission) — one extraentry.path()per directory. Threading the decision fromdescendthrough to the event would need a change inwalk.rs, and it didn't seem worth it for a cost only paid by people who asked for the feature. Happy to do it if you'd rather.Verifying
mkdir -p demo/{src,target/debug,node_modules/pkg,logs} head -c 200000 /dev/urandom > demo/target/debug/binary head -c 150000 /dev/urandom > demo/node_modules/pkg/index.js head -c 50000 /dev/urandom > demo/logs/app.log head -c 3000 /dev/urandom > demo/logs/important.log head -c 8000 /dev/urandom > demo/src/main.rs printf 'target/\n**/node_modules/\n*.log\n!important.log\n' > ignore.txt dua aggregate -A demo # 411.61 KB total dua aggregate -A --ignore-from ignore.txt demo # 11.22 KB total: src, plus logs at 3.13 KB dua i -A --ignore-from ignore.txt demo # same in the TUI dua aggregate -A --ignore-from ignore.txt demo/target # still 200.10 KB — asking for it directly wins dua aggregate --ignore-from missing.txt demo # exits 1: Failed to read ignore patterns from missing.txtI ran the TUI one under a pty to confirm the tree matches.
Tests and scope
Nine new tests: gitignore semantics through the public API (negation, dir-only, anchoring,
**, comments), precedence across multiple files, a missing pattern file being an error rather than silently empty, pruning through a real walk, the path-resolution rule, input-path filtering, an end-to-end assertion thataggregate's reported size drops, and CLI parsing.make check-fmt clippy unit-testsis clean.One journey test, "a broken link in multiple roots", fails for me on macOS — it expects exit 1 and gets 0. It fails identically on
9bc8f74with no changes applied, so it's not from this PR; every other journey test passes. I left it alone rather than fold an unrelated fix in here.Deliberately out of scope: a separate
--include-from(negation covers it),.gitignorediscovery during traversal (that's the existingItoggle's job), and a config-file key, since--ignore-dirsdoesn't have one either.Regression risk is mostly in
iter_from_paths, where the event filter was rewritten: if it were wrong, entries would go missing or root skipping would break with no pattern file present at all. The existing walk and traversal tests plus the journey snapshots cover that path, andwalk_options_fromnow returns aResult, which is the only signature change to a public-ish function.Disclosure: I used Claude Code while working on this. I've read every line, and the verification above is output I actually ran.