Skip to content

feat: add --ignore-from to exclude paths with gitignore-style patterns - #358

Merged
Byron merged 2 commits into
Byron:mainfrom
pelazas:feat/ignore-from
Aug 3, 2026
Merged

feat: add --ignore-from to exclude paths with gitignore-style patterns#358
Byron merged 2 commits into
Byron:mainfrom
pelazas:feat/ignore-from

Conversation

@pelazas

@pelazas pelazas commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Fixes #277
Fixes #312

Adds --ignore-from FILE, which reads gitignore-style patterns and leaves everything they match out of the report — in both aggregate and interactive mode.

The motivation from both issues is the same: people already keep an exclude list for their backup tool, and they want to point dua at it to answer "how much of this is actually being backed up?". --ignore-dirs can't do that — it takes absolute directory paths, so there's no way to say **/node_modules/ or *.log.

cat .duaignore
# target/
# **/node_modules/
# *.log
# !important.log

dua --ignore-from .duaignore

The option is repeatable (later files win over earlier ones), works on the root command and both subcommands, and can be set via DUA_IGNORE_FROM.

On the design

In #312 you said you'd want this powered by gix-glob, and that it should be able to exclude or include. So this uses gix::ignore::Search, which is already pulled in through the excludes feature — no new dependency. That also means the include case falls out for free through gitignore's own ! negation rather than a separate flag, which keeps it to one option and one well-known syntax. Negation, anchoring, **, and directory-only patterns all behave exactly like Git, including the rule that you can't re-include something under an excluded directory.

Matching is case-sensitive on every platform. Git flips this based on core.ignorecase, but here that would mean the same pattern file producing different totals on Linux and macOS, which seemed worse than the inconsistency with Git.

Two things that turned out to matter more than I expected:

What the patterns are matched against. My first attempt matched paths relative to each walk root, and the flag then did nothing for plain dua. The reason is extract_paths_maybe_set_cwd — given a single directory, dua changes into it and walks each of its entries as a separate root, so every top-level directory is at depth 0 and a target/ pattern never sees a path to match. Patterns are now matched against the path dua reports an entry under, i.e. relative to the directory being looked at, which is the same thing a .gitignore at the top of a repo matches. Roots outside the current directory have no such path and fall back to being relative to the root they were found under; pattern_relative_path is the whole rule and is unit-tested directly.

Excluded directories are pruned, not just hidden. Exclusion is checked in the descend predicate and in the event filter, so a matching directory is neither descended into nor emitted — otherwise its children would arrive without a parent in integrate_traversal_event. Excluded top-level paths are dropped in extract_paths_maybe_set_cwd before the walk starts, because filtering them during the walk left them in the output as 0 B rows, which reads as "this directory is empty" rather than "this directory was excluded".

Cost when unused

The exclusion predicate short-circuits on an empty pattern set before touching a path, so an invocation without --ignore-from does one atomic-free bool check per entry and nothing else. When the option is used, directories are matched twice (once to decide descent, once to decide emission) — one extra entry.path() per directory. Threading the decision from descend through to the event would need a change in walk.rs, and it didn't seem worth it for a cost only paid by people who asked for the feature. Happy to do it if you'd rather.

Verifying

mkdir -p demo/{src,target/debug,node_modules/pkg,logs}
head -c 200000 /dev/urandom > demo/target/debug/binary
head -c 150000 /dev/urandom > demo/node_modules/pkg/index.js
head -c 50000  /dev/urandom > demo/logs/app.log
head -c 3000   /dev/urandom > demo/logs/important.log
head -c 8000   /dev/urandom > demo/src/main.rs
printf 'target/\n**/node_modules/\n*.log\n!important.log\n' > ignore.txt

dua aggregate -A demo                        # 411.61 KB total
dua aggregate -A --ignore-from ignore.txt demo   # 11.22 KB total: src, plus logs at 3.13 KB
dua i -A --ignore-from ignore.txt demo       # same in the TUI
dua aggregate -A --ignore-from ignore.txt demo/target  # still 200.10 KB — asking for it directly wins
dua aggregate --ignore-from missing.txt demo # exits 1: Failed to read ignore patterns from missing.txt

I ran the TUI one under a pty to confirm the tree matches.

Tests and scope

Nine new tests: gitignore semantics through the public API (negation, dir-only, anchoring, **, comments), precedence across multiple files, a missing pattern file being an error rather than silently empty, pruning through a real walk, the path-resolution rule, input-path filtering, an end-to-end assertion that aggregate's reported size drops, and CLI parsing. make check-fmt clippy unit-tests is clean.

One journey test, "a broken link in multiple roots", fails for me on macOS — it expects exit 1 and gets 0. It fails identically on 9bc8f74 with no changes applied, so it's not from this PR; every other journey test passes. I left it alone rather than fold an unrelated fix in here.

Deliberately out of scope: a separate --include-from (negation covers it), .gitignore discovery during traversal (that's the existing I toggle's job), and a config-file key, since --ignore-dirs doesn't have one either.

Regression risk is mostly in iter_from_paths, where the event filter was rewritten: if it were wrong, entries would go missing or root skipping would break with no pattern file present at all. The existing walk and traversal tests plus the journey snapshots cover that path, and walk_options_from now returns a Result, which is the only signature change to a public-ish function.

Disclosure: I used Claude Code while working on this. I've read every line, and the verification above is output I actually ran.

Reads gitignore-syntax patterns from one or more files and leaves everything
they match out of the report, in both aggregate and interactive mode. This is
the equivalent of rsync's --exclude-from and restic's --exclude-file, so the
same pattern file can answer "how much of this would actually be backed up?".

Matching is powered by gix-ignore, which is already a dependency, so negation,
anchoring, `**` and directory-only patterns all behave exactly like Git.
Excluded directories are pruned from the walk rather than only hidden, and
excluded top-level paths are dropped before the walk so they are absent from
the report instead of appearing as empty.

Fixes Byron#277
Fixes Byron#312
@Byron
Byron force-pushed the feat/ignore-from branch from b03e705 to fa1ab05 Compare August 3, 2026 08:23
@Byron

Byron commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Thanks a lot! This works!

@Byron
Byron force-pushed the feat/ignore-from branch from fa1ab05 to 2c52909 Compare August 3, 2026 09:08
<!-- agent -->
- [P2] Preserve all repeated ignore files across command levels — src/main.rs:284-288
  When `--ignore-from` is supplied both before and after a subcommand, this
merge discards every subcommand-level file whenever the global list is nonempty.
For example, `dua --ignore-from first aggregate --ignore-from second` loads
only `first`, so patterns in `second` never apply and the documented later-file
precedence is violated; concatenate the lists in command-line order instead.

- [P2] Keep the ignore base stable across interactive refreshes — src/common.rs:320-323
  In interactive mode with multiple absolute roots outside the current
directory, refreshing a subtree passes that subtree as a new traversal root,
causing this fallback to rebase all patterns. An entry initially excluded by
`nested/secret` therefore reappears after refreshing `nested` because it is now
matched as `secret`; conversely, anchored patterns can begin excluding deeper
entries. Preserve each original input root as the pattern base during refresh
scans.

Assisted-by: GPT 5.6
Co-authored-by: GPT 5.6 <codex@openai.com>
@Byron
Byron force-pushed the feat/ignore-from branch from 2c52909 to af83aa5 Compare August 3, 2026 09:09
@Byron
Byron merged commit 1db53ae into Byron:main Aug 3, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pattern based exclude/include optially given through file on disk Add option --ignore-from FILE same as rsync's --exclude-from

2 participants