Skip to content

fdu 0.3.0

Choose a tag to compare

@github-actions github-actions released this 30 Sep 17:26
· 481 commits to main since this release
Immutable release. Only release title and notes can be modified.
17b23c1

fdu 0.3.0 makes the default command faster on Linux and fixes how a scan treats automount points, unusual .gitignore files, and very large files under content analysis. On Linux the default fdu PATH tree took 48% less time than 0.2.1’s on the Linux kernel source, 14% less on a node_modules tree, and 15% less on a generated million-entry tree, where its peak memory fell from 293 to 57 MiB. In one paired run per tree on a virtualized Linux host, fdu’s default command led pdu, at its default and at --max-depth 2, and diskus on all three trees. A scan on Linux no longer mounts an unmounted automount point by statting it, a .gitignore with a byte-order mark or a NUL byte reads as git reads it, and content analysis no longer holds a one-line source or a large Markdown file whole in memory.

Four Rust API changes are incompatible: counters::Counts, scan::ScanBackendDiagnostics, Error, and content::CoverageReason each gain fields or a variant, and Counts and ScanBackendDiagnostics are now #[non_exhaustive]. No command-line option, report or cache schema, or Python API changed. Read Upgrading from 0.2.1 before updating a Rust caller.

Performance

Every figure here is from Linux, on a quiet 4-vCPU virtualized ext4 host with warm filesystem caches, in paired, interleaved runs. The real trees are the Linux v6.12 source (92,474 entries, 358 .gitignore files) and a directory-dense node_modules (79,957 entries); the generated tree has 1,000,001 entries.

  • Against 0.2.1. Measured end to end in one cell of 20 pairs per tree, the release engine’s default tree took 48% less time than 0.2.1’s on the kernel source (209 to 110 ms), 14% less on node_modules (135 to 116 ms), and 15% less on the generated tree (1.11 to 0.95 s), and its default summary 35%, 12%, and 17% less (exp-202). The development records measured the same work in steps: the overnight round took the default tree 39% and 10% off and the default summary 26% and 7% (exp-194, exp-195), and three later changes took the default tree another 3.1% and 8.9% off and the default summary 6.1% on the kernel source (exp-201).
  • A default tree that keeps only the files it can show. When a one-shot report needs nothing but the tree, as the default fdu PATH does, fdu keeps every directory and only the largest files that could reach the smallest share the tree prints, and folds the rest into their directories’ totals. The answer is unchanged. On the million-entry tree the default tree’s peak memory fell 80%, from 293 to 57 MiB (exp-202). The tree also decides from each child’s totals whether the child will show before building its row, and on Linux takes each directory’s and symlink’s kind from its parent’s listing rather than statting it again (exp-197, exp-199).
  • Faster .gitignore classification. Each ignore file’s rules are indexed by literal name, extension, and suffix, and the rules left over are ruled out by byte checks before any glob runs. On the kernel source, what .gitignore handling adds to the default tree fell from 52% of a walk without it to 1.6% (exp-178, exp-194). The default summary finds the rules for each directory by its bytes rather than its path components (exp-198). Which paths the rules match is unchanged, and fdu’s ignored set agrees with git check-ignore on every path of the kernel source.
  • A native directory reader on Linux. Builds against glibc, which include the Linux Python wheels, read each directory with getdents64 into a reused buffer and stat its entries with statx relative to the directory, on every route that lists directories. A directory the reader cannot list is read the portable way, and musl builds use only the portable reader. On the kernel source the default command’s fstat calls fell from 5,773 to 4, and the default summary took 9–10% less time on both real trees (exp-185, exp-186). Its tests run on x86-64 and arm64 Linux.

Against its peers, each paired 20 times with an adjacent fdu run in one interleaved run per tree, the release’s default command led on all three trees (exp-202):

Tool Work returned Linux v6.12 source node_modules Generated, 1M entries
fdu default tree: five levels, 1% share floor 0.110 s, baseline 0.106 s, baseline 0.951 s, baseline
pdu 0.24.0 --max-depth 2 the root and its children +10% [+2%, +12%] +12% [+2%, +19%] +19% [+15%, +20%]
pdu 0.24.0 default tree: ten levels, 1% floor +15% [+12%, +19%] +18% [+15%, +23%] +25% [+21%, +28%]
diskus 0.9.0 one total +17% [+3%, +20%] +12% [+9%, +17%] +24% [+20%, +28%]
fdu 0.2.1 the same default tree +95% [+86%, +112%] +20% [+18%, +22%] +21% [+18%, +22%]

Each percentage is the median paired change in wall time against the adjacent fdu run, with its 95% interval; positive means the peer took longer. fdu’s time includes reading the kernel tree’s .gitignore files, which neither peer reads, and on every tree its output matched 0.2.1’s byte for byte. The narrowest leads are over pdu --max-depth 2: on both real trees their intervals reach down to +2%. How large a lead is depends on the session. This one ran about 30% slower than exp-201’s for every tool on the real trees; in exp-201 the leads over pdu --max-depth 2 were 3% and 10%. On the generated tree 0.2.1 was level with pdu --max-depth 2 in this run, where earlier sessions’ figures put pdu about 11% ahead of it, so by that estimate about ten points of the 19% lead reflect this session’s host. The previous run on that tree, on an earlier engine from this cycle, had pdu --max-depth 2 3% faster than fdu’s default tree (Linux comparison).

None of these changes was measured on macOS or Windows. The native reader and the automount fix are Linux-only, and taking kinds from the listing changes nothing on macOS, whose listing already carries every child’s attributes, or on Windows, which stats every entry; the folded tree and the .gitignore changes apply on every platform.

What’s Changed

Features

  • .gitignore counters. FDU_COUNTERS=1 reports three more rows in its control state group: ignore patterns tested, the rules whose glob ran against an entry; ignore bucket probes, the lookups of an entry’s name in an ignore file’s indexed rules; and ignore bucket hits, the lookups that found a matching rule.
  • Watch entry points in Rust. watch_session::Session::changed_report answers only when what a reader would see has changed, and changed_report_with_progress does the same while reporting its build as Summarizing; fdu --watch uses both.

Fixes

  • Automount points on Linux. No stat of a listed child triggers an automount: an unmounted autofs trigger under the scanned tree (/net, /misc, a systemd automount unit) is reported as the trigger, as lstat and GNU du report it, and a walk no longer mounts, or hangs on, a network filesystem it would not descend into. 0.2.1 mounted it on glibc builds; musl builds were never affected (exp-196 screened the change for regressions).
  • Directories that list but cannot be searched. A directory that is readable but not searchable (mode 0400) is reported the same way on every route: each of its children as an error, and none as an entry. In 0.2.1 the default --view summary counted such a directory’s subdirectory as a directory and reported it again when it failed to open, and reported nothing for its symlink.
  • .gitignore byte-order marks and NUL bytes. A .gitignore that starts with a UTF-8 byte-order mark now applies its first rule, and a pattern ends at the first NUL byte in its line, as git reads both. 0.2.1 matched the mark as part of the first rule, so that rule never applied, and kept the bytes after a NUL as part of the rule.
  • Content analysis memory. Each worker now holds a bounded part of a file:
    • --analyze code classifies a line in pieces over a 64 KiB window as it arrives, so a one-line minified or generated source costs a worker the window, not the file. Every count is unchanged.
    • --analyze words renders a Markdown file of up to 64 MiB exactly and counts a larger one as plain text, saying so in its row (counted as text), in machine output (text_only), and in a note; 0.2.1 held a Markdown file of any size whole. No option lifts the 64 MiB bound in this release.
    • Analysis schedules candidate files in batches of 4,096 rather than building every candidate before the first read, which cost hundreds of megabytes on a million-file tree. The progress count is still exact.
  • Language detection. A C header is no longer labelled C++ because namespace, template, std::, or constexpr appears inside an identifier, a comment, or a string: on the Linux v6.12 source, 168 of 25,308 headers were. A modeline names a language only as a whole token, so mode: conf-colon is no longer C and ft=css no longer reads as cs, while Emacs mode: shell-script and a Vim compound filetype such as ft=c.doxygen still name their language. Line counts are unchanged; only the language breakdown moves.
  • --watch repaints. fdu --watch repaints its views only when what they print, on standard output or as notes and warnings on standard error, would change: a touch that leaves a file’s size alone, or a change to an entry the selection leaves out, no longer prints the same tree again under a new timestamp. Machine formats still repaint when a modification time they carry moves, and change records are never deduplicated.
  • --watch progress. The progress line stays up, drawn as Summarizing, while the first answer is built, where 0.2.1 showed nothing for the seconds a heavy view such as --view full takes on a large tree.
  • Totals past u64. A tree whose bytes sum past what a u64 holds, as three sparse 8 EiB files on tmpfs, XFS, or btrfs do, fails with Error::UnrepresentableTotal on every route, and fdu exits 1 naming the file; 0.2.1 printed a wrapped total (8,191 PiB) in release builds and panicked in debug builds.
  • fdu --version from a source build. Built from the source distribution inside an unrelated git repository, such as a home directory kept in git, 0.2.1 stamped that repository’s commit into fdu --version and the installed skill’s “Generated by” line. A build now names a commit only when the repository tracks fdu’s own manifest.

Guidelines and Content

  • Agent skill. The skill that --install-skill writes now says that the default tree keeps every directory but only the files large enough to show as rows, and that content analysis, --watch, and an index from the Rust or Python open read and write the snapshot cache, and that --analyze words counts a Markdown file over 64 MiB as plain text and says so.

Documentation

  • Comparison to alternatives. The README compares fdu with du, ncdu, dust, dua, gdu, pdu, diskus, and dumac, and its code analysis with the source-line counters scc and tokei, in features and speed (Comparison to Alternatives). The SLOC tools survey checks the three counters’ answers on the Linux kernel’s C sources.

Install

uvx --no-build fdu@latest --help    # run the latest release without installing it
uv tool install --no-build fdu      # the command line from a prebuilt wheel
cargo install --locked fdu          # the command line, built with Rust 1.85 or newer
pip install fdu                     # the Python package: import fdu
cargo add fdu                       # the Rust library, re-exported with the command line
cargo add fdu-core --features watch # the engine alone, with the watch layer

Re-run uvx --no-build fdu@latest --install-skill after upgrading so the agent skill describes this release.

Compatibility

fdu is 0.x: a minor release may change the command line or the Rust or Python API incompatibly, and the CHANGELOG names each change. This release changes the Rust API in the four places listed below. No command-line option or Python API changed. Reports stay fdu.report/10, cache status fdu.cache/3, and the watch stream fdu.stream/2; a report’s words coverage map has one more possible key, text_only.

Upgrading from 0.2.1

  • First run scans cold. 0.3.0 reuses no snapshot or content sidecar written by 0.2.x: the engine fingerprint includes the release version and the .gitignore semantics version, which is now 4. Each cached tree’s first run walks it, and its first analyzed run reads every file again.
  • Rust: counters::Counts. It gains three public fields, ignore_patterns_tested, ignore_bucket_probes, and ignore_bucket_hits, and is #[non_exhaustive], so a later counter is an additive change. Outside the engine crate, build one with Counts::default() and assign its fields, since neither a struct literal nor ..Counts::default() compiles there, and end a destructuring pattern with ... Code that only reads its fields is unaffected.
  • Rust: scan::ScanBackendDiagnostics. It gains three public fields, linux_dents_attempts, linux_dents_successes, and linux_dents_fallbacks, each an Option<u64> counting the Linux native reader’s listings and None off Linux or on a build without glibc. It is now #[non_exhaustive], so a later field is an additive change: outside the engine crate it cannot be built with a struct literal, and a destructuring pattern ends with ... Code that only reads its fields is unaffected.
  • Rust: Error. It gains UnrepresentableTotal { path, counter }, returned for the tree described under Fixes. A match over Error needs an arm for it or a wildcard.
  • Rust: content::CoverageReason. It gains TextOnly, the coverage of a Markdown file counted as plain text; like Analyzed it carries a value, and AnalyzerCoverage counts it as analyzed. A match over CoverageReason needs an arm for it or a wildcard.
  • Output. Under --analyze code, some C headers move from C++ to C in the language breakdown. A script that waits for a --watch repaint after a change sees none when nothing the view prints has changed. fdu PATH on a tree whose total overflows a u64 exits 1 rather than printing a wrapped total.

Known Limitations

  • Performance evidence is from one Linux host. Every figure above is from one quiet 4-vCPU virtualized host, and each peer standing is one run per tree. Each development record measured the engine it names, and the release adds three changes after them: a stat per listing, until one succeeds, that proves a directory searchable before its listing’s kinds are trusted (it keeps 89% of exp-197’s statx saving on the kernel source and 83% on node_modules), a checked running total per file, and a streaming code-line classifier. exp-202 measured the release engine, which contains all three, end to end against 0.2.1 and the peers on all three trees, and those are the figures above; it measured none of the three alone, and only the default tree and summary, so the streaming classifier, which runs only under --analyze code, is not in its figures.
  • macOS and Windows. No change in this release was measured there. The macOS comparison in the README is of a pre-0.2.0 build, and Windows has no performance claims.
  • Memory. Only a one-shot tree report with a share floor and an unfiltered selection folds its files. A report that needs the full index, such as --full, a filter, a files view, --ignored=exclude or only, --cache on, content analysis, --watch, or open, still holds every entry, so its peak memory grows with the tree. In 0.2.1 the ignore-aware summary held about 20 MiB more than a walk without .gitignore on the Linux v6.12 tree; that is still unexplained and was not re-measured.
  • Code analysis on a first run. On the Linux v6.12 source, with an earlier engine from this cycle (ebc06c78, before the streaming classifier), --analyze code with the cache off took 6.4 times as long as scc and 4.2 times as long as tokei, in 12 pairs on the same Linux host; run again under the default cache policy it answered in 0.55 s.
  • The 0.2.0 limitations on links, cache retention, watch backends, opened roots, and JSON integers still apply.

Security

Report vulnerabilities through GitHub’s private vulnerability reporting, as SECURITY.md describes.

Full changelog: CHANGELOG.md

Full commit history: changes from v0.2.1 to v0.3.0