Skip to content

fdu 0.1.0

Choose a tag to compare

@jlevy jlevy released this 26 Sep 05:10
· 958 commits to main since this release
Immutable release. Only release title and notes can be modified.
v0.1.0
7cf7f1b

fdu is a fast, incremental file roll-up engine. One walk over a directory tree answers, for every directory at once, how big it is, how many files it holds, what changed most recently, and what kinds of files it contains; the index is cached between runs and can be kept live as the tree changes. Reports form a cost ladder, from the aggregate-only fdu --no-gitignore --view summary PATH through the default fdu PATH tree and the path-only --view types to --analyze, the only option that reads file contents. This first release ships the same engine three ways: the fdu command line, the fdu Python package, and the fdu-core Rust crate.

On 2026-09-16, an exploratory macOS/APFS calibration of the release candidate built a reusable exact index and ten-row tree over a reproducible generated corpus of 1,000,001 entries in a 5.206-second median. The M1 Pro host ran under uncontrolled load, so this is not a portable absolute time or a cross-tool ranking; the full comparison preserves the complete evidence and its limits.

Highlights

  • Command line. fdu PATH prints a size-sorted tree, in allocated bytes unless --size apparent; --view picks among list (the default without --analyze), summary, tree, families, types, extensions, languages, documents, largest, recent, files, and full, and several views share one walk. Output is text, tree, paths, long, json, jsonl, or yaml, and exit status separates a complete result (0), a failure (1), and a partial result or usage error (2). At an interactive terminal, a run that takes longer than half a second shows one progress line on stderr, with its phase, the counts so far, and elapsed time, and erases it before any output; nothing is drawn into a pipe, a file, or CI, and --progress controls it. fdu --docs prints the usage guide, fdu --install-skill writes a portable skill for coding agents under the project root, and fdu --skill prints it.
  • Directory queries. Directories are selected like files, and each one’s size and age cover its whole eligible subtree, so fdu ~/projects --kind dir --include node_modules --modified-before 30d --long lists stale build directories with the bytes below each one and the age of its newest modification. --format paths prints only the paths, a flat list is complete and size-ranked, and --tree and --long are shorthands for --format tree and --format long. In JSON, JSON Lines, and YAML, a report with no --view or --analyze is this list view: one row per matching entry, with exact sizes, subtree counts, completeness, and a signed age against one report instant. --view tree gives the directory hierarchy.
  • Content analysis, only when asked: --analyze lines, code (standard lines of code for 15 languages), words (prose and reader-visible Markdown), or all, with results kept in a sidecar so unchanged files are not read again.
  • Python package. fdu.report, fdu.open, and fdu.scan return typed, immutable values, Report.as_dict() is the command line’s JSON, and Index.watch() is a closable change feed. Under the default auto cache policy, Index.refresh() and an iterated Index.watch() save what they verify, so a later cache-only open sees it. The wheel also installs the native fdu command.
  • Rust library. fdu-core is the engine, with open, a retained Index of pre-computed per-directory roll-ups, one-shot prepare_report, and an exact change feed; prepare_report_with_progress reports the phases the command line’s progress line draws. The fdu crate re-exports it with the command line.
  • Opened roots for interactive clients. OpenedIndex, in Rust and as fdu.opened in Python, answers while discovery is still running: coherent reads of several projections at one engine version, bounded pages with continuations, a resumable change journal, explicit refresh, discovery priorities, and a joined close.
  • .gitignore roll-ups, read by default on every surface: the command line, the one-shot library report, fdu.report, watch, and opened roots. Ignored and unignored totals for every directory come from every .gitignore inside the root, with git’s pattern semantics. Summary, tree, and extension rows end with the ignored part of their size, as (128 B ignored), and carry it as a tally in machine formats, where file rows carry a flag; --exclude-ignored and --only-ignored report one side, with sizes, ordering, and --min-size following the entries shown; and --no-gitignore, or read_controls=False, reads no rules and answers a typed “not observed” rather than guessing. Two limits bound the rules an index retains, --gitignore-budget SIZE|all (4 MiB by default) and --gitignore-line-limit SIZE|all (16 KiB): a .gitignore past either is refused rather than fatal, sizes stay exact, and a note names the directories skipped and the flag that lifts the limit that fired.
  • Cache and watch. A snapshot per root makes later runs cheaper where the cache supplies more than the filesystem will: --cache sets the policy, and --cache-status and --cache-clear manage snapshots without scanning. Status gives every file in the cache directory a state, and clearing reclaims snapshots another fdu version or format wrote, along with fdu’s own leftovers, while never removing a file that is not fdu’s. fdu --watch repeats the query as the tree changes, with every event verified by stat and every backend overflow reconciled. Cache-only content restore avoids repeated roll-up, candidate, classification, and path work, streams records into the retained index, and repeated unfiltered metric views share one entry traversal. Individual development comparisons on exploratory, uncontrolled, warm-steady M1 Pro/APFS runs measured roughly 4% to 13% lower wall time for cache-only restore changes and 10% lower peak RSS from streamed restore. A separate workload that included setup plus 100 four-view reports measured 19% lower wall time. These results are not additive and do not describe an ordinary one-shot command or Linux; the performance evidence preserves each workload and interval.

Install

uvx fdu@latest --help               # run the latest release without installing it
uv tool install fdu                 # the command line from a prebuilt wheel; uv tool upgrade fdu
cargo install --locked fdu          # the command line, built with Rust 1.85 or newer
pip install fdu                     # the Python package: import fdu
cargo add fdu                       # the Rust library, re-exported with the command line
cargo add fdu-core --features watch # the engine alone, with the watch layer

fdu --install-skill writes the agent skill to .agents/skills/fdu/SKILL.md and .claude/skills/fdu/SKILL.md under the project root, or to DIR/skills/fdu/SKILL.md with --agent-base DIR for one agent’s user scope; the skill prefers an fdu on PATH and otherwise runs uvx fdu@latest.

--locked builds against the lockfile published with the crate, which keeps the reviewed dependency set; see SUPPLY-CHAIN-SECURITY.md.

Platforms

Platform Command line and library Python wheel
macOS 11+, x86-64 and arm64 Supported; the scanner uses getattrlistbulk abi3 wheel
Linux glibc 2.17+, x86-64 and arm64 Supported abi3 wheel (manylinux2014); arm64 is cross-built and inspected, not run, before release
Windows x86-64 Supported, with no performance claims abi3 wheel
  • The Python package supports CPython 3.12 and newer through one stable-ABI (abi3) extension, tested on 3.12 and 3.14. Free-threaded CPython cannot load an abi3 extension, so it is not supported; use a GIL-enabled interpreter, as in uv tool install --python 3.14 fdu.
  • Other systems, including musl Linux and Windows arm64, build the source distribution with Rust 1.85 or newer.
  • Performance evidence comes mainly from an M1 Pro MacBook with a local APFS SSD; Linux measurements are from virtualized hosts, and the evidence report says what each figure is evidence about.

Compatibility

fdu is 0.x. A minor release (0.1 to 0.2) may change the Rust or Python API or the command line incompatibly, and its CHANGELOG names each change; a patch release never does. A machine consumer pins the schema string a report carries, which changes whenever a field does.

This release emits fdu.report/7 for every report, including the list rows above, fdu.stream/2 for a --watch stream, and fdu.cache/2 for cache status, which is its own document rather than a report and carries the identity of every tier each cached file holds. Treat an unrecognized value as a version you cannot parse rather than guessing at the fields.

Known Limitations

  • Memory. fdu builds an exact index that later questions reuse, and every report but an unfiltered --no-gitignore --view summary retains it, so peak memory grows with the entries retained. The 2026-09-16 tool comparison, on a generated 1,000,001-entry tree, measured fdu’s depth-one tree report, with its cache off, near 285 MiB of peak RSS, against 29 MiB for dumac, 21 MiB for dua, and 641 MiB for dust; each tool was invoked under a comparison contract that returned only a total. --no-gitignore --view summary retains no index: on that tree it took 4.876 s at 15.0 MiB, against 4.942 s at 285.7 MiB for the default --view summary, with identical totals. The default summary retains the index because classifying entries against .gitignore needs it. On a 328k-file checkout with many .gitignore files, runs of the same command peaked between 70 and 130 MiB, against about 13 MiB without the index: the cost follows the entries retained and varies run to run, so expect a range, not a fixed multiple.
  • Links. A symbolic link is listed as an entry but never followed, and adds nothing to a directory’s totals; no option follows links. A file with several hard links is counted once for each path, where du counts it once.
  • Cache retention. Nothing prunes snapshots of roots that are never scanned again. --cache-clear takes one root or the whole directory, so there is no way to clear only the stale snapshots an upgrade left behind; and since a newer snapshot format is stale to an older build just as an older one is, an older build’s --cache-clear=all removes a newer build’s snapshots. Changing either .gitignore limit invalidates a snapshot that read .gitignore, and a snapshot written with --no-gitignore cannot serve a run that reads the rules, so the next run that loads a snapshot scans cold.
  • Content analysis is one-shot on every surface: --watch refuses --analyze, and a Rust Session or Python Index.watch() refuses an index opened with analysis. Markdown analysis retains the current Markdown source and parser state per active worker. Code analysis retains the current logical line and its allocation, so a one-line minified or generated source file can add file-sized memory per worker.
  • .gitignore handling reads only .gitignore files inside the scanned root: not .git/info/exclude, core.excludesFile, or files above the root. A .gitignore inside a directory that is already ignored is still read and charged against --gitignore-budget, which git never does; classification stays right, but a tree that keeps most of its rule files under ignored directories can reach the 4 MiB default and see refusals for rules that could not have changed a verdict.
  • Ignored shares appear in summary, tree, extension, and file rows, not in types, families, languages, or documents rows, though the ignore filters do apply there.
  • Watch. A watch uses the platform’s native event backend on every filesystem, with no polling fallback for network filesystems that do not deliver events. Under --exclude-ignored or --only-ignored a stream keeps its entry set exact across rule edits; under the default selection it keeps membership live but does not restate a row’s ignored bit after a rule edit, so re-read a listing when the bit itself matters.
  • Opened roots. A Tree page can exceed its max_work by the width of one directory level. The opened-root types (OpenedIndex projections, TreePage, ReadResponse, and their fdu.opened mirrors) are expected to change in 0.2.
  • JSON integers. Fingerprints such as type_rules_fingerprint and nanosecond timestamps exceed 2^53, so JavaScript’s JSON.parse loses their precision; use a parser that preserves big integers.

Upgrading from a Pre-Release Build

Only for anyone who ran fdu built from a development checkout:

  • A cached tree can scan cold once. A snapshot serves only the engine and scope that wrote it. Development builds already carried version 0.1.0, but the engine fingerprint also mixes in the snapshot format, now version 5, and a version for the per-entry facts revalidation checks, which was raised on every platform when Windows validation gained change time and file identity. A snapshot from a development build without that change (anything built from main before pull request #117 merged on 2026-09-23) is not served, nor is one written under other type rules or .gitignore limits, or without .gitignore for a run that reads it; that tree’s first run scans cold and replaces it. The content sidecar, format 7, carries the same fingerprint and serves only the analyzer set that wrote it, so the first analyzed run after upgrading, or with another --analyze set, reads every file again. The crate version moves at every release, so each later upgrade costs one cold run per cached tree.
  • fdu --cache-clear=all reclaims the snapshots that invalidation strands. Clearing now removes snapshots written by another fdu version or format as well as current ones, so nothing has to be deleted by hand. fdu --cache-status=all lists each one as stale with the reason before you clear, and files that are not fdu’s are never removed.
  • The gitignore build feature is gone; remove features = ["gitignore"].
  • Flags, report schema versions, and Rust and Python interfaces were renamed before this release; the CHANGELOG gives each under its released name. A script that read the directory tree from a machine format without --view adds --view tree, because the default there is the list view.

Security

Report vulnerabilities through GitHub’s private vulnerability reporting, as SECURITY.md describes.

Full changelog: CHANGELOG.md

Full commit history: https://github.com/jlevy/fdu/commits/v0.1.0