Repository navigation
fdu 0.1.0
fdu is a fast, incremental file roll-up engine. One walk over a directory tree answers, for every directory at once, how big it is, how many files it holds, what changed most recently, and what kinds of files it contains; the index is cached between runs and can be kept live as the tree changes. Reports form a cost ladder, from the aggregate-only fdu --no-gitignore --view summary PATH through the default fdu PATH tree and the path-only --view types to --analyze, the only option that reads file contents. This first release ships the same engine three ways: the fdu command line, the fdu Python package, and the fdu-core Rust crate.
On 2026-09-16, an exploratory macOS/APFS calibration of the release candidate built a reusable exact index and ten-row tree over a reproducible generated corpus of 1,000,001 entries in a 5.206-second median. The M1 Pro host ran under uncontrolled load, so this is not a portable absolute time or a cross-tool ranking; the full comparison preserves the complete evidence and its limits.
Highlights
- Command line.
fdu PATHprints a size-sorted tree, in allocated bytes unless--size apparent;--viewpicks amonglist(the default without--analyze),summary,tree,families,types,extensions,languages,documents,largest,recent,files, andfull, and several views share one walk. Output istext,tree,paths,long,json,jsonl, oryaml, and exit status separates a complete result (0), a failure (1), and a partial result or usage error (2). At an interactive terminal, a run that takes longer than half a second shows one progress line on stderr, with its phase, the counts so far, and elapsed time, and erases it before any output; nothing is drawn into a pipe, a file, or CI, and--progresscontrols it.fdu --docsprints the usage guide,fdu --install-skillwrites a portable skill for coding agents under the project root, andfdu --skillprints it. - Directory queries. Directories are selected like files, and each one’s size and age cover its whole eligible subtree, so
fdu ~/projects --kind dir --include node_modules --modified-before 30d --longlists stale build directories with the bytes below each one and the age of its newest modification.--format pathsprints only the paths, a flatlistis complete and size-ranked, and--treeand--longare shorthands for--format treeand--format long. In JSON, JSON Lines, and YAML, a report with no--viewor--analyzeis thislistview: one row per matching entry, with exact sizes, subtree counts, completeness, and a signed age against one report instant.--view treegives the directory hierarchy. - Content analysis, only when asked:
--analyze lines,code(standard lines of code for 15 languages),words(prose and reader-visible Markdown), orall, with results kept in a sidecar so unchanged files are not read again. - Python package.
fdu.report,fdu.open, andfdu.scanreturn typed, immutable values,Report.as_dict()is the command line’s JSON, andIndex.watch()is a closable change feed. Under the defaultautocache policy,Index.refresh()and an iteratedIndex.watch()save what they verify, so a later cache-only open sees it. The wheel also installs the nativefducommand. - Rust library.
fdu-coreis the engine, withopen, a retainedIndexof pre-computed per-directory roll-ups, one-shotprepare_report, and an exact change feed;prepare_report_with_progressreports the phases the command line’s progress line draws. Thefducrate re-exports it with the command line. - Opened roots for interactive clients.
OpenedIndex, in Rust and asfdu.openedin Python, answers while discovery is still running: coherent reads of several projections at one engine version, bounded pages with continuations, a resumable change journal, explicit refresh, discovery priorities, and a joined close. .gitignoreroll-ups, read by default on every surface: the command line, the one-shot library report,fdu.report, watch, and opened roots. Ignored and unignored totals for every directory come from every.gitignoreinside the root, with git’s pattern semantics. Summary, tree, and extension rows end with the ignored part of their size, as(128 B ignored), and carry it as a tally in machine formats, where file rows carry a flag;--exclude-ignoredand--only-ignoredreport one side, with sizes, ordering, and--min-sizefollowing the entries shown; and--no-gitignore, orread_controls=False, reads no rules and answers a typed “not observed” rather than guessing. Two limits bound the rules an index retains,--gitignore-budget SIZE|all(4 MiB by default) and--gitignore-line-limit SIZE|all(16 KiB): a.gitignorepast either is refused rather than fatal, sizes stay exact, and a note names the directories skipped and the flag that lifts the limit that fired.- Cache and watch. A snapshot per root makes later runs cheaper where the cache supplies more than the filesystem will:
--cachesets the policy, and--cache-statusand--cache-clearmanage snapshots without scanning. Status gives every file in the cache directory a state, and clearing reclaims snapshots another fdu version or format wrote, along with fdu’s own leftovers, while never removing a file that is not fdu’s.fdu --watchrepeats the query as the tree changes, with every event verified by stat and every backend overflow reconciled. Cache-only content restore avoids repeated roll-up, candidate, classification, and path work, streams records into the retained index, and repeated unfiltered metric views share one entry traversal. Individual development comparisons on exploratory, uncontrolled, warm-steady M1 Pro/APFS runs measured roughly 4% to 13% lower wall time for cache-only restore changes and 10% lower peak RSS from streamed restore. A separate workload that included setup plus 100 four-view reports measured 19% lower wall time. These results are not additive and do not describe an ordinary one-shot command or Linux; the performance evidence preserves each workload and interval.
Install
uvx fdu@latest --help # run the latest release without installing it
uv tool install fdu # the command line from a prebuilt wheel; uv tool upgrade fdu
cargo install --locked fdu # the command line, built with Rust 1.85 or newer
pip install fdu # the Python package: import fdu
cargo add fdu # the Rust library, re-exported with the command line
cargo add fdu-core --features watch # the engine alone, with the watch layerfdu --install-skill writes the agent skill to .agents/skills/fdu/SKILL.md and .claude/skills/fdu/SKILL.md under the project root, or to DIR/skills/fdu/SKILL.md with --agent-base DIR for one agent’s user scope; the skill prefers an fdu on PATH and otherwise runs uvx fdu@latest.
--locked builds against the lockfile published with the crate, which keeps the reviewed dependency set; see SUPPLY-CHAIN-SECURITY.md.
Platforms
| Platform | Command line and library | Python wheel |
|---|---|---|
| macOS 11+, x86-64 and arm64 | Supported; the scanner uses getattrlistbulk |
abi3 wheel |
| Linux glibc 2.17+, x86-64 and arm64 | Supported | abi3 wheel (manylinux2014); arm64 is cross-built and inspected, not run, before release |
| Windows x86-64 | Supported, with no performance claims | abi3 wheel |
- The Python package supports CPython 3.12 and newer through one stable-ABI (
abi3) extension, tested on 3.12 and 3.14. Free-threaded CPython cannot load anabi3extension, so it is not supported; use a GIL-enabled interpreter, as inuv tool install --python 3.14 fdu. - Other systems, including musl Linux and Windows arm64, build the source distribution with Rust 1.85 or newer.
- Performance evidence comes mainly from an M1 Pro MacBook with a local APFS SSD; Linux measurements are from virtualized hosts, and the evidence report says what each figure is evidence about.
Compatibility
fdu is 0.x. A minor release (0.1 to 0.2) may change the Rust or Python API or the command line incompatibly, and its CHANGELOG names each change; a patch release never does. A machine consumer pins the schema string a report carries, which changes whenever a field does.
This release emits fdu.report/7 for every report, including the list rows above, fdu.stream/2 for a --watch stream, and fdu.cache/2 for cache status, which is its own document rather than a report and carries the identity of every tier each cached file holds. Treat an unrecognized value as a version you cannot parse rather than guessing at the fields.
Known Limitations
- Memory. fdu builds an exact index that later questions reuse, and every report but an unfiltered
--no-gitignore --view summaryretains it, so peak memory grows with the entries retained. The 2026-09-16 tool comparison, on a generated 1,000,001-entry tree, measured fdu’s depth-one tree report, with its cache off, near 285 MiB of peak RSS, against 29 MiB for dumac, 21 MiB for dua, and 641 MiB for dust; each tool was invoked under a comparison contract that returned only a total.--no-gitignore --view summaryretains no index: on that tree it took 4.876 s at 15.0 MiB, against 4.942 s at 285.7 MiB for the default--view summary, with identical totals. The default summary retains the index because classifying entries against.gitignoreneeds it. On a 328k-file checkout with many.gitignorefiles, runs of the same command peaked between 70 and 130 MiB, against about 13 MiB without the index: the cost follows the entries retained and varies run to run, so expect a range, not a fixed multiple. - Links. A symbolic link is listed as an entry but never followed, and adds nothing to a directory’s totals; no option follows links. A file with several hard links is counted once for each path, where
ducounts it once. - Cache retention. Nothing prunes snapshots of roots that are never scanned again.
--cache-cleartakes one root or the whole directory, so there is no way to clear only the stale snapshots an upgrade left behind; and since a newer snapshot format is stale to an older build just as an older one is, an older build’s--cache-clear=allremoves a newer build’s snapshots. Changing either.gitignorelimit invalidates a snapshot that read.gitignore, and a snapshot written with--no-gitignorecannot serve a run that reads the rules, so the next run that loads a snapshot scans cold. - Content analysis is one-shot on every surface:
--watchrefuses--analyze, and a RustSessionor PythonIndex.watch()refuses an index opened with analysis. Markdown analysis retains the current Markdown source and parser state per active worker. Code analysis retains the current logical line and its allocation, so a one-line minified or generated source file can add file-sized memory per worker. .gitignorehandling reads only.gitignorefiles inside the scanned root: not.git/info/exclude,core.excludesFile, or files above the root. A.gitignoreinside a directory that is already ignored is still read and charged against--gitignore-budget, which git never does; classification stays right, but a tree that keeps most of its rule files under ignored directories can reach the 4 MiB default and see refusals for rules that could not have changed a verdict.- Ignored shares appear in summary, tree, extension, and file rows, not in
types,families,languages, ordocumentsrows, though the ignore filters do apply there. - Watch. A watch uses the platform’s native event backend on every filesystem, with no polling fallback for network filesystems that do not deliver events. Under
--exclude-ignoredor--only-ignoreda stream keeps its entry set exact across rule edits; under the default selection it keeps membership live but does not restate a row’signoredbit after a rule edit, so re-read a listing when the bit itself matters. - Opened roots. A
Treepage can exceed itsmax_workby the width of one directory level. The opened-root types (OpenedIndexprojections,TreePage,ReadResponse, and theirfdu.openedmirrors) are expected to change in 0.2. - JSON integers. Fingerprints such as
type_rules_fingerprintand nanosecond timestamps exceed 2^53, so JavaScript’sJSON.parseloses their precision; use a parser that preserves big integers.
Upgrading from a Pre-Release Build
Only for anyone who ran fdu built from a development checkout:
- A cached tree can scan cold once. A snapshot serves only the engine and scope that wrote it. Development builds already carried version
0.1.0, but the engine fingerprint also mixes in the snapshot format, now version 5, and a version for the per-entry facts revalidation checks, which was raised on every platform when Windows validation gained change time and file identity. A snapshot from a development build without that change (anything built frommainbefore pull request #117 merged on 2026-09-23) is not served, nor is one written under other type rules or.gitignorelimits, or without.gitignorefor a run that reads it; that tree’s first run scans cold and replaces it. The content sidecar, format 7, carries the same fingerprint and serves only the analyzer set that wrote it, so the first analyzed run after upgrading, or with another--analyzeset, reads every file again. The crate version moves at every release, so each later upgrade costs one cold run per cached tree. fdu --cache-clear=allreclaims the snapshots that invalidation strands. Clearing now removes snapshots written by another fdu version or format as well as current ones, so nothing has to be deleted by hand.fdu --cache-status=alllists each one asstalewith the reason before you clear, and files that are not fdu’s are never removed.- The
gitignorebuild feature is gone; removefeatures = ["gitignore"]. - Flags, report schema versions, and Rust and Python interfaces were renamed before this release; the CHANGELOG gives each under its released name. A script that read the directory tree from a machine format without
--viewadds--view tree, because the default there is thelistview.
Security
Report vulnerabilities through GitHub’s private vulnerability reporting, as SECURITY.md describes.
Full changelog: CHANGELOG.md
Full commit history: https://github.com/jlevy/fdu/commits/v0.1.0