fs-inventory 1.0.0

Compare two scans with the fs_diff DuckDB macros, much faster scans of very deep trees, and directories deleted while they're read counted as vanished instead of as errors. New tests check the output against GNU find, scans of a tree that changes throughout, random names through every text encoder, and trees 2,000 levels deep or with a million entries in one directory.

From 1.0 on, the command line, the output tables and their columns, the manifest, and the DuckDB macros follow semantic versioning: 1.x releases can add options, columns (at the end of a table, with a new schema_version), and macros, but won't remove or rename them.

Added

• fs_diff and fs_diff_summary DuckDB macros compare two Parquet scans of the same roots: one row per added, removed, or changed entry, with the columns that changed and the rows before and after, or the counts and net size change by kind. Entries are matched by their exact path bytes; atime and dev are ignored unless ignore := [] is given. They work on version 2 scans too.

Changed

• convert takes its output format with -f/--format, like scan. --to, its name in 0.6.0, still works.

Fixed

• Directories whose paths are longer than PATH_MAX (4,096 bytes on Linux, 1,024 on macOS) are opened in a few calls instead of one call per path component. Opening every directory of a 2,000-level tree took millions of calls: scanning one on macOS went from 7.5 minutes to 6 seconds.
• A directory deleted while it was being read counts as vanished, like other deletions during a scan, instead of as a readdir error (Linux reports it as ENOENT). Busy file systems no longer end a scan with exit code 1 for routine churn.

Full changelog: v0.6.0...v1.0.0

fs-inventory 0.6.0

Re-export Parquet scans as CSV or JSON Lines with convert, record symlink targets, and look up user and group names.

Added

• fs-inventory convert re-exports a Parquet scan as CSV or JSON Lines, into a new scan directory or to stdout, with optional zstd or gzip compression. The output equals what the scan would have written in that format, using the scan's own column, encoding, and time settings. Older (schema version 2) scans convert too.
• --resolve-ids writes users and groups tables with the name of every user and group ID in a scan (null for IDs with no account, such as deleted users). Crawl threads collect the IDs as they go, and each is looked up once when the scan finishes: with getent on Linux, which reaches LDAP, sssd, and NIS even from the static binary, and dscacheutil on macOS. Checkpoints keep the IDs, so resumed scans get complete tables. New DuckDB macros: fs_users, fs_groups, and fs_entries_named, which adds user_name and group_name to every entry.
• --symlink-targets records where each symlink points, in a new symlink_target column, with the exact bytes in symlink_target_raw (symlink_target_b64 in text) when the target isn't plain UTF-8. It costs one more call per symlink, and a failed read is recorded in the errors table as readlink.

Changed

• Schema version 3: the entries table gains symlink_target and symlink_target_raw, at the end, so the CSV and JSON Lines macros read version 2 scans too. verify, rollup, and convert accept version 2 scans; resume continues only scans of the current version.

Full changelog: v0.5.0...v0.6.0

fs-inventory 0.5.0

Release binaries and an installer, a documentation website, scale tests up to a billion entries, and a --max-memory that budgets the whole scan.

Added

• Release binaries: each release has static Linux binaries (x86_64 and ARM) and macOS binaries (Apple silicon and Intel), with a SHA256SUMS file. Pushing a version tag builds them and drafts the release.
• A documentation website, https://robert-mcdermott.github.io/fs-inventory/, generated from the repository's Markdown: the user guide, schema, architecture, and changelog, with navigation, search across every page, copy buttons, and dark mode. Publishing a release rebuilds it from that release's docs, and CI fails a pull request that breaks a link between pages or sections.
• install.sh installs a release with curl ... | sh: it verifies the SHA-256 checksum and the binary's version before installing to ~/.local/bin (or --install-dir), and never uses sudo or changes PATH.
• Generated trees for scale tests: the hidden option scan --synthetic 100M scans a deterministic tree computed in memory instead of the file system, with exactly the requested number of entries and the variety of a real tree, including names that aren't valid UTF-8. A billion-entry tree costs no disk space, and resume continues it. The manifest's options.synthetic marks such scans.
• bench/scale.py scans a generated tree of any size, records memory and throughput every second, can kill and resume the scan, and checks that every entry was written exactly once.
• bench/server.py benchmarks a server's storage in one run of about 15 minutes, and writes a report to send back. It covers the CPU ceiling, find, probe, and time-boxed scans at several thread counts. On NFS, it also records RPCs per entry. An optional kill-and-resume test runs on a smaller tree.

Changed

• --max-memory budgets the whole scan. It first sets aside what the crawl threads and Parquet writers need (memory grows with threads, measured at about 1.7 GB with 64 threads and 3 GB with 256 without a budget), uses only as many writers as fit unless --writers is given, and shrinks row buffers as threads are added. It also has the allocator return freed memory at once, which cut peak memory by 40 to 50% on Linux for 5 to 8% more CPU time. A budget too small for the threads gets a warning with the budget they need. Scans without --max-memory run exactly as before.

Removed

• --guess-encoding, which was never implemented. Guesses of a name's legacy character set are unreliable on short names, and the exact bytes of every name are kept, so they can be decoded later with the right character set.

Fixed

• The benchmark scripts run on Python 3.6 and later, as RHEL 8's system Python is; run.py needed 3.9.

Full changelog: v0.4.0...v0.5.0

fs-inventory 0.4.0

Milestone M4: resumable scans.

Added

• Checkpoints and fs-inventory resume. Every --checkpoint-interval (10 minutes by default; 0 disables them), a scan pauses as each crawl thread finishes its current directory, commits every open part file, and records the directories still to read in checkpoint/ inside the scan directory. fs-inventory resume SCAN_DIR continues a scan that was interrupted, killed, or on a machine that went down. It deletes the parts written after the last checkpoint and reads their directories again, so the finished scan matches an uninterrupted one, with every row once. After Ctrl-C, a final checkpoint makes resuming exact. Resume can change threads, writers, memory, the stall timeout, the checkpoint interval, and progress; the manifest records each resume under resumes. A lock keeps two processes from writing one scan.

Changed

• With checkpoints, the spill file and the --detect-cycles record of visited directories live in checkpoint/ until the scan finishes, instead of being unlinked at once.
• Each checkpoint commits the open part files, so long scans have more, smaller parts. Parts written after a resume are named part-rN-wNN-NNNNNN.

Resuming is described in the user guide, and how it works in the architecture overview.

Full changelog: v0.3.0...v0.4.0

fs-inventory 0.3.0

First tagged release. See CHANGELOG.md.