Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,9 @@ follow-up:
one-line synopsis; `registry_test` enforces presence + shape);
- a new / changed **global flag** -> its `cli::GlobalFlag` entry in `globals.cc`
(`globals_test` enforces it; set `alias` / `display` for short or alternate forms);
- a new / changed **cookbook recipe** (`cli/help.cc` `RenderCookbook`) -> a matching
execution case in `//xff/examples:cookbook_test`; its guard case fails CI if a recipe ships
without one, so the user-facing examples are run, not just rendered;
- the hand-maintained `kHelpText` usage page in `cli/main.cc`;
- any prose docs the change affects (`docs/design-*.md`, `TODO.md`).

Expand Down
174 changes: 95 additions & 79 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,44 +3,80 @@

# xff - eXtended File Find

`xff` is a `find`(1)-compatible file finder with modern extensions. It walks each
starting path and acts on the entries matching an expression, exactly like `find`,
then adds the conveniences you always wished `find` had: content and language
search, structured output, per-run summaries and histograms, safe deletes, native
hashing, and a shared `{field}` vocabulary that threads through `-printf`, `-exec`,
and every renderer.
`xff` is a `find(1)`-compatible file finder with modern extensions. It walks each starting path and acts on the entries matching an expression, exactly like `find`, then adds the conveniences you always wished `find` had: content and language search, structured output, per-run summaries and histograms, safe deletes, native hashing, and a shared `{field}` vocabulary that threads through `-printf`, `-exec`, and every renderer.

Anything `find` does, `xff` does the same way. Everything else is opt-in.

> This README is a short overview. The complete, always-current reference lives in
> [`XFF.md`](XFF.md) (generated from the binary; see [Documentation](#documentation)).

## Highlights

- **`find`-compatible core.** The standard primaries (`-name`, `-type`, `-size`,
`-mtime`, `-regex`, `-exec`, `-prune`, ...), operators, and exit codes behave as
in GNU/BSD `find`. Invoked as `find`, it is strict `find` and nothing more.
- **Content and metadata matching.** `-grep` / `-content` search inside files,
`-lang 'C*'` and `-mime 'image/*'` match by inferred language or media type,
`-text` / `-binary` / `-eofnl` classify content, and `-hash=sha256` emits a
checksum manifest.
- **Structured output.** Stream matches as plain text, NUL-delimited, JSONL, CSV /
TSV, an aligned table, a tree, or a Markdown table, all from one walk.
- **Summaries and histograms.** `--summary=ext` folds matches into counts and
totals; `--histogram='ext:sum(lines)'` draws bars. No external `awk | sort`.
- **One `{field}` vocabulary.** The same named fields (`{relpath}`, `{size}`,
`{lang}`, `{hash}`, `{capture.NAME}`, ...) drive `-printf`, `-exec`, `--format`,
and `--summary`, with `s///` rewrite and `m//` extract qualifiers.
- **Safe by default.** `-delete` implies `-depth` and honors `--dry-run` / `--safe`;
config loaded from an `--xffrc` file cannot run `-exec` or `-delete` unless armed.
- **Fast and parallel.** `-j N` parallelizes the walk and concurrent `-exec`;
`--sort` gives deterministic ordering when you want it.

## Quick start

`xff` builds with [Bazel](https://bazel.build/) and runs on macOS and Linux.

```sh
This `README.md` is a short overview. The complete, always-current reference lives in [XFF.md](./XFF.md) (generated from the binary; see Documentation).

---

## Core Highlights & Architectural Advantages

- **`find`-Compatible Core:** The standard primaries (`-name`, `-type`, `-size`, `-mtime`, `-regex`, `-exec`, `-prune`, ...), operators, and exit codes behave exactly as in GNU/BSD `find`. Invoked as `find`, it is strict `find` and nothing more.
- **Content & Metadata Matching:** `-grep` / `-content` search inside files, `-lang 'C*'` and `-mime 'image/*'` match by inferred language or media type, `-text` / `-binary` / `-eofnl` classify content, and native `-hash` primitives emit optimized checksum manifests.
- **Structured Layout Engines:** Stream matches natively as plain text, NUL-delimited, `JSONL`, `CSV` / `TSV`, an aligned console table, a visual tree, or a standard Markdown table, all calculated from one single filesystem walk.
- **Summaries & Histograms:** `--summary=ext` folds matches into counts and totals; `--histogram='ext:sum(lines)'` draws terminal bar charts using Unicode block characters. No external `awk | sort` pipeline overhead required.
- **Unified `{field}` Vocabulary:** The same named fields (`{relpath}`, `{size}`, `{lang}`, `{hash}`, `{capture.NAME}`, ...) drive `-printf`, `-exec`, `--format`, and `--summary`, complete with powerful `s///` regex rewrite and `m//` extraction qualifiers.
- **Safe by Default:** `-delete` implicitly forces `-depth` and strictly honors `--dry-run` / `--safe`. Configuration tiers loaded via an `--xffrc` file are sandboxed: they cannot execute dangerous directives (`-exec`, `-execdir`, `-ok`, `-capture`, or `-delete`) unless explicitly armed via a trusted CLI flag (`--allow-exec`).
- **Fast and Parallel:** `-j N` parallelizes the filesystem walk and concurrent `-exec` pools across available CPU worker cores; `--sort` delivers deterministic sibling ordering when requested.
- **Virtual Archive Inspection:** Native capability to descend into compressed archives (`--archive`) like `.tar`, `.zip`, `.zstd`, and `.lz4` as virtual paths, executing content matching transparently on uncompressed streams without extraction disk space overhead.

---

## Tool Feature Comparison Matrix

The matrix below maps the primary feature landscapes across legacy and modern single-purpose utilities to show what `xff` consolidates into a single execution pass.

| Feature / Capability | `find` | `fd` | `grep` | `ripgrep` (`rg`) | `shasum` | `xff` |
| :----------------------------------- | :----: | :--: | :----: | :--------------: | :------: | :--------------------------------------- |
| **Multi-threaded Traversal** | - | ✓ | - | ✓ | - | **✓ Native Pool (`-j`)** |
| **Implicit VCS/Gitignore Awareness** | - | ✓ | - | ✓ | - | **✓ Configurable (`-g`)** |
| **Regex Content Search Engine** | - | - | ✓ | ✓ | - | **✓ Native (`-rxc`/`-grep`)** |
| **Inline Cryptographic Hashes** | - | - | - | - | ✓ | **✓ Native (`-hash`)** |
| **Virtual Archive Traversal** | - | - | - | - | - | **✓ Native (`--archive`)** |
| **Custom Layout Templating** | GNU | - | - | - | - | **✓ Rich (`--template`)** |
| **Fine-Grained Line-Ending Linting** | - | - | - | - | - | **✓ Native (`-eofnl`/`-eofcrlf`)** |
| **Deterministic Reduction Tables** | - | - | - | - | - | **✓ Native (`--summary`/`--histogram`)** |

`find`'s "Custom Layout Templating" is marked **GNU** because it is GNU find's `-printf`, a
GNU extension; POSIX and BSD/macOS `find` have no format primary (only `-print` / `-exec`).

---

## Flavor & Behavioral Shift Matrix

`xff` runs one unified grammar under three operational flavors. The flavor is selected automatically by the program binary name and can be explicitly overridden or layered using the `--config` flag (where the last specified style wins).

The table below illustrates how traditional shell workflows shift into optimized `xff` unified expressions.

| Target Intent / Use Case | Legacy Command / Pipeline | The `xff` Unified Expression | Architectural Advantage / Behavioral Shift |
| :-------------------------------- | :----------------------------------------- | :-------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------ |
| **Strict Compliance** | `find . -type f -name "*.cpp"` | `find . -type f -name "*.cpp"` <br>_or_ `xff --config=find ...` | **Strict POSIX compatibility mode:** Turns off all modern extensions; modern flags become immediate usage errors.\* |
| **Modern Structural Search** | `fd -e cc` | `xff -regex '.*\.cc$'` <br>_or_ `xff --config=xff ...` | **Evolved mode (Default):** Expands `find`'s grammar with modern extensions, enabling sorted output and human sizes (`--human=si`). |
| **Clean Developer Grep** | `fd -H -E ".git" \| xargs rg "TODO"` | `xff --config=rg -grep "TODO"` | **Opinionated Developer Mode:** Implicitly respects nested `.gitignore` files, skips hidden files, and uses smart-case matching logic. |
| **High-Performance Verification** | `find . -type f -exec sha256sum {} \;` | `xff -type f -hash=sha256` | **Zero-Fork Speed:** Eliminates system process-spawning overhead. Reads files directly into the native read loop buffer to hash inline. |
| **Missing Newline Code Linting** | _Complex multi-line awk scripts or loops._ | `xff -text ! -eofnl -print` | **Native Classification:** Instantly flags text files violating POSIX trailing newline rules without streaming lines to the shell. |
| **Cross-OS Time Constraints** | `find . -mmin -60` | `xff -mtime "-3 weeks 3 hours"` | **Advanced Parsing:** Uses human-readable compound duration strings interpreted cleanly via explicit IANA `--timezone` modifiers. |
| **Compressed Asset Auditing** | `tar -ztf src.tar.gz \| grep "cfg"` | `xff --archive -path "*src.tar.gz*cfg*"` | **Virtual File-tree Mapping:** Treats archives as virtual read-only directories, matching inner structures without manual disk extraction. |
| **Isolated Variable Outputting** | `find . -printf "%p,%s\n"` | `xff --format=csv --columns=path,size` | **Structured Sanitization:** Formats data cleanly into formal arrays with safe, native C-escape column handling (`--path-encoding=escape`). |

\* **Strict `find` mode is still xff's engine, not a wrapper around the OS `find`.** It keeps
find's vocabulary and turns the xff extensions into usage errors, but the implementation is one
fast, mostly platform-independent binary. The clearest divergence is regex: `-regex` / `-iregex`
default to **RE2** (linear-time, no catastrophic backtracking) and behave identically on Linux
and macOS - where GNU find instead defaults to its Emacs dialect and BSD/macOS find to BRE.
`-regextype` selects xff's uniform grammar set (RE2, EXACT, FNMATCH, GLOB, SHGLOB, plus PCRE2 in
a full build), never GNU's dialect names. Otherwise strict mode is find's documented behavior,
made uniform across platforms.

---

## Quick Start

`xff` builds with Bazel and runs on macOS and Linux.

```bash
# Build and run the stock binary.
bazel run //xff -- . -type f -name '*.md'

Expand All @@ -49,9 +85,11 @@ bazel build //xff
cp bazel-bin/xff/cli/xff /usr/local/bin/xff
```

---

## Examples

```sh
```bash
# Ten largest files (-printf builds any columnar line; the shell sorts).
xff . -type f -printf '%s\t%p\n' | sort -rn | head

Expand All @@ -71,65 +109,43 @@ xff . -type f -hash=sha256
xff . -type f -mtime -1 --format=jsonl
```

See the [`XFF.md` cookbook](XFF.md#examples) for more worked examples, including
native per-author `git blame` line counts with no shell pipe.
See the [XFF.md](./XFF.md) cookbook for more worked examples, including native per-author `git blame` line counts computed with no shell pipes.

## Flavors

`xff` runs one grammar under three flavors. The flavor is selected by the program
name and can be overridden with `--config`:

| Invoked as | Flavor | Behavior |
| ------------- | ------------------ | ----------------------------------------------------------------- |
| `find` | strict `find` | only the standard vocabulary; the xff extensions are usage errors |
| `xff` | evolved (default) | find's grammar plus every extension, sorted output, human sizes |
| `--config=rg` | opinionated (`rg`) | respects `.gitignore`, skips hidden files, smart-case matching |

An explicit `--config=find|xff|rg` always wins over the program name.
---

## Documentation

The vocabulary and options are defined once in the binary (the registry is the
single source of truth), so every documentation surface is generated from it and
cannot drift:
The vocabulary and options are defined once inside the C++ binary (the engine registry acts as the single source of truth), ensuring that every documentation surface is automatically generated and cannot drift:

- **[`XFF.md`](XFF.md)** - the full reference, in Markdown. It is verbatim
`xff --markdown` output, regenerated by [`xff-md-update.sh`](xff-md-update.sh) and
guarded by `//xff/cli:xff_markdown_test`, which fails CI on any drift.
- `xff --help` for the usage page, `xff --help=TOPIC` for a topic (`fields`,
`printf`, `time`, `size`, `grammars`, `stats`, ...), and `xff --help=full` for
everything.
- `xff --man` renders the roff man page; `xff --markdown` prints the reference above.
- **`XFF.md`**: The full comprehensive reference in Markdown. It is a verbatim dump of `xff --markdown`, regenerated by `xff-md-update.sh` and guarded by the `//xff/cli:xff_markdown_test` target, which fails CI if any code-to-docs drift occurs.
- **`xff --help`**: Renders the main utility usage page. Use `xff --help=TOPIC` to review specific sub-topics (`fields`, `printf`, `time`, `size`, `grammars`, `stats`, etc.), or `xff --help=full` to dump all help sections.
- **`xff --man`**: Renders the standard `roff` man page stream.

## Building
---

The default build is a lean core. Heavier capabilities (the PCRE2 regex grammar,
archive diving) are composable build-time extras, off by default, so the default
build stays small and dependency-light. The extended binary links them all:
## Building & Dependencies

```sh
# The full binary, with every extra (PCRE2, ...).
The default build provides a lean, dependency-light core. Heavier processing capabilities (such as the advanced `PCRE2` regex grammar or recursive archive diving) are decoupled as composable build-time extras that are disabled by default to keep the core binary small. The extended target links them all:

```bash
# The full binary, with every extra (PCRE2, archive diving, etc.).
bazel build --config=xff_full //xff/cli:xff_full
```

The `//xff` alias follows the active config: it is the lean binary by default and
the full binary under `--config=xff_full`, so `bazel run //xff` gives you whichever
you configured. The two underlying targets stay explicit and config-stable:
`//xff/cli:xff` is always lean, `//xff/cli:xff_full` always full.
The `//xff` target alias follows your active workspace configuration automatically: it resolves to the lean binary by default, and switches to the full binary under `--config=xff_full`. The underlying targets remain explicit and configuration-stable: `//xff/cli:xff` is always lean, and `//xff/cli:xff_full` is always full.

> **Compile-Time Enforcement:** The CLI options for extras (e.g., `--regextype=PCRE2` or `--archive`) are always exposed on the interface. Attempting to invoke an extra feature in a lean build that did not compile it will yield an immediate, explicit error rather than a silent failure or fallback.

The CLI surface for an extra (`--regextype=PCRE2`, `--archive`) is always present;
using one in a build that did not link it is an immediate, explicit error, never a
silent no-op.
- **Requirements:** Bazel accompanied by a modern C++23 toolchain (`clang-22` or newer). A fully hermetic LLVM toolchain is available out-of-the-box via `--config=clang`.

Requirements: Bazel with a C++23 toolchain (clang-22 or newer; a hermetic LLVM is
available via `--config=clang`).
---

## Contributing

See [`CONTRIBUTING.md`](CONTRIBUTING.md), the agent / contributor rules in
[`AGENTS.md`](AGENTS.md), and the C++ style in [`STYLE_CPP.md`](STYLE_CPP.md).
Design notes live under [`docs/`](docs/) and the roadmap in [`TODO.md`](TODO.md).
See [CONTRIBUTING.md](./CONTRIBUTING.md), the LLM agent and contributor guidelines in [AGENTS.md](./AGENTS.md), and the core systems architecture style in [STYLE_CPP.md](./STYLE_CPP.md). In-depth design notes live under `docs/` and the active development roadmap can be found in `TODO.md`.

---

## License

Apache License 2.0. See [`LICENSE`](LICENSE) and [`NOTICE`](NOTICE).
Apache License 2.0. See `LICENSE` and `NOTICE` for details.
6 changes: 4 additions & 2 deletions xff/cli/help.cc
Original file line number Diff line number Diff line change
Expand Up @@ -309,8 +309,10 @@ struct Recipe {

// The `--help=cookbook` topic (aliases examples / recipes), folded into --help=full: task-oriented
// worked examples that compose xff's building blocks end to end. The SOT is the recipe list below;
// every command is kept runnable as written. This complements the reference topics (--help=fields /
// --help=stats / --help=NAME), which describe pieces in isolation.
// every command is kept runnable as written, and is executed end to end by the tests in
// //xff/examples:cookbook_test - whose guard case fails CI if a recipe is added or reworded here
// without a matching test, so these examples ship tested, not just rendered. This complements the
// reference topics (--help=fields / --help=stats / --help=NAME), which describe pieces in isolation.
std::string RenderCookbook() {
const std::vector<Recipe> recipes = {
{.task = "Ten largest files",
Expand Down
29 changes: 29 additions & 0 deletions xff/examples/BUILD.bazel
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# SPDX-FileCopyrightText: Copyright (c) The helly25 authors (helly25.com)
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

load("@helly25_bashtest//bashtest:bashtest.bzl", "bashtest")

package(default_visibility = ["//visibility:private"])

# Execution tests for the user-facing examples: every `--help=cookbook` recipe (SOT:
# xff/cli/help.cc `RenderCookbook`) is RUN here against a fixture tree, not just rendered,
# so a copy-paste of any recipe actually works. A guard case (cookbook_recipes_are_all_tested)
# fails if a recipe is added or changed without a matching test here - examples ship tested.
bashtest(
name = "cookbook_test",
size = "small",
srcs = ["cookbook_test.sh"],
data = ["//xff/cli:xff"],
)
Loading
Loading