feat: TOML Part 1 — filter DSL engine + 14 built-in filters

[FlorianBruniaux]

TOML Part 1 of 3 — establishes the DSL engine. See also Part 2 and Part 3.

What this PR does

Adds a declarative TOML-based filter engine that lets anyone add a new command filter without writing Rust. A TOML block in src/builtin_filters.toml (or the user's .rtk/filters.toml) is enough for commands with stable, predictable output.

This is a foundational change: it lowers the contribution bar for simple filters from ~200 LOC of Rust to a ~10-line TOML block.

---

Architecture

New files

File	Role
src/toml_filter.rs	Engine: TOML parsing, filter application, run_fallback() integration
src/builtin_filters.toml	Built-in filters (shipped with the binary via include_str!)
src/verify_cmd.rs	rtk verify command — runs inline TOML tests, supports --require-all

How it works

 rtk tofu plan
      │
      ▼
 ┌──────────┐    known command    ┌─────────────────┐
 │   Clap   │ ──────────────────► │   Rust module   │
 │  parse   │                     │  (git, cargo…)  │
 └──────────┘                     └─────────────────┘
      │
      │ unknown command (parse error)
      ▼
 ┌──────────────┐
 │ run_fallback │
 └──────┬───────┘
        │
        ├── TOML filter match? ── NO ──► exec raw (passthrough, unchanged)
        │
        │ YES
        ▼
   exec command
   capture stdout
        │
        ▼
   apply filter          ◄── primitives applied in declaration order
   primitives
        │
        ▼
   print filtered
   output + exit code

User filters in .rtk/filters.toml are loaded at startup and merged with built-ins. User filters take precedence over built-ins.

Filter pipeline

 raw stdout
     │
     ▼
 strip_ansi ─────────────────────────────────── (if enabled)
     │
     ▼
 match_output ── pattern match? ──► emit short message, exit early
     │ no match
     ▼
 strip_lines_matching   ──► drop lines matching any regex
 keep_lines_matching    ──► keep only lines matching any regex
     │
     ▼
 replace ────────────────────────────────────── (regex find+replace)
     │
     ▼
 truncate_lines_at ──────────────────────────── (chars per line cap)
     │
     ▼
 head_lines / tail_lines ───────────────────── (first/last N lines)
     │
     ▼
 max_lines ──────────────────────────────────── (total line cap)
     │
     ▼
 on_empty ── output empty? ──► emit fallback message
     │
     ▼
 print to stdout

TOML vs Rust — decision guide

 New command to support?
         │
         ▼
  Output is stable &        NO ──────────────────────────────────────┐
  line-oriented?                                                      │
         │ YES                                                        │
         ▼                                                            │
  Filtering = strip noise,  NO → JSON parsing, aggregation,          │
  keep summaries,               state machines, HashMap?             │
  head/tail, truncate?          │                                    │
         │ YES                  ▼                                    ▼
         ▼              Write a Rust module              Write a Rust module
   Add a TOML filter        (_cmd.rs)                       (_cmd.rs)
   in builtin_filters.toml
   or .rtk/filters.toml

Filter primitives (8 total)

Primitive	Effect
strip_ansi	Remove ANSI escape sequences
strip_lines_matching	Drop lines matching any regex in the list
keep_lines_matching	Keep only lines matching any regex
replace	Regex find+replace on the full output
match_output	Short-circuit: if output matches pattern, emit a short message and stop
truncate_lines_at	Truncate lines longer than N chars
head_lines / tail_lines	Keep first/last N lines
max_lines	Cap total output lines
on_empty	Message to emit when filtered output is empty

---

14 new built-in filters

#240 — OpenTofu (4 filters)

Mirror of the existing terraform-* filters, targeting tofu CLI.

Filter	Command	Savings target
tofu-plan	tofu plan	~80% (strips Refreshing state, lock lines, blanks)
tofu-init	tofu init	~70% (strips provider download noise)
tofu-validate	tofu validate	short-circuit on Success!
tofu-fmt	tofu fmt	on_empty for no changes

#284 — du (1 filter)

Strips blank lines, truncates long paths at 120 chars, caps at 40 lines.

#281 — fail2ban-client (1 filter)

Strips blank lines, caps at 30 lines. Note: no sudo prefix — the hook rewrite does not capture sudo, so the prefix would be dead code.

#282 — iptables (1 filter)

Defensive regex: strips only Docker-autogenerated chains (^Chain DOCKER, ^Chain BR-). Does not strip the rule lines that follow those chains. Caps at 50 lines, truncates at 120 chars.

#310 — Elixir mix (2 filters)

Filter	Command	Strategy
mix-format	mix format	on_empty → mix format: ok
mix-compile	mix compile	strips Compiling N files, Generated, blanks; preserves warnings/errors

#280 — Shopify Theme (1 filter)

Strips Uploading/Downloading lines, keeps last 5 lines (the summary) via tail_lines, caps at 15.

#231 — PlatformIO (1 filter, partial)

pio run: strips build noise (CONFIGURATION, LDF, Compiling, Linking, Building, Checking size). Errors and the final memory summary are preserved.

Note: this closes the pio run case. Other commands from #231 (npm test, python, stty) remain open.

#338 — Maven (1 filter, partial)

mvn compile|package|clean|install: uses strip_lines_matching (not keep) to strip [INFO] ---, [INFO] Building, download lines, and blanks. This preserves stacktraces, plugin warnings, and any unexpected output — only known noise is stripped.

Note: mvn test (Surefire state machine) is out of scope for TOML; it requires a Rust module.

---

Inline test suite

Every filter has at minimum 2 inline tests (realistic fixture + edge case). Run with:

rtk verify             # 38/38 pass
rtk verify --require-all   # fails if any filter has zero tests

Test results at time of this PR: 38/38 passed.

---

Issues closed / addressed

Closed by this PR

Issue	Status
#299	This PR — TOML filter DSL feature request
#298	Duplicate of #299 — can be closed
#286	rtk must pass through unrecognized commands — resolved by run_fallback() + TOML routing
#240	OpenTofu support — 4 built-in filters added
#284	du support — filter added
#281	fail2ban-client support — filter added
#282	iptables support — filter added
#310	mix format and mix compile — 2 filters added

Partially addressed

Issue	What's covered	What remains
#231	pio run filter added	npm test, python, stty, other 7 commands from the issue
#280	shopify theme push/pull filter added	Other Shopify subcommands
#338	mvn compile/package/clean/install filter added	mvn test (needs Rust state machine for Surefire)

Not addressed (documented reasons)

Issue	Reason
#275 docker exec	Output is unpredictable — depends on the inner command; no stable format to filter
#283 stat	macOS stat -f vs Linux format divergence; savings < 60%; low utility
#333 ssh	Command dispatch for remote execution needs Rust routing
#271 jj	Needs compression-quality filtering (~60-80%); TOML primitives cap at ~10-20%

---

PRs with technical coordination needed

Based on the pre-merge impact analysis (claudedocs/toml-impact-triage-2026-03-05.md):

Conflicts (should merge before this PR ideally)

PR	Author	Zone	Type
#306	@jbgriesner	lazy_static → LazyLock migration	This PR uses lazy_static! in toml_filter.rs. If #306 merges first, we switch to LazyLock — cleaner result.
#326	@rursache	run_fallback() stderr/-- separator	This PR also modifies run_fallback(). Merging #326 first avoids a painful rebase.

Architectural note (no code conflict)

PR	Author	Note
#268	@dragonGR	Stream proxy output. TOML captures stdout via Stdio::piped for matched commands; streaming behavior for unmatched commands (inherit) is unchanged. No conflict, but related architecturally.

PRs NOT affected by this PR

All 7 PRs deep-dived in the impact analysis are Rust-required — they use state machines, JSON aggregation, or HashMaps that TOML primitives cannot express:

PR	Author	Why Rust is needed
#341	@philbritton	25 JSON handlers, serde_json::Value, metrics aggregation
#308	@cmolder	BazelBuildState / BazelTestState, BTreeMap, streaming
#312	@kherembourg	test/connected/deps subcommands = state machines
#263	@yonatankarp	in_failure_block failure tracking
#288	@rubixhacker	6 filters with aggregation + reformatting
#290	@charlesvien	Regex extraction + summary reformatting
#253	@TheGlitching	serde_json + HashMap<String, usize> aggregation

The ~35 other open PRs (Rust filters, bug fixes, docs, infra, hooks) are unaffected.

---

Out of scope (documented for follow-up)

• Hook rewrite coverage (Hook rewrite: add coverage for uv run, pnpm exec, Python path variants, and more #294): The 14 new filters only activate if the hook rewrites the command to rtk <cmd>. Adding tofu, du, fail2ban-client, iptables, mix, shopify, pio, mvn to the hook is a separate PR.
• rtk discover TOML recommendations: rtk discover could suggest TOML filters for unhandled commands — separate issue.
• CONTRIBUTING.md: Should document when to use TOML vs Rust — separate PR.

---

Test plan

# Quality gates
cargo fmt --all && cargo clippy --all-targets && cargo test --all
# → 656 passed, 0 errors

# Inline tests
cargo build --release
./target/release/rtk verify             # 38/38 passed
./target/release/rtk verify --require-all

# Performance (20 total filters should not impact startup)
/usr/bin/time -p ./target/release/rtk git status  # < 50ms wall

🤖 Generated with Claude Code

[pszymkowiak]

Code Review — TOML Filter DSL

The TOML engine itself is well-designed: lazy_static registry, deny_unknown_fields, 8-stage pipeline, 38/38 inline tests, schema_version forward-compat. Solid work.

Note: This branch needs a rebase onto current master before merge (missing Graphite support, diff limit fixes, Discord notification, 0.27.2 changelog). The issues below are specific to this PR's code, not rebase artifacts.

---

P1 — Important

P1-1: Missing trailing newline in TOML-filtered output
main.rs:1004 — print!("{}", filtered) without \n. Confirmed in live testing: rtk make all | wc -l returns 0 for 1 line of content. Breaks pipe compatibility.

P1-2: Dead TOML filters — git-checkout, git-remote, git-merge, cargo-run never fire
Clap routes these to git.rs/cargo_cmd.rs before reaching run_fallback. The filters compile and their inline tests pass, giving false confidence that they work.

P1-3: is_none_or requires Rust 1.82+
toml_filter.rs:442 — No MSRV declared. Fix: filter_name_opt.map_or(true, |f| name == f).

P1-4: Tee hint silently discarded for TOML-filtered commands
main.rs:996-1001 — let _ = tee::tee_and_hint(...) discards the hint string. Every other caller in the codebase uses if let Some(hint) = ... { println!(...) }. LLMs never learn the tee file exists when a TOML-filtered command fails.

P1-5: rtk verify --require-all skips hook integrity check
main.rs:1874-1881 — if filter.is_some() || require_all branches away from integrity::run_verify(). CI with --require-all silently skips the hook integrity check that the default rtk verify runs.

---

P2 — Minor

#	Issue
P2-1	RTK_NO_TOML=0 disables the engine (.is_ok() checks presence, not value). Confirmed in live testing.
P2-2	max_lines off-by-one: truncation message adds 1 line beyond the cap (documented in test but surprising)
P2-3	make filter returns "" on clean build (no on_empty defined, unlike terraform-plan)
P2-4	TOML filters not in hook rewrite registry — terraform plan typed in Claude Code shell is not intercepted by the hook
P2-5	^terraform\s+plan matches terraform planning (no word boundary after plan). Confirmed with RTK_TOML_DEBUG=1.
P2-6	^iptables\b matches iptables-save and iptables-restore (\b fires at hyphen). Fix: ^iptables(\s|$)
P2-7	^git\s+merge matches git merge-base, git mergetool. Fix: ^git\s+merge(\s|$)
P2-8	Commands with absolute paths (/usr/bin/make all) never match TOML filters. Confirmed.
P2-9	utils.rs docstrings translated to French — inconsistent with the English codebase
P2-10	git -C test weakened — directory == vec!["/path"] assertion removed, replaced by bare is_ok()
P2-11	test_meta_command_list_is_complete missing "verify" despite it being in RTK_META_COMMANDS
P2-12	Filter priority is alphabetical (BTreeMap), not definition order — undocumented
P2-13	rtk verify: failures on stderr, summary on stdout — inconsistent for CI piping
P2-14	CHANGELOG says 14 built-in filters, actual count is 18

---

P3 — Nits

#	Issue
P3-1	trim_end_matches('\n') on both actual and expected in verify tests could mask trailing blank line bugs
P3-2	verify --filter nonexistent --require-all exits 0 silently (filter name not validated)
P3-3	ARCHITECTURE.md: header says 59 (38+21), breakdown says 34+20=54
P3-4	O(N) scan in find_filter_in — fine at 18 filters, monitor when scaling to 200+
P3-5	strip_ansi regex misses OSC sequences and hyperlinks (pre-existing in utils.rs)

---

Action items

1. Rebase onto master HEAD first
2. Fix P1-1 (println! instead of print!)
3. Fix P1-3 (map_or instead of is_none_or)
4. Fix P1-4 (print tee hint)
5. Fix P1-5 (always run integrity check)
6. Decide on P1-2: remove dead filters or document the limitation
7. Fix regex patterns for P2-5/P2-6/P2-7 (add word boundaries)