feat(template): add data-wrangling MiniJinja filters shared across commands#3921
Merged
Conversation
…mmands
Add a shared `src/minijinja_filters.rs` module registering pure, always-on
MiniJinja filters/functions that fill real gaps (verified absent from
minijinja 2.20 core, minijinja-contrib, and qsv's existing filters):
- regex_replace / regex_match / regex_find (runtime-cached patterns)
- floor / ceil (core round has no rounding mode)
- datefmt(fmt[, prefer_dmy]) (parse 19+ messy date formats via
qsv-dateparser, then reformat)
- zfill / lpad / rpad (pycompat lacks zfill/rjust/ljust)
- slugify (URL/DB/CKAN-safe slugs)
- blake3 (stable surrogate/content keys)
- fromjson / parse_json (JSON-in-a-cell -> indexable value)
- coalesce(a, b, ...) (first non-empty arg)
Wired into all four MiniJinja-powered commands via a single register() call:
template, fetchpost, describegpt, and profile. No cargo feature gate -- every
dependency used (regex, blake3, qsv-dateparser, serde_json) is always compiled
in, so the filters are available in qsv, qsvlite, qsvdp, and qsvmcp.
All functions are pure and Send + Sync, so the single Environment that
`template` shares across rayon worker threads calls them concurrently; the
regex cache uses read-lock-then-clone so matching never holds the lock.
Adds 13 tests to tests/test_template.rs; updates `template` USAGE and
regenerates docs/help/template.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 38 |
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
) - regex cache: cap at 256 entries so data-derived patterns (e.g. regex_match(pattern_column)) can't grow the process-global cache unbounded and exhaust memory. - floor/ceil: reject NaN/infinity and out-of-i64-range values via a to_i64() guard instead of an `as` cast that silently saturates to 0/i64::MIN/i64::MAX. - add regression tests for the out-of-range floor guard. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…borev #2589) i64::MAX as f64 rounds up to 2^63 (9223372036854775808.0), so the prior inclusive range check admitted 2^63 and then saturated it to i64::MAX on cast. Switch to an exclusive upper bound at 2^63 (i64::MIN is exactly representable, so the lower bound stays inclusive). Extend the regression test to cover the finite out-of-range value 2^63, not only infinity. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
) floor/ceil of an integer is the integer itself, so integer-string inputs now short-circuit via an i64 parse before any f64 conversion. This makes valid boundary values like i64::MAX (9223372036854775807, which rounds UP to 2^63 as f64) round-trip exactly instead of being rejected by the 2^63 range guard. Only genuinely fractional inputs take the f64 path. Add an i64::MAX/i64::MIN boundary regression test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Returning a float removes the i64-cast range/precision guardrails entirely
(no saturation, no 2^63 boundary edge cases, no precision loss for huge
integers). NaN/infinity now render transparently rather than silently
becoming a wrong integer. Users pipe `|int` when an integer is wanted
(`{{ v|floor|int }}`).
Updates the USAGE note + regenerates docs/help/template.md, and replaces the
i64-cast regression tests with float-output, `|int`, and non-numeric-error
cases.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Returning pure f64 lost precision for large integer inputs (e.g. i64::MAX or any ID above 2^53 rounded during the f64 parse). Restore an integer- string fast path: integer inputs pass through exactly as an integer Value, and only genuinely fractional inputs go through f64 (returning a float). No i64 cast, so still no saturation/range pitfalls. Re-add boundary regression coverage for 2^53+1, i64::MAX, and i64::MIN. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…2593) Extend the exact integer fast path to u64 so large unsigned IDs (up to u64::MAX) pass through unchanged. Integer-syntax strings that fit neither i64 nor u64 now error instead of silently approximating through f64, making the "integers stay exact" contract honest. Add regression coverage for i64::MAX+1 and u64::MAX (exact) plus u64::MAX+1 and i64::MIN-1 (error). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a shared minijinja_filters module for data-wrangling-oriented MiniJinja filters/functions and wires it into the MiniJinja-powered commands so templates can use the same helpers consistently.
Changes:
- Adds shared filters/functions for regex, rounding, date formatting, padding, slugging, hashing, JSON parsing, and coalescing.
- Registers the shared filter set in
template,fetchpost,describegpt, andprofile, plus all binary entry points. - Adds template integration tests and updates template help documentation.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
src/minijinja_filters.rs |
Implements and registers the shared MiniJinja filters/functions. |
src/main.rs |
Adds the shared module to the main binary. |
src/mainlite.rs |
Adds the shared module to the lite binary. |
src/maindp.rs |
Adds the shared module to the datapusher-plus binary. |
src/cmd/template.rs |
Documents and registers the shared filters for template. |
src/cmd/fetchpost.rs |
Registers the shared filters for payload templates. |
src/cmd/describegpt.rs |
Registers the shared filters for markdown and prompt rendering. |
src/cmd/profile/formula_helpers.rs |
Registers the shared filters in profile formula environments. |
tests/test_template.rs |
Adds coverage for the new filters through qsv template. |
docs/help/template.md |
Updates generated help for the new template filters. |
The new USAGE block started with `qsv ` (triggering help_markdown_gen's console auto-fence) and relied on indentation the generator strips, so docs/help/template.md rendered the intro as a console block and collapsed the aligned filter list. Reword the intro to not start with `qsv ` and wrap the list in an explicit ``` fence (preserved verbatim by the generator), so the columns and multi-line entries render correctly. Regenerate the help md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#2595) format_examples passed literal `> ...` blockquote lines (e.g. GitHub `[!NOTE]` admonitions) straight to the catch-all without a trailing blank line, so the note wasn't closed and the following paragraph could be absorbed as a CommonMark lazy continuation. Add a blockquote branch mirroring the existing `#`-comment handling: emit the line, then a blank line once the next non-empty line is not a blockquote line. Regenerate docs/help/template.md (only file affected). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…roborev #2596) Add regression tests for the blockquote fix: one asserts a blank line separates a `> [!NOTE]` block from following prose (guarding against the lazy-continuation bug), another asserts adjacent `>` lines stay in a single blockquote. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a shared
src/minijinja_filters.rsmodule of pure, always-on MiniJinja filters/functions that close real gaps for data-wrangling templates, wired into all four MiniJinja-powered commands (template,fetchpost,describegpt,profile) via a singleregister(env)call.Each filter was verified to be genuinely missing — not provided by minijinja 2.20 core,
minijinja-contrib, or qsv's existing filters (e.g. nothing anywhere offers regex;pycompathas nozfill/rjust/ljust; coreroundhas no rounding-mode arg).Filters / functions added
regex_replace(pattern, repl)$1/${name}capture refs)regex_match(pattern){% if %}regex_find(pattern)""floor/ceildatefmt(fmt[, prefer_dmy])qsv-dateparser) and reformatzfill(width)/lpad(width[, fill])/rpad(...)slugifyblake3fromjson/parse_jsoncoalesce(a, b, ...)Design notes
regex,blake3,qsv-dateparser,serde_jsonare always compiled in, so the filters exist inqsv,qsvlite,qsvdp, andqsvmcp.Send + Sync, so the singleEnvironmentthattemplateshares across rayon worker threads can call them concurrently. The regex cache uses read-lock-then-clone (RegexisArc-backed), so matching never holds the lock.minijinja::Error; intemplate, a bad pattern/value degrades to a per-rowRENDERING ERROR(caught + counted), not a crash.Context
This is the outcome of evaluating a "Luau-in-templates" idea, which on review mostly overlapped with existing capabilities (pycompat, printf
format, qsv'sformat_float/round_banker/lookup) and carried real costs (per-thread Lua VM, per-row context serialization, two languages in one template). These targeted filters deliver the practical value at a fraction of the complexity; heavy logic remains better served byqsv luauin a pipeline.Testing
tests/test_template.rs; all passing.template(52),profile(64),describegpt(74).-F all_features,-F lite,-F datapusher_plus.cargo clippy -F all_featuresclean for the new code;cargo +nightly fmtapplied.templateUSAGE updated anddocs/help/template.mdregenerated viaqsv --generate-help-md.🤖 Generated with Claude Code