Repository navigation
v1.5.0
Added
- Schema discovery mode - Added
--discover[=table|json]for parsed-input schema profiling and--discover-final[=table|json]for post-filter/post-transform output schema profiling. Discovery includes nested field flattening, reservoir-sampled examples, and cap warnings for high-cardinality streams. - Cascade format mode - Pass a comma-separated list to
-f(e.g.-f json,line) to try each parser in order per line; the first success wins. Each event is tagged with the winning parser name in_format. Supported formats:json,logfmt,syslog,cef,combined,line,raw. Putlinelast as a catch-all fallback. - Chronological merge for sorted files - Added
--merge-sortedto merge multiple already-sorted inputs into one chronological stream with a memory-bounded k-way merge. This is intended for structured batch files with reliable timestamps, works with any input format except-f auto-per-file, and aborts on missing timestamps, merge-time parse failures, and per-file disorder. - Per-file auto-detection - Added
-f auto-per-filefor batches where each file is internally consistent but different files use different formats. --includenow works with--filter- Helper functions defined in an include file (-I) can now be called from--filterexpressions. Constraint: the include file must contain only function definitions — top-level statements are rejected with a clear error.- New Rhai display helpers - A cohesive set for scripted terminal summaries:
- Numeric:
human_bytes/human_bytes_si,format_decimals(v, n),format_percent(ratio, n); values just below a unit threshold round up cleanly to the next unit (for example1.0 GiBinstead of1024.0 MiB) - Alignment (Unicode-width-aware):
ljust(n),rjust(n),center(n),shorten(n) - ANSI styling:
red(),green(),yellow(),blue(),cyan(),magenta(),bold(),dim()— chainable; no-op when colour is disabled - Charts:
bar(value, max, width)for block bars;sparkline(array)for trend lines. For ratios in 0.0–1.0, setmaxto1.0.
- Numeric:
- New Rhai projection helpers - Added
map.keep()andmap.drop()for event field projection. - New Rhai time/random helpers - Added
dt.ceil_to()andsample_prob(). - New PII validators - Added
ssnandphonepattern validators tonormalized(), including SSA-aware SSN checks and NANP-aware validation for US/CA numbers plus permissive support for other international numbers.
Changed
- CLI diagnostics - Improved actionable guidance for wrong mode, wrong format, missing
--keys, missing input, and common flag conflicts. - CLI option validation -
--discover/--drainconflicts with--parallelare now validated at CLI parse time with clearer errors.
Fixed
- Parser strict mode behavior - JSON and CEF parsers now correctly respect the strict parsing flag.
- Pre-epoch rounding - Fixed
round_to/ceil_tobehavior for timestamps before Unix epoch. - Auto-detection behavior -
-f autoand-f auto-per-filenow consistently detect from the first non-empty line, matching the built-in help and format reference.-f auto-per-filealso now preserves sequential state across files and reports detected formats per file in--stats.
What's Changed
- Support filter includes without side effects by @dloss in #135
- Add map.keep/drop projection helpers for Rhai events by @dloss in #136
- rhai: add map.keep and map.drop to select/exclude top-level fields by @dloss in #137
- Add floor_to, ceil_to, and sample_prob functions by @dloss in #139
- Add --discover-fields for schema profiling of log streams by @dloss in #140
- docs(cli): add Field Discovery (
--discover) docs to CLI reference by @dloss in #141 - discover: flatten nested fields, reservoir-sample examples, warn on cap by @dloss in #142
- Add output formatting helpers: human_bytes, format_decimals, format_percent by @dloss in #143
- Add bar and sparkline Rhai helpers for end-of-stream stats by @dloss in #144
- Add cascade format mode for mixed-format streams by @dloss in #145
- rhai: add map.keep and map.drop to select/exclude top-level fields by @dloss in #138
- Add parallel cascade test and update mixed_format_demo by @dloss in #146
- Add SSA and NANP validators for SSN and phone PII patterns by @dloss in #148
- Fix ANSI blue and cyan to use bright variants for readability by @dloss in #149
- Validate --discover and --drain conflict with --parallel at CLI level by @dloss in #150
- Update CHANGELOG.md: add missing Unreleased entries since v1.4.10 by @dloss in #151
- Field discovery: UTF-8-safe truncation for display and normalize warning text by @dloss in #152
- Add --merge-ts option to merge multiple JSON input files by timestamp by @dloss in #147
- Add
auto-per-fileinput format for per-file auto-detection by @dloss in #153 - Fix human_bytes/human_bytes_si displaying '1024.0 MiB' at unit boundaries by @dloss in #154
- fix: --ts-field no longer normalizes timestamp strings in output by @dloss in #155
- docs: clarify SSN format strictness in normalized() pattern docs by @dloss in #156
- docs: improve changelog clarity for cascade mode, --include/--filter, and CLI diagnostics by @dloss in #157
- docs: clarify bar() description and add display_helpers example by @dloss in #158
Full Changelog: v1.4.10...v1.5.0