Skip to content

v1.5.0

Choose a tag to compare

@github-actions github-actions released this 10 Apr 19:31
· 509 commits to main since this release

Added

  • Schema discovery mode - Added --discover[=table|json] for parsed-input schema profiling and --discover-final[=table|json] for post-filter/post-transform output schema profiling. Discovery includes nested field flattening, reservoir-sampled examples, and cap warnings for high-cardinality streams.
  • Cascade format mode - Pass a comma-separated list to -f (e.g. -f json,line) to try each parser in order per line; the first success wins. Each event is tagged with the winning parser name in _format. Supported formats: json, logfmt, syslog, cef, combined, line, raw. Put line last as a catch-all fallback.
  • Chronological merge for sorted files - Added --merge-sorted to merge multiple already-sorted inputs into one chronological stream with a memory-bounded k-way merge. This is intended for structured batch files with reliable timestamps, works with any input format except -f auto-per-file, and aborts on missing timestamps, merge-time parse failures, and per-file disorder.
  • Per-file auto-detection - Added -f auto-per-file for batches where each file is internally consistent but different files use different formats.
  • --include now works with --filter - Helper functions defined in an include file (-I) can now be called from --filter expressions. Constraint: the include file must contain only function definitions — top-level statements are rejected with a clear error.
  • New Rhai display helpers - A cohesive set for scripted terminal summaries:
    • Numeric: human_bytes / human_bytes_si, format_decimals(v, n), format_percent(ratio, n); values just below a unit threshold round up cleanly to the next unit (for example 1.0 GiB instead of 1024.0 MiB)
    • Alignment (Unicode-width-aware): ljust(n), rjust(n), center(n), shorten(n)
    • ANSI styling: red(), green(), yellow(), blue(), cyan(), magenta(), bold(), dim() — chainable; no-op when colour is disabled
    • Charts: bar(value, max, width) for block bars; sparkline(array) for trend lines. For ratios in 0.0–1.0, set max to 1.0.
  • New Rhai projection helpers - Added map.keep() and map.drop() for event field projection.
  • New Rhai time/random helpers - Added dt.ceil_to() and sample_prob().
  • New PII validators - Added ssn and phone pattern validators to normalized(), including SSA-aware SSN checks and NANP-aware validation for US/CA numbers plus permissive support for other international numbers.

Changed

  • CLI diagnostics - Improved actionable guidance for wrong mode, wrong format, missing --keys, missing input, and common flag conflicts.
  • CLI option validation - --discover/--drain conflicts with --parallel are now validated at CLI parse time with clearer errors.

Fixed

  • Parser strict mode behavior - JSON and CEF parsers now correctly respect the strict parsing flag.
  • Pre-epoch rounding - Fixed round_to/ceil_to behavior for timestamps before Unix epoch.
  • Auto-detection behavior - -f auto and -f auto-per-file now consistently detect from the first non-empty line, matching the built-in help and format reference. -f auto-per-file also now preserves sequential state across files and reports detected formats per file in --stats.

What's Changed

  • Support filter includes without side effects by @dloss in #135
  • Add map.keep/drop projection helpers for Rhai events by @dloss in #136
  • rhai: add map.keep and map.drop to select/exclude top-level fields by @dloss in #137
  • Add floor_to, ceil_to, and sample_prob functions by @dloss in #139
  • Add --discover-fields for schema profiling of log streams by @dloss in #140
  • docs(cli): add Field Discovery (--discover) docs to CLI reference by @dloss in #141
  • discover: flatten nested fields, reservoir-sample examples, warn on cap by @dloss in #142
  • Add output formatting helpers: human_bytes, format_decimals, format_percent by @dloss in #143
  • Add bar and sparkline Rhai helpers for end-of-stream stats by @dloss in #144
  • Add cascade format mode for mixed-format streams by @dloss in #145
  • rhai: add map.keep and map.drop to select/exclude top-level fields by @dloss in #138
  • Add parallel cascade test and update mixed_format_demo by @dloss in #146
  • Add SSA and NANP validators for SSN and phone PII patterns by @dloss in #148
  • Fix ANSI blue and cyan to use bright variants for readability by @dloss in #149
  • Validate --discover and --drain conflict with --parallel at CLI level by @dloss in #150
  • Update CHANGELOG.md: add missing Unreleased entries since v1.4.10 by @dloss in #151
  • Field discovery: UTF-8-safe truncation for display and normalize warning text by @dloss in #152
  • Add --merge-ts option to merge multiple JSON input files by timestamp by @dloss in #147
  • Add auto-per-file input format for per-file auto-detection by @dloss in #153
  • Fix human_bytes/human_bytes_si displaying '1024.0 MiB' at unit boundaries by @dloss in #154
  • fix: --ts-field no longer normalizes timestamp strings in output by @dloss in #155
  • docs: clarify SSN format strictness in normalized() pattern docs by @dloss in #156
  • docs: improve changelog clarity for cascade mode, --include/--filter, and CLI diagnostics by @dloss in #157
  • docs: clarify bar() description and add display_helpers example by @dloss in #158

Full Changelog: v1.4.10...v1.5.0