Skip to content

v0.4.1

Choose a tag to compare

View all tags
@bolei-fleak bolei-fleak released this 13 Mar 18:54
· 158 commits to main since this release
v0.4.1
62d628e
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

ZephFlow v0.4.1 Release Notes

Highlights

Java 21 Upgrade — ZephFlow now requires Java 21 (up from Java 17), with new language features applied across the codebase.
(#166)

New Features

  • InfluxDB v2.x Metrics Provider — Added support for InfluxDB v2.x as a metrics backend with performance optimizations.
  • Delta Lake Sink SDK API — New SDK API for Delta Lake sink with integration tests. (#138)
  • S3 Sink Parquet Support — S3 sink now supports writing Parquet files. (#130)
  • Databricks Sink Schema Validation — Added schema validation for Databricks sink to catch mismatches early. (#127)
  • Raw Data Sampler — Added raw data sampler to source commands for debugging/inspection. (#156)
  • Configurable Debug Logging — ZephFlow debug level can now be set directly in the DAG file. (#151)
  • New FEEL Functions — Added in function and additional FEEL functions. (#146, #150)

Improvements

  • KV Parser Refactor — Rewritten as a single-pass O(n) state machine; now aggregates duplicate keys into arrays. (#139,
    #126)
  • NodeExecutionException — Better error context in DAG execution with node ID included in dead letter records. (#140, #143,
    #153)
  • Dead Letter Improvements — Refined dead letter file paths with prefix from jobContext. (#154)
  • Eval Compiler Lenient Mode — Eval compiler no longer breaks on invalid expressions. (#136)
  • BufferedWriter Validation — Flush interval validated to be positive; default S3 flush interval set to 10 seconds. (#135)
  • Assertion Messages — Assertion expression now included in error messages. (#160)
  • Metrics Tags — KafkaSource, DeltaLake, and Databricks sinks now send tag object properties with metrics.
  • Input Record Sanitization — Input records removed from command error messages. (#161)

Bug Fixes

  • KafkaSink Counting — Fixed incorrect error counting in KafkaSink. (#159)
  • Python Eval — Fixed python() eval failing with Map/dict arguments by recursive conversion. (#163)
  • JsonConfigParser — Fixed null handling and S3 key trailing slash issues. (#133)
  • DAG Copy — DAG copy now creates a deep copy; S3 sink no longer accepts JSON_OBJECT encoding in batch mode. (#134)
  • Parser Null Fields — Parser now works with null fields; arr_foreach gracefully handles non-array input. (#144)

Dependency Updates

  • Databricks SDK: 0.73.0 → 0.100.0
  • TestContainers: 1.21.2 → 1.21.4
  • Added InfluxDB v2 client (7.2.0)
  • ClickHouse: 0.9.5 → 0.9.7
  • Gradle and plugins updated
Assets 4
Loading