Skip to content

Version 0.9.8

Choose a tag to compare

@github-actions github-actions released this 10 May 08:29
· 11 commits to main since this release
705be40

What's Changed

Bug fixes

  • #74 — Fix NullPointerException in Block.code when from_json (or any other
    CodegenFallback expression) is used under whole-stage codegen with DataFlint
    instrumentation enabled. TimedWithCodegenExec now reports supportCodegen = false
    for wrapped operators that contain a CodegenFallback, mirroring Spark's own
    CollapseCodegenStages check that the transparent wrapper had been hiding.
  • Compatible with Spark 3.0 → 4.x (uses TreeNode.find instead of the 3.2-only
    TreeNode.exists).

Hardening

  • TimedExec.postRddId now overwrites the rddId metric instead of summing across
    re-executions of the same plan instance.
  • TimedExec and TimedWithCodegenExec no longer compare equal — fixes a corner
    case in plan canonicalization / AQE plan reuse.
  • executeCollect write-path is now bounds-safe; falls back to the standard path
    on unexpected plan shapes (vendor write commands, future Spark layouts).
  • rddId metric switched from a "size" type to plain sum (no longer rendered as
    bytes — "12 B" — in the SparkUI).

Tests / CI

  • New DataFlintCodegenFallbackSpec regression test for #74.
  • pluginspark4 now runs version-portable specs against Spark 4.0.1 in CI.
  • CI bumped to Java 17 (required by Spark 4); published jars stay Java 8 compatible.

Full Changelog: v0.9.7...v0.9.8