Version 0.9.8
What's Changed
Bug fixes
- #74 — Fix
NullPointerExceptioninBlock.codewhenfrom_json(or any other
CodegenFallbackexpression) is used under whole-stage codegen with DataFlint
instrumentation enabled.TimedWithCodegenExecnow reportssupportCodegen = false
for wrapped operators that contain aCodegenFallback, mirroring Spark's own
CollapseCodegenStagescheck that the transparent wrapper had been hiding. - Compatible with Spark 3.0 → 4.x (uses
TreeNode.findinstead of the 3.2-only
TreeNode.exists).
Hardening
TimedExec.postRddIdnow overwrites therddIdmetric instead of summing across
re-executions of the same plan instance.TimedExecandTimedWithCodegenExecno longer compare equal — fixes a corner
case in plan canonicalization / AQE plan reuse.executeCollectwrite-path is now bounds-safe; falls back to the standard path
on unexpected plan shapes (vendor write commands, future Spark layouts).rddIdmetric switched from a "size" type to plain sum (no longer rendered as
bytes —"12 B"— in the SparkUI).
Tests / CI
- New
DataFlintCodegenFallbackSpecregression test for #74. pluginspark4now runs version-portable specs against Spark 4.0.1 in CI.- CI bumped to Java 17 (required by Spark 4); published jars stay Java 8 compatible.
Full Changelog: v0.9.7...v0.9.8