Skip to content

Optimize Spark hex null handling#23688

Open
floze-the-genius wants to merge 1 commit into
apache:mainfrom
floze-the-genius:perf/23674-spark-hex-null-fast-path
Open

Optimize Spark hex null handling#23688
floze-the-genius wants to merge 1 commit into
apache:mainfrom
floze-the-genius:perf/23674-spark-hex-null-fast-path

Conversation

@floze-the-genius

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

Spark hex encoding preserves input validity exactly, but the byte-array path rebuilt the null bitmap row by row and always iterated through Option values. The input arrays already expose both the original NullBuffer and an all-valid fast path.

What changes are included in this PR?

  • pass the byte array accessor into hex_encode_bytes so the output can clone and reuse the input NullBuffer
  • split nullable and no-null loops, avoiding per-row validity checks when the input has no null buffer
  • preserve exact offset, overflow, casing, dictionary, and null semantics
  • add a sliced-array regression that verifies semantic equality and bitmap pointer reuse
  • add no-null output coverage and dedicated UTF-8/binary Criterion cases

Are these changes tested?

Yes.

  • cargo test -p datafusion-spark function::math::hex --lib: 10 passed
  • cargo fmt --all -- --check: passed
  • cargo clippy -p datafusion-spark --all-targets --all-features --no-deps -- -D warnings: passed
  • git diff --check: passed
  • benchmark target compiled with --features core

The workspace dependency clippy invocation without --no-deps reaches an unrelated existing dead_code warning in datafusion/physical-expr/src/expressions/dynamic_filters/mod.rs:464; the changed crate is clean.

Benchmark

Command (same warmed build and profile for upstream main and this branch):

CARGO_BUILD_JOBS=2 \
CARGO_PROFILE_BENCH_LTO=false \
CARGO_PROFILE_BENCH_CODEGEN_UNITS=16 \
cargo bench -p datafusion-spark --features core --bench hex -- \
  'hex_(utf8|binary)_no_nulls'

Mean times from the immediate upstream-then-branch sequence:

case upstream main this PR change
UTF-8, 1,024 rows 79.895 us 59.181 us -25.9%
UTF-8, 4,096 rows 317.64 us 244.88 us -22.9%
UTF-8, 8,192 rows 509.64 us 411.27 us -19.3%
Binary, 1,024 rows 86.135 us 40.600 us -52.9%
Binary, 4,096 rows 346.16 us 253.13 us -26.9%
Binary, 8,192 rows 635.98 us 424.78 us -33.2%

Are there any user-facing changes?

No API or behavior changes. This is an internal performance optimization for Spark-compatible hex expressions.

Reuse the input null buffer and split nullable and no-null encoding paths to avoid rebuilding validity and per-value Option checks. Add pointer-reuse and no-null regressions plus dedicated Criterion cases.
@github-actions github-actions Bot added the spark label Jul 18, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 71.08434% with 24 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.67%. Comparing base (1734d4b) to head (4e9f34a).

Files with missing lines Patch % Lines
datafusion/spark/src/function/math/hex.rs 71.08% 12 Missing and 12 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #23688      +/-   ##
==========================================
- Coverage   80.67%   80.67%   -0.01%     
==========================================
  Files        1087     1087              
  Lines      367541   367560      +19     
  Branches   367541   367560      +19     
==========================================
- Hits       296529   296524       -5     
- Misses      53355    53367      +12     
- Partials    17657    17669      +12     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Further optimize Spark hex byte encoding: reuse input NullBuffer and special-case the no-nulls path

2 participants