Add useRawBytes config to Murmur and Murmur3 partition functions by xiangfu0 · Pull Request #17932 · apache/pinot

xiangfu0 · 2026-03-22T02:54:09Z

Summary

Add useRawBytes option to functionConfig for MurmurPartitionFunction and Murmur3PartitionFunction (defaults to false)
When useRawBytes=true, the hex-encoded partition value is decoded back to raw bytes via BytesUtils.toBytes() before hashing, instead of treating it as UTF-8 text via value.getBytes(UTF_8)
This enables correct partition assignment for BYTES columns where the value is hex-encoded but should be hashed on the original raw bytes

Test plan

Added testMurmurPartitionFunctionUseRawBytes — verifies hex-decoded bytes produce expected murmur2 hash partition, and that default behavior is unchanged
Added testMurmur3PartitionFunctionUseRawBytes — verifies hex-decoded bytes for x86_32, x64_32 variants, and non-zero seed, and that default behavior is unchanged
All existing PartitionFunctionTest tests pass

🤖 Generated with Claude Code

When partitioning on BYTES columns, the partition value is hex-encoded. With useRawBytes=true in functionConfig, the hex string is decoded back to raw bytes before hashing, ensuring partition assignment matches the original byte values rather than treating the hex string as UTF-8 text. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Copilot

Pull request overview

This PR adds a useRawBytes option to Murmur/Murmur3 partition functions so BYTES values that are hex-encoded strings can be partitioned by hashing the original decoded bytes (instead of hashing the UTF-8 bytes of the hex text).

Changes:

Add useRawBytes parsing and hex-string decoding via BytesUtils.toBytes() in MurmurPartitionFunction and Murmur3PartitionFunction.
Pass functionConfig into MurmurPartitionFunction from PartitionFunctionFactory.
Add unit tests covering useRawBytes=true behavior and default behavior parity.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/partition/MurmurPartitionFunction.java`	Adds `useRawBytes` support and changes constructor to accept `functionConfig`.
`pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/partition/Murmur3PartitionFunction.java`	Adds `useRawBytes` support for both x86_32 and x64_32 hashing paths.
`pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/partition/PartitionFunctionFactory.java`	Wires `functionConfig` through for Murmur/Murmur2 factory creation.
`pinot-segment-spi/src/test/java/org/apache/pinot/segment/spi/partition/PartitionFunctionTest.java`	Adds coverage for `useRawBytes` behavior and updates Murmur ctor usage.
`pinot-controller/src/test/java/org/apache/pinot/controller/utils/SegmentMetadataMockUtils.java`	Updates test helper to use the new Murmur constructor signature.

- Murmur and Murmur3 partition functions now persist and return the functionConfig via getFunctionConfig(), ensuring partition metadata written to ZK retains the config so brokers can reconstruct the function correctly. - Re-introduce MurmurPartitionFunction(int) constructor delegating to the new (int, Map) constructor to preserve SPI backward compatibility. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

- Store functionConfig as Collections.unmodifiableMap() to prevent external mutation from diverging with parsed fields. - Annotate _functionConfig field and getFunctionConfig() with @nullable in both MurmurPartitionFunction and Murmur3PartitionFunction. - Annotate Murmur3PartitionFunction constructor parameter with @nullable. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

Verify that PartitionFunctionFactory.getPartitionFunction() correctly wires functionConfig through to the partition function for all three aliases, and that getFunctionConfig() roundtrips the config. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

codecov-commenter · 2026-03-22T04:46:36Z

Codecov Report

❌ Patch coverage is 95.00000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 63.24%. Comparing base (550afb7) to head (5f21f1c).
⚠️ Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
...segment/spi/partition/MurmurPartitionFunction.java	87.50%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #17932      +/-   ##
============================================
+ Coverage     63.19%   63.24%   +0.04%     
- Complexity     1481     1490       +9     
============================================
  Files          3191     3191              
  Lines        192592   192609      +17     
  Branches      29537    29542       +5     
============================================
+ Hits         121710   121813     +103     
+ Misses        61356    61251     -105     
- Partials       9526     9545      +19

Flag	Coverage Δ
custom-integration1	`100.00% <ø> (ø)`
integration	`100.00% <ø> (ø)`
integration1	`100.00% <ø> (ø)`
integration2	`0.00% <ø> (ø)`
java-11	`63.21% <95.00%> (+0.05%)`	⬆️
java-21	`63.15% <95.00%> (-0.01%)`	⬇️
temurin	`63.24% <95.00%> (+0.04%)`	⬆️
unittests	`63.24% <95.00%> (+0.04%)`	⬆️
unittests1	`55.49% <95.00%> (-0.01%)`	⬇️
unittests2	`34.27% <0.00%> (+0.04%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…inot#17932)

…inot#17932) (#561) Co-authored-by: Pinot Docs Bot <docs-bot@pinot.apache.org>

…che#17932) * Add useRawBytes config to Murmur and Murmur3 partition functions When partitioning on BYTES columns, the partition value is hex-encoded. With useRawBytes=true in functionConfig, the hex string is decoded back to raw bytes before hashing, ensuring partition assignment matches the original byte values rather than treating the hex string as UTF-8 text. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Override getFunctionConfig() and restore backward-compatible constructor - Murmur and Murmur3 partition functions now persist and return the functionConfig via getFunctionConfig(), ensuring partition metadata written to ZK retains the config so brokers can reconstruct the function correctly. - Re-introduce MurmurPartitionFunction(int) constructor delegating to the new (int, Map) constructor to preserve SPI backward compatibility. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Defensive copy and @nullable annotations for functionConfig - Store functionConfig as Collections.unmodifiableMap() to prevent external mutation from diverging with parsed fields. - Annotate _functionConfig field and getFunctionConfig() with @nullable in both MurmurPartitionFunction and Murmur3PartitionFunction. - Annotate Murmur3PartitionFunction constructor parameter with @nullable. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add factory end-to-end test for useRawBytes with Murmur/Murmur2/Murmur3 Verify that PartitionFunctionFactory.getPartitionFunction() correctly wires functionConfig through to the partition function for all three aliases, and that getFunctionConfig() roundtrips the config. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

xiangfu0 requested review from Jackie-Jiang, abhioncbr and Copilot March 22, 2026 02:55

xiangfu0 added the enhancement Improvement to existing functionality label Mar 22, 2026

Copilot started reviewing on behalf of xiangfu0 March 22, 2026 02:56 View session

Copilot AI reviewed Mar 22, 2026

View reviewed changes

xiangfu0 added the feature New functionality label Mar 22, 2026

xiangfu0 requested a review from Copilot March 22, 2026 03:07

Copilot started reviewing on behalf of xiangfu0 March 22, 2026 03:08 View session

Copilot AI reviewed Mar 22, 2026

View reviewed changes

xiangfu0 requested a review from Copilot March 22, 2026 03:47

Copilot started reviewing on behalf of xiangfu0 March 22, 2026 03:48 View session

Copilot AI reviewed Mar 22, 2026

View reviewed changes

Comment thread ...gment-spi/src/main/java/org/apache/pinot/segment/spi/partition/Murmur3PartitionFunction.java

Comment thread ...gment-spi/src/main/java/org/apache/pinot/segment/spi/partition/PartitionFunctionFactory.java

Jackie-Jiang approved these changes Mar 22, 2026

View reviewed changes

xiangfu0 merged commit 1329e1f into apache:master Mar 22, 2026
16 checks passed

xiangfu0 deleted the add-use-raw-bytes-partition-function branch March 22, 2026 05:19

xiangfu0 pushed a commit to pinot-contrib/pinot-docs that referenced this pull request Mar 22, 2026

docs: add useRawBytes config for Murmur partition functions (apache/p…

61f19ed

…inot#17932)

xiangfu0 mentioned this pull request Mar 22, 2026

docs: add useRawBytes config for Murmur/Murmur3 partition functions pinot-contrib/pinot-docs#561

Merged

xiangfu0 added a commit to pinot-contrib/pinot-docs that referenced this pull request Mar 22, 2026

docs: add useRawBytes config for Murmur partition functions (apache/p…

fbc43d6

…inot#17932) (#561) Co-authored-by: Pinot Docs Bot <docs-bot@pinot.apache.org>

xiangfu0 mentioned this pull request Apr 6, 2026

Fixing partitioning function for byte array #12102

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add useRawBytes config to Murmur and Murmur3 partition functions#17932

Add useRawBytes config to Murmur and Murmur3 partition functions#17932
xiangfu0 merged 4 commits intoapache:masterfrom
xiangfu0:add-use-raw-bytes-partition-function

xiangfu0 commented Mar 22, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

codecov-commenter commented Mar 22, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

xiangfu0 commented Mar 22, 2026

Summary

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

codecov-commenter commented Mar 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov-commenter commented Mar 22, 2026 •

edited

Loading