Skip to content

Add mask scalar function for partition normalizer#18452

Closed
xiangfu0 wants to merge 1 commit into
apache:masterfrom
xiangfu0:add-normalizer-scalar-functions
Closed

Add mask scalar function for partition normalizer#18452
xiangfu0 wants to merge 1 commit into
apache:masterfrom
xiangfu0:add-normalizer-scalar-functions

Conversation

@xiangfu0
Copy link
Copy Markdown
Contributor

@xiangfu0 xiangfu0 commented May 10, 2026

Summary

  • Add integral mask(value) scalar function for the PartitionIdNormalizer.MASK logic by stripping the sign bit.
  • Cover scalar registration, SQL parsing, ingestion evaluator, and MSQ planner paths.

User Manual

mask(value) supports INT and LONG inputs and clears the sign bit, matching PartitionIdNormalizer.MASK. Callers can apply modulo separately when they want a partition id.

Sample Data

Assume a table normalizer_demo with:

hash_int hash_long
-2147483648 -9223372036854775808
-10 -10
10 10

Sample Queries And Results

SELECT
  hash_int,
  mask(hash_int) AS masked_int,
  hash_long,
  mask(hash_long) AS masked_long
FROM normalizer_demo;
hash_int masked_int hash_long masked_long
-2147483648 0 -9223372036854775808 0
-10 2147483638 -10 9223372036854775798
10 10 10 10

Callers can apply modulo separately:

SELECT
  hash_int,
  mask(hash_int) % 16 AS masked_partition
FROM normalizer_demo;
hash_int masked_partition
-2147483648 0
-10 6
10 10

Tests

  • ./mvnw -pl pinot-common -am -Dtest=BitwiseScalarFunctionTest,CalciteSqlCompilerTest -Dsurefire.failIfNoSpecifiedTests=false test
  • ./mvnw -pl pinot-core -am -Dtest=org.apache.pinot.core.data.function.BitFunctionsTest -Dsurefire.failIfNoSpecifiedTests=false test
  • ./mvnw -pl pinot-query-planner -am -Dtest=org.apache.pinot.query.QueryCompilationTest#testPolymorphicBitwiseScalarFunctionsPlanQuery -Dsurefire.failIfNoSpecifiedTests=false test
  • ./mvnw spotless:apply checkstyle:check license:format license:check -pl pinot-common,pinot-core,pinot-query-planner,pinot-integration-tests
  • git diff --check

@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented May 10, 2026

Codecov Report

❌ Patch coverage is 85.71429% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.65%. Comparing base (a3d7b2a) to head (989392d).
⚠️ Report is 2 commits behind head on master.

Files with missing lines Patch % Lines
...on/function/scalar/bitwise/MaskScalarFunction.java 85.71% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##             master   #18452   +/-   ##
=========================================
  Coverage     63.65%   63.65%           
  Complexity     1684     1684           
=========================================
  Files          3256     3257    +1     
  Lines        199519   199562   +43     
  Branches      30989    30986    -3     
=========================================
+ Hits         127003   127034   +31     
- Misses        62377    62389   +12     
  Partials      10139    10139           
Flag Coverage Δ
custom-integration1 100.00% <ø> (ø)
integration 100.00% <ø> (ø)
integration1 100.00% <ø> (ø)
integration2 0.00% <ø> (ø)
java-21 63.65% <85.71%> (+<0.01%) ⬆️
temurin 63.65% <85.71%> (+<0.01%) ⬆️
unittests 63.65% <85.71%> (+<0.01%) ⬆️
unittests1 55.72% <85.71%> (+<0.01%) ⬆️
unittests2 34.98% <57.14%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@xiangfu0 xiangfu0 requested a review from Jackie-Jiang May 10, 2026 22:19
@xiangfu0 xiangfu0 added the functions Related to scalar or aggregation functions label May 10, 2026
@xiangfu0 xiangfu0 force-pushed the add-normalizer-scalar-functions branch from a400751 to 989392d Compare May 10, 2026 22:58
@xiangfu0 xiangfu0 changed the title Add scalar functions for partition normalizers Add mask scalar function for partition normalizer May 10, 2026
@xiangfu0 xiangfu0 closed this May 11, 2026
@xiangfu0 xiangfu0 deleted the add-normalizer-scalar-functions branch May 11, 2026 08:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

functions Related to scalar or aggregation functions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants