Skip to content

Add datasketches HLL sketch aggregate functions#63143

Open
nooneuse wants to merge 12 commits into
apache:masterfrom
nooneuse:add_datasketches_union_aggregate_functions
Open

Add datasketches HLL sketch aggregate functions#63143
nooneuse wants to merge 12 commits into
apache:masterfrom
nooneuse:add_datasketches_union_aggregate_functions

Conversation

@nooneuse
Copy link
Copy Markdown

@nooneuse nooneuse commented May 11, 2026

What problem does this PR solve?

An aggregate function is required to process user data containing Datasketches HLL sketches. In many data aggregation scenarios, users pre‑aggregate detailed data in Hive using the sketching techniques provided by Apache Datasketches, and then analyze the resulting sketches across various OLAP engines. Compared with the HLL union aggregate functions natively offered by these engines, there are two key diff to using Datasketches HLL sketches: firstly, the use cases differ; and secondly, HLL sketches can be used seamlessly across different engines—for example, simultaneously in ES, Doris, and ClickHouse. Such requirements are common in many production environments.

Issue Number:

Summary:
Implemented a built-in aggregate function that integrates the Datasketches HLL sketch. This aggregate function cannot rely on the Java UDF environment. Considering that in the Java UDF environment, Strings are encoded in UTF-8, which corrupts the binary data of sketches, the serialization/deserialization operations for sketches must be implemented on the BE side. (additionally, since Apache Datasketches has been added to the contrib directory via a git submodule, it will become very easy to add other sketches such as theta sketch in the future.)

see: #63142
use case: see regression test & #63142

Release note

Implemented an aggregate function that integrates the Datasketches HLL sketch.

Check List (For Author)

  • Test

    • [v] Regression test —— checked
    • [v] Unit Test —— checked
  • Behavior changed:

    • [v] No.
  • Does this need documentation?

    • [v] No. No separate documentation is needed; the usage is easy to understand, and it is clearly explained in the regression tests.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

Comment thread fe/pom.xml
Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

setting the Maven version constraint to [3.6.3,) is sufficient for normal compilation; [3.9.0,) is not required.

@BePPPower
Copy link
Copy Markdown
Contributor

run buildall

@nooneuse
Copy link
Copy Markdown
Author

run buildall

@nooneuse
Copy link
Copy Markdown
Author

compile

@hello-stephen
Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 4.00% (1/25) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 78.05% (1849/2369)
Line Coverage 64.73% (33222/51327)
Region Coverage 65.25% (16441/25198)
Branch Coverage 55.81% (8780/15732)

@hello-stephen
Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 4.00% (1/25) 🎉
Increment coverage report
Complete coverage report

@nooneuse
Copy link
Copy Markdown
Author

run buildall

@nooneuse
Copy link
Copy Markdown
Author

run buildall

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants