Skip to content

[docs] Add RoaringBitmap SQL functions documentation - #3823

Merged
wuchong merged 3 commits into
apache:mainfrom
pbanakar:FIP37-docs
Aug 9, 2026
Merged

[docs] Add RoaringBitmap SQL functions documentation#3823
wuchong merged 3 commits into
apache:mainfrom
pbanakar:FIP37-docs

Conversation

@pbanakar

@pbanakar pbanakar commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: Part of #3289

Adds a dedicated SQL Functions reference page for the 12 RoaringBitmap SQL functions introduced by FIP-37 (PRs #3319, #3398, #3492, #3777). These are Flink-side functions registered by FlussCatalog and are distinct from the storage-level rbm32/rbm64 aggregators.
Documenting them on the Aggregation Merge Engine page blurred the ownership boundary, so this PR moves the content to a dedicated page.

Brief change log

Added website/docs/engine-flink/sql-functions.md:

  • New page titled "SQL Functions" placed after Delta Joins in the Flink engine sidebar
  • Opens with a "How to Use" section explaining catalog setup and how to enable the functions
  • Documents all 12 functions (4 aggregate, 8 scalar) with signature, null semantics table,
    and a self-contained deterministic VALUES-based example showing expected output
  • Notes where functions have no server-side counterpart (rb_and_agg, rb_xor_agg)

Modified website/docs/table-design/merge-engines/aggregation.md:

  • Removed the RoaringBitmap SQL Functions section
  • Added a short cross-reference tip block next to rbm32/rbm64 pointing to the new page

Tests

Documentation-only change. Verified locally by running the website dev server.
SQL examples verified against implemented functions in PRs #3319, #3398, #3492, and #3777.

API and Format

No API or storage format changes. Documentation only.

Documentation

Yes — this PR adds the user-facing function reference for FIP-37, completing the documentation commitment made in PRs #3319, #3398, #3492, and #3777.

@pbanakar

pbanakar commented Aug 2, 2026

Copy link
Copy Markdown
Contributor Author

@polyzos @platinumhamburg Could you please take a look at this PR?

@wuchong wuchong left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @pbanakar , I left some comments.

Comment thread website/docs/table-design/merge-engines/aggregation.md Outdated
Constructs a serialized `RoaringBitmap` from an array of integers within a single row.

- **Signature**: `rb_build(values ARRAY<INT>) -> BYTES`
- **Null Handling**: Null elements in the array are ignored; returns `NULL` if all elements are null

@wuchong wuchong Aug 5, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rb_build does not return NULL when all array elements are null. The implementation returns a serialized empty bitmap for any non-null array, and the existing test explicitly verifies that an all-null array produces a non-null empty bitmap. Therefore, rb_build(ARRAY[CAST(NULL AS INT)]) IS NULL is actually false, and rb_cardinality(...) returns 0. Please update the documentation to state that only a null array returns NULL; empty arrays and all-null arrays return an empty bitmap.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed rb_build null semantics: a NULL array returns NULL; an empty array or all-null array returns an empty bitmap

@pbanakar

pbanakar commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @wuchong for review, this looks more structured and well organized than before, PTAL another look when you have some time thanks!

@pbanakar pbanakar changed the title [docs] Add RoaringBitmap SQL functions to aggregation documentation [docs] Add RoaringBitmap SQL functions documentation Aug 5, 2026
Comment thread website/docs/engine-flink/sql-functions.md Outdated
Comment thread website/docs/engine-flink/sql-functions.md Outdated

**Signature:** `rb_build_agg(value INT) → BYTES`

| Input | Output |

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The null-handling tables take up a large portion of the page and feel repetitive, while the same rules would be easier to scan as short prose. Please use the earlier compact format and add a **Null Handling:** ... item alongside **Signature:** for each function instead of using a table.

Comment thread website/docs/table-design/merge-engines/aggregation.md
SELECT rb_cardinality(rb_or_agg(bmap)) AS result
FROM (
VALUES
(X'3A30000000000000020000000102'), -- bitmap {1, 2}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please build these inputs with rb_build rather than hard-coded serialized bytes, for example by placing arrays in VALUES and applying rb_build before rb_or_agg. This keeps the example self-explanatory and avoids opaque binary literals whose contents readers cannot verify or safely modify.

@pbanakar

pbanakar commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Hi @wuchong addressed all comments, ready for another look, Thanks 🙏

pbanakar and others added 3 commits August 9, 2026 12:31
Construct bitmap inputs with rb_build in the aggregate function and rbm32 examples.

Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-5.6-sol
AI-Contributed/Feature: 35/35
AI-Contributed/UT: 0/0
Document qualified function references and add a windowed bitmap aggregation example backed by the rbm32 merge engine.

Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-5.6-sol
AI-Contributed/Feature: 86/86
AI-Contributed/UT: 0/0

@wuchong wuchong left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @pbanakar , I added 2 commits to improve the SQL examples. LGTM.

@wuchong
wuchong merged commit 4259349 into apache:main Aug 9, 2026
1 check passed
wuchong pushed a commit that referenced this pull request Aug 9, 2026
…3823)

AI-Contributed/Feature: 0/317
AI-Contributed/UT: 0/0
@pbanakar
pbanakar deleted the FIP37-docs branch August 9, 2026 10:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants