Tracking: bitmap functions

[ariesdevil]

Summary
Scalar Functions:

• to_bitmap feat: add bitmap data type #11097
• bitmap_from_string feat: add bitmap data type #11097
• bitmap_to_string feat: add bitmap data type #11097
• bitmap_count feat: add bitmap_count function #11304
• build_bitmap feat: add bitmap_count function #11304
• bitmap_or
• bitmap_and
• bitmap_xor
• bitmap_not
• bitmap_and_not
• bitmap_subset_limit
• bitmap_subset_in_range
• sub_bitmap
• bitmap_contains
• bitmap_has_all
• bitmap_has_any
• bitmap_max
• bitmap_min

Aggregate Functions:

• bitmap_union
• bitmap_and_count
• bitmap_and_not_count
• bitmap_xor_count
• bitmap_or_count
• intersect_count
• bitmap_interssect

[byrantwithyou]

Hi, I'm new to Databend and I want to contribute. I have a working knowledge of database kernel and Rust. Could you assign some of the tasks in this list to me?

[byrantwithyou]

Hi, I'm new to Databend and I want to contribute. I have a working knowledge of database kernel and Rust. Could you assign some of the tasks in this list to me?

If nobody is doing the same thing, I want to do bitmap_or, bitmap_and, bitmap_xor, bitmap_not, bitmap_and_not together first, because they are kinda similar.

[ariesdevil]

@byrantwithyou Thanks for being interested in these funcs, When things are done, I'll put the pr address after the corresponding function.

[silver-ymz]

Hello, I'm new to Databend and want to have a try for this. I want to try bitmap_contains, bitmap_has_all, bitmap_has_any, bitmap_max, bitmap_min together firstly. Could you assign them to me?

[byrantwithyou]

I'm so sorry that I couldn't do this task because I'm a little bit busy these days. Please assign it to anyone else.

[gitccl]

Hi, I'd like to try bitmap_or, bitmap_and, bitmap_xor, bitmap_not, bitmap_and_not.

[gitccl]

Hi @sundy-li , Can bitmap_hash use ahash as the hash function?

[sundy-li]

Can bitmap_hash use ahash as the hash function?

Yes， I don't think we should add the extra function bitmap_hash , we can support bitmap type in other hash functions.

[gitccl]

It seems that bitmap_hash isn't the hash of bitmap type, instead it hashes any value, and then constructs a bitmap with the hash value.

[liangjiawei1110]

Hello, I am new to Databend and I'd like to try bitmap_subset_limit , bitmap_subset_in_range , sub_bitmap.

[ariesdevil]

Hello, I am new to Databend and I'd like to try bitmap_subset_limit , bitmap_subset_in_range , sub_bitmap.↳

Hi @liangjiawei1110 , these three have been implemented by @Mehrbod2002 , you may take others:)

[akoshchiy]

@ariesdevil Hello! I want to complete the list and have some questions:

1. As I understand, intersect_count is the alias of bitmap_and_count, isn't it?
2. Is there any explanation/examples of api and behaving orthogonal_* functions?

[ariesdevil]

Hi @akoshchiy , thanks for your contribution.

1. As I understand, intersect_count is the alias of bitmap_and_count, isn't it?

The intersect_count is diff from bitmap_and_count, the syntax is INTERSECT_COUNT(bitmap_column, column_to_filter, filter_values).

1. bitmap_column: The bitmap column. This is the bitmap data you want to perform intersection operations on.
2. column_to_filter: The dimension column for filtering. This is the column you want to filter the bitmap data based on certain conditions.
3. filter_values: The values of the filter dimension column. This variable-length parameter represents different values of the dimension column to filter.

The purpose of the INTERSECT_COUNT function is to first filter the bitmap data based on the column_to_filter and filter_values, then calculate the intersection count between the filtered bitmaps. This can help you find the number of data points satisfying specific conditions within a larger dataset.

bendsql :) select tag, bitmap_to_string(user_id) from pv_bitmap where tag in (3, 4);
+------+-----------------------------+
| tag  | bitmap_to_string(`user_id`) |
+------+-----------------------------+
|    4 | 1,2,3                       |
|    3 | 1,2,3,4,5                   |
+------+-----------------------------+
2 rows in set (0.012 sec)

bendsql :) select intersect_count(user_id, tag, 3, 4) from pv_bitmap;
+----------------------------------------+
| intersect_count(`user_id`, `tag`, 3, 4)|
+----------------------------------------+
|                                      3 |
+----------------------------------------+
1 row in set (0.014 sec)

2. Is there any explanation/examples of api and behaving orthogonal_* functions?

The orthogonal_* prefixed functions are just same as the original functions without this prefix, except using a different approach to calculation, we not need them now, so I deleted them.

[ariesdevil]

We made it, thank you all!