Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update data-structure-on-rocksdb.md for Hyperloglog #207

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Update data-structure-on-rocksdb.md for Hyperloglog #207

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5af4eec
Update data-structure-on-rocksdb.md for Hyperloglog
tutububug Mar 8, 2024
53c56f9
Update data-structure-on-rocksdb.md for hyperloglog
tutububug Mar 8, 2024
2748009
HLL: update storage format
tutububug Apr 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions community/data-structure-on-rocksdb.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ The values encoded for other data types in flags can be found in the table below
| Stream | 8 |
| BloomFilter| 9 |
| JSON | 10 |
| Hyperloglog| 12 |

In the encoding version `0`, `expire` is stored in seconds and as a 4byte field (32bit integer), `size` is stored as also a 4byte field (32bit integer);
while in the encoding version `1`, `expire` is stored in milliseconds and as a 8byte field (64bit integer), `size` is stored as also a 8byte field (64bit integer).
Expand Down Expand Up @@ -314,3 +315,27 @@ where the `payload` is a string encoded in the corresponding `format`:
| CBOR | 1 |

Also, if we decide to add a more IO-friendly format to avoid reading all payload to the memory before searching an element via JSONPath or seperate a relatively large JSON to multiple key-values, we can take advantage of the `format` field.

## Hyperloglog

Redis hyperloglog can be thought of as a static array with a length of 16384. The array elements are called registers, which are used to store the maximum count of consecutive 0s. This register array is the input parameter for the hyperloglog algorithm.
In Kvrocks, the hyperloglog data structure is stored in following two parts:

#### hyperloglog metadata

```text
+----------+------------+-----------+-----------+
key => | flags | expire | version | size |
| (1byte) | (Ebyte) | (8byte) | (Sbyte) |
+----------+------------+-----------+-----------+
```
#### hyperloglog sub keys-values
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you mind try Bitmap structure? "bitfield" command might help in this case?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had changed my previous implementation of the storage format, currently, the format is the same as the bitmap's, and each HLL subkey represents a segment with 1024 registers.
Also, I tried to use bitfield, but finally, I found that it requires byte alignment because of 'memcpy' (it should only be able to operate bits within a byte). The implementation of Redis only uses 6 bits, not aligned, to save more space, so it seems not suitable.
However, another problem arises. When the register is set to 6-bit, unit test runs fail on a few OS releases in CI, but there is no problem when it is set to 8-bit, maybe it has something to do with byte alignment, but I don’t know the reason yet. Therefore, the current implementation still maintains 8 bits without using bitfield.
If I understand something wrong, correct me, thanks.


```text
+-----------------------+-----+
key|version|register_index => | 0s count (1byte) | ... |
+-----------------------+-----+
```
The register index is calculated using the first 14 bits of the user element's hash value (64 bits), which is why the register array length is 16384.
The length of consecutive zeros is calculated using the last 50 digits of the hash value of the user key.
Inspired by the bitmap implementation, hyperloglog divides the register array into 16 segments, each with 1024 registers.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So it's now not actually register_index? But something like segment index.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it is register_index, the same as the bitmap segment in which the 1st segment subkey is 'subkey0', the 2nd is 'subkey1024', and so on.

Loading