feat(storage): support run-length encoding

[ludics]

storage: support run-length encoding #255

• add 3 BlockType with prefix Rle
• add rle_block_builder, rle_block_iterator
• update corresponding ColumnBuilder and ColumnIterator

close #255

Signed-off-by: Lu Di leonludics@gmail.com

[yingjunwu]

It's called run-length encoding (RLE), not running-length encoding.

[skyzh]

Thanks a lot for this feature! But before proceeding to review, I'd like to know why we need 2 separate RLEBuilder / RLEIterator for variable / fixed-length types? If we write bounds from the Array:

pub struct RleBlockIterator<A, B>
where
    A: Array,
    B: BlockIterator<A>,

This should cover all cases, including both primitive types and Bytes types. Did you meet any difficulty when implementing in this way?

(... in #255, I only considered the case for primitive types. But as long as you have implemented RLE for all types, I think it's reasonable to de-dup code.)

[skyzh]

By the way, I've force enabled is_rle and run TPC-H tests on this branch. It seems that...

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: TestError { kind: QueryResultMismatch { sql: "select     sum(l_extendedprice * l_discount) as revenue from     lineitem where     l_shipdate >= date '1994-01-01'     and l_shipdate < date '1994-01-01' + interval '1' year     and l_discount between 0.08 - 0.01 and 0.08 + 0.01     and l_quantity < 24;", expected: "163821038.3323", actual: "163823134.1083" }, loc: Location { file: "tests/sql/tpch-full/_q6.slt", line: 1 } }', src/main.rs:217:41

Some queries are producing wrong results. See https://github.com/risinglightdb/risinglight/blob/main/docs/01-tpch.md#developers-add-new-tpc-h-tests on how to run TPC-H tests with RisingLight.

Don't worry about that, I think I might find the root cause when I review.

[xxchan]

This should cover all cases, including both primitive types and Bytes types. Did you meet any difficulty when implementing in this way?

If the answer is yes, I think we'd better choose not to implement it to keep our codebase simple. Because RLE may not be very useful for non-primitive types?

[skyzh]

Because RLE may not be very useful for non-primitive types?

There's a one-char column l_returnflag in lineitem table in TPC-H test. RLE is still helpful for those using char to store enum-like types.

[ludics]

This should cover all cases, including both primitive types and Bytes types. Did you meet any difficulty when implementing in this way?

In commit f163ed, I added a new trait RLETypeEncode and implemented only one builder/iterator for both primitive types and Bytes types. However, the RLEBlockBuilder is not very efficient as the previous_value is stored as Option<Vec<u8>> and the item to be appended will also be transformed to Vec<u8> for primitive types to make a comparison. Storing previous_value as Option<&[u8]> is more efficient, but there will be some lifetime problems...

[skyzh]

We may store previous value as <A::Item as ToOwned>::Owned? This is equivalent to i32, String, etc.

[skyzh]

Rest LGTM, excellent work!

[skyzh]

Rest LGTM! You may get it merged and fix the remaining small problems later. Thanks for your contribution.

I'm still investigating why enabling RLE will produce wrong result in TPC-H. I'll create an issue later about this problem.

[skyzh]

Well, I've finally found the issue with skip. I'll start another issue on this.