Skip to content

v1.2.966-nightly

@SkyFan2002 SkyFan2002 tagged this 19 Nov 01:42
* feat(storage): read blocks stored in column groups

A split block stores its columns in several group files. Readers took
`location` and `col_metas` as the only file, so a split block, whose
`col_metas` is empty, read every column as its default value.

BlockMeta::data_files, column_chunk and column_metas describe where each
column of a block lives. FuseBlockPartInfo carries the group files a part
reads. Merge IO reads the files of a block at the same time, and
deserialization decompresses each chunk with the codec of its file and
caches it under the path of its file.

Scan, prewhere, row fetch, mutation source, compact, MERGE INTO, REPLACE
INTO, analyze, the low-level recluster reader, range pruning and
fuse_column/fuse_page/fuse_encoding read split blocks. Range pruning
looked up col_metas to decide whether a block lacks a column, and would
prune a split block on a column without statistics.

Virtual column refresh, table index refresh and column-oriented segments
reject split blocks. Decoding segment block metas checks the column
group invariants.

* ci(storage): integration tests for reading split blocks

Forge split blocks by moving columns of every block into group files
with their own codecs, and compare scans, range pruning, UPDATE, DELETE,
MERGE INTO, REPLACE INTO, compact, recluster, analyze and the fuse_*
table functions with a reference table of legacy blocks.

* ci(storage): drop trailing blank line in column group tests
Assets 2
Loading