ART: inline row IDs into node pointers

[taniabogatsch]

Overview

This PR supports inlining row IDs into node pointers if a leaf only contains a single row ID, which is in most cases.

Previously, we would create a Leaf node of 16 bytes, even if we only wanted to store a single row ID in that leaf. Now, we keep that row ID directly in the node pointer to the leaf, removing the need for an additional leaf node. For example, for an ART on 100k unique integers, we previously allocated 100k leaves, i.e., 1,600,000 bytes = 1.6MB. Thus, with this PR, we save 1.6MB. Inlining row IDs also increases the performance of index operations as we traverse one fewer node (the Leaf).

Memory and performance improvements

Here are some numbers on memory improvements (tests taken from test/sql/index/art/memory/test_art_non_linear.test_slow). For 100k values, we save ~1.8MB by inlining (because we allocate blocks of 256KB). For many duplicates, we still need to allocate leaves holding the row IDs so we do not gain any memory improvements.

Workload	feature [in MB]	PR [in MB]
100k short strings	6	4.2
100k long strings	6.8	5
100K mostly distinct BIGINT keys	4.2	2.4
100K mostly duplicate INTEGER keys	2.1	2.1

The performance of index operation (slightly) increases for many workloads, specifically for constraint checking; here are a few examples (taken from benchmark/micro/index/).

Workload	feature [in s]	PR [in s]
lookup_art_constraint	2.04	1.50
create_art	0.71	0.56
create_art_random	0.52	0.50
create_art_varchar_long	0.26	0.23
create_art_varchar_short	0.17	0.15

Next steps

There are still many issues of people running out of memory during CREATE INDEX operations (e.g., #8066, #7760, ...). This is most likely due to the overhead in memory allocation in the Sink calls of PhysicalCreateIndex. I will try to address this issue next. After, I will continue with the improvements documented in #5865.

Implementation details

Leaves

The size of the Leaf node increased from 16 bytes to 48 bytes. That is because we now store up to four row IDs in that leaf, a pointer to a consecutive leaf, and a count. If we need to store more than four row IDs in a leaf, than we will end up with a chain of leaf nodes, similar to our implementation for prefix nodes.

uint8_t count;
row_t row_ids[Node::LEAF_SIZE];
Node ptr;

Node pointers

0: serialized flag
1 - 7: type (set, if not serialized)
8 - 23: offset, 24 - 63: buffer ID (if not serialized) / block ID (if serialized) OR 
8 - 63: row ID

A Node pointer consists of 64 bits. Previously, we were using bit fields for our Node class. This approach became more complicated because we either store offset + buffer/block ID in the last 56 bits, or the row ID. Additionally, compiling Windows in a way that respects the bit field constraints is not trivial (here). Therefore, we would end up with 16 bytes on Windows machines instead of the intended 8 bytes. Thus, we decided to perform all bit shifting/AND/OR operations ourselves. The Node class contains the respective constexprs and the Getters and Setters for the different fields.

Maximum row IDs

DuckDB uses temporary row IDs for local changes during transactions. Because we inline the row ID into 56 bits, we adjust the internal maximum numbers for row IDs.

const row_t MAX_ROW_ID = 36028797018960000ULL;       // 2^55
const row_t MAX_ROW_ID_LOCAL = 72057594037920000ULL; // 2^56

Other changes

• renamed swizzled to serialized
• renamed GetARTNodeType to GetType
• less function-call overhead during vacuum operations
• more tests for code coverage/edge cases, more assertions
• no more separate SwizzleablePointer class
• close [ART] RowID Leaf node specialization #5365

[pdet]

Great PR and results!

I have two small comments

1. Should we add a test that stresses the chaining of the leaf? As in inserts one row, stores, restarts, query, query, insert another row (now not inlined but actually a leaf node), repeat until getting a chain, then remove all the way back?
2. About the chaining, it can potentially add a lot of random access to data that has a lot of repetition. What about dynamic resizing? The disadvantage ofc of resizing is that it can become a peak bottleneck if resizing big arrays, so maybe a combination of both? :-)

[taniabogatsch]

@pdet, thanks for having a look! :) I will add that stress test!

About the dynamic resizing, I just remembered why that wasn't feasible haha. We have fixed-size allocators for fast node allocations, and they require fixed-size nodes. So different/dynamic node sizes for prefixes/leaves are impossible if we want their memory to be managed by these allocators. And allocating/managing their memory separately is (I think) not worth it/adds too much complexity.

[pdet]

@pdet, thanks for having a look! :) I will add that stress test!

About the dynamic resizing, I just remembered why that wasn't feasible haha. We have fixed-size allocators for fast node allocations, and they require fixed-size nodes. So different/dynamic node sizes for prefixes/leaves are impossible if we want their memory to be managed by these allocators. And allocating/managing their memory separately is (I think) not worth it/adds too much complexity.

Oh yeah, that makes 100% sense.

[Mytherin]

Thanks!