Column value details

Column value representation

Implementation reference checked against f5e786f. Fundamental explanations remain in docs/arch.

io.h defines TColumn and TRowSet.

Column field	Meaning
Data	Fixed-width values, packed booleans, or variable-width bytes
DataBitOffset	Offset for bitmap-backed values
Mask / MaskBitOffset	Validity bitmap and its bit offset; null mask means all valid
Offsets / OffsetWidth	String offsets; width 4 or 8, zero for fixed-width

A rowset carries column/count metadata, selection, optional precomputed hashes, and owner/destruction fields. A hash is defined only for selected rows. Column materialization centralizes conversion to kernel values; Qumir managed strings cannot be substituted for byte views.

Generated type helpers

The AST generator recursively produces helpers for the concrete pair of types:

rh_hash(LookupKey) -> i64
rh_hash(StoredKey) -> i64
rh_key_equal(StoredKey, LookupKey) -> bool
rh_key_equal(StoredKey, StoredKey) -> bool
key_owned_bytes(LookupKey) -> i64
key_clone_owned(LookupKey, <ptr u8>) -> StoredKey

Both hash overloads must use the same byte hash and struct combiner. Equality must compare lengths first and then bytes. Composite helpers recurse through nested fields and ignore generated padding.

key_clone_owned is recursive and type-directed. Fixed-width leaves are copied directly. String leaves copy bytes into successive slices of one caller-provided block and construct OwnedString handles pointing into those slices.

Hashing changes require compatibility and performance checks; lookup and stored representations must keep equal hashes for equal logical values. Hash values are process-local table details; they are not persisted.

Owned byte storage

Each new group receives one contiguous byte slice containing all string leaves of its key. Generated clone code sets each OwnedString.Data into that slice.

Slices come from a table-owned chunked arena. Chunks start at 4 KiB and grow geometrically up to 2 MiB; a request larger than that receives an exactly sized chunk. Each chunk has a 24-byte {magic, capacity, used} header and allocations bump the used offset. The magic distinguishes arena chunks from foreign or legacy registry entries before their allocator fields are read; a mismatch fails allocation without modifying the registry. This turns the allocation cost from one allocation per unique string-bearing group into roughly one allocation per arena chunk.

Only the most recent chunk accepts new slices. If a request does not fit, its remaining tail is not revisited; this keeps allocation constant-time at the cost of up to roughly 2x fragmentation for adversarial sizes. Ordinary short string keys leave much smaller tails.

The table owns a registry of chunk bases. aht_destroy delegates its cleanup to the shared owned_arena_destroy lifecycle helper, so allocation ownership has one destruction path across aggregation and join kernels. Rehash copies OwnedString handles and never clones or frees their bytes. If insertion fails after allocation, the most recent slice is rewound; an empty chunk may remain registered for reuse.

The registry reuses the three previously unused 8-byte scratch slots in HashTable: OwnedBlocks, OwnedBlockCount, and OwnedBlockCapacity at offsets 40, 48, and 56. The table ABI therefore remains 104 bytes.

Allocation must happen only after borrowed lookup reports a miss. A failure during allocation or insertion must leave the table logically unchanged and return an error.