Type FlatBuffers

[dominiklohmann]

📔 Description

This refactors our type system to be based on a single, extensible FlatBuffers table. This comes with multiple advantages:

• type is now cache friendly. It's probably safe to say that this was the biggest downside of the old type system. Iterating over the leaves of a a record_type is no longer significantly slower than iterating over just the fields.
• type is now sliceable from other FlatBuffers table (e.g., Table Slices), and shares its lifetime with the sliced-from underlying chunk. type must no longer be deserialized as its underlying data format is a contiguous buffer now.

I've taken the opportunity to clean up the type API at the same time. This includes the following developer-facing changes:

• The alias_type no longer exists. Visiting a type steps through aliases transparently.
• Attributes are now called tags. (reverted late on)
• The record_type API no longer operates with keys directly, but rather only with indices and offsets. The functions record_type::resolve_prefix and record_type::resolve_suffix must be used to convert a key into offsets.
• The enumeration_type API allows for explicitly assigning keys to individual fields. This is preparatory work for aligning our data model with Apache Arrow's, which offers the related arrow::DictionaryType.
• The functions record_type::fields and record_type::leaves return iterables over fields and leaves of a record respectively.
• Complex types are no longer default-constructible.
• The new concrete_type, basic_type, and complex_type concepts make writing generic visitors a lot easier.

There exists conversions between types and legacy type in both directions to ease the transition. The conversion from type to legacy type will be removed towards the end of this PR.

The user-facing changes of this PR are hopefully minimal. The conversion from legacy_type to type when exporting old data from pre-existing databases may incur a small performance overhead, but for newly imported data this should speed up both the import and export pipelines across the board. There are a lot of smaller bug fixes for edge cases with leaf field key handling and alias type resolution throughout the code base.

🚦 Status

• Design the new type FlatBuffers table and the basic type API.
• Add support for basic types.
• Add support for type aliases and tags.
• Add support for list types.
• Add support for map types.
• Add support for enumeration types.
• Add support for record types.
• Implement new versions of encoding-specific table slice FlatBuffers tables.
• Convert libvast to operate on new type system directly.
• Convert unit tests, tools, and plugins to operate on new type system directly.
• Remove now dead code from the legacy type API (most operations should be unnecessary now).
• Consider implementing a new version of the partition FlatBuffers table. Not in this PR.
• Consider making record_type::iterable::iterator a random access iterator. Not in this PR.
• Consider making record_type::leaf_iterable::iterator a bidirectional iterator. Not in this PR.

📝 Checklist

• All user-facing changes have changelog entries.
• The changes are reflected on docs.tenzir.com/vast, if necessary.
• The PR description contains instructions for the reviewer, if necessary.

🎯 Review Instructions

1. Look at the new Type FlatBuffers table and at the new type.hpp file.
2. Look at the individual commits. Most of them should be self-explanatory.
3. Wait until this PR is out of draft mode.
4. Measure overall performance impact on both pre-existing and newly created databases.

[tobim]

This branch has been running on a testbed at Tenzir for the past 15 hours with no unexpected performance changes.

The time to load a partition with the old serialization format has increased slightly increased, but we expect that to return back to the old numbers once the partition uses the new serialization format for its layout.