Use dedicated partitions for each layout

[tobim]

This makes it possible to use homogeneous partitions, so only the data from one particular event type ends up in a partition.

Requires #2120!

📝 Checklist

• All user-facing changes have changelog entries.
• The changes are reflected on docs.tenzir.com/vast, if necessary.
• The PR description contains instructions for the reviewer, if necessary.

🎯 Review Instructions

[mavam]

I'd be curious to benchmark this. If I get it right, it introduces concurrency as function of unique schemas/layouts.

[dominiklohmann]

We talked about this briefly in a call earlier, and I wanted to write my thoughts down somewhere: This is mostly done, but some important adjustments are yet missing.

• The memory usage for this is through the roof. We need to scale down the default partition size.
• Queries currently go to all active partitions, which is just unnecessary now. We probably should register active partitions with the meta index catalog, although I'm not quite sure if that should be part of this PR. It's not critical, but definitely warrants a follow-up story that we tackle in the short- to medium-term.
• The biggest issue in practice when using this is low-volume types never being flushed to disk, because the partition size is set globally. We need some timeout parameter after which we close an active partition, similar to how we have a both a batch-timeout and a batch-size for the table slices produced in the sources.

[mavam]

This makes sense. Regarding memory usage, I want to raise the concern of slicing out types from record batches that keep alive large chunks of memory while only a small fraction is ever needed. This is the detach discussion we had the other day. My intuition is that this is now amplified. I'd like to figure out a way to measure it.

[dominiklohmann]

@mavam We already detach the schema in table slices, so this is not a concern.

[tobim]

We probably should register active partitions with the meta index catalog, although I'm not quite sure if that should be part of this PR. It's not critical, but definitely warrants a follow-up story that we tackle in the short- to medium-term.

I veto this approach. The catalog has no business with active partitions and it must stay that way. We want to be able to run the active and passive parts of the database on different nodes, and such a change would only add communication overhead.
I think the query should get sent to each active partition, but the lookup function should consult the active partition synopsis. This is completely orthogonal though and it should not be done in this PR.

[mavam]

This is completely orthogonal though and it should not be done in this PR.

👆 That. Exactly. Let's not couple things together can operate in a decoupled manner.

For this PR, it's completely fine to not run queries over active partitions (or "partition builders").

[dominiklohmann]

I pushed the changes we discussed, @tobim. I approve of yours earlier in the PR, but would really like to see this PR get reviewed and tested by someone else. @dispanser maybe?

[dominiklohmann]

@tobim I am approving of your changes. If you approve of mine, please go ahead with the merge.

[tobim]

I approve of @dominiklohmann 's changes as well.