Optimize memory usage for S3StreamMetadataImage.

[superhx]

Who is this for and what problem do they have today?

Why is solving this problem impactful?

In the scenario of a 10w partition, even if a stream object compaction is performed every 1 hour, running for a day will still generate 100000 * 24 stream objects, ultimately occupying at least 100+ MiB of metadata memory. If we consider longer retention time and multiple streams per Partition, the actual memory usage of metadata will be even higher.

Additional notes

[superhx]

Assuming the strategy of stream compaction is to compact the size of each individual object in a stream up to 1GiB, then for a cluster that stores 1PiB of data, the memory consumption per object would be around 50MiB, which is calculated as 1024 * 1024 * (memory usage per object's metadata).

Therefore, the optimization goals have changed to:

• Optimize the memory structure of Image, saving memory overhead caused by data structures like Map.
• Optimize the stream compaction strategy, compacting each stream object to 1GiB and eliminating unnecessary blocks based on the startOffset of the stream during the compaction process.

[superhx]

1PB of data, 10w partitions, how many stream objects will there be in the end?
If it's accumulated in 10GiB batches, then there will be 100000 stream objects.

Simulating 1w => s3streamobject, 100 partitions, write for 100s is sufficient.

Result: 5000 stream objects generated.
5000, multiplied by 3, Controller Image + Broker Image + Controller processing layer

15000 s3streamobjects occupy 1MiB
s3objects occupy 1.5MiB

S3StreamsMetadataImage occupies 1.5MiB
S3ObjectsImage occupies 1.3MiB

Calculating: Estimated memory consumption is 2MiB for every 1w stream objects generated, and 20MiB for 10w stream objects.