Be adaptive to stream speed

[shawncao]

Today, for real time streaming data source, such as Kafka, nebula create each data block based on offset start and end, and seal a data block when all records arrived, then put this sealed block into queryable blocks pool. Query will scan blocks that are in the pool.

For busy streams, this may not be an issue, as the system will place data blocks very often. But for slow streams, it may wait for a few minutes to generate a sizable batch, our current solution is to decrease the batch size for that stream, this will end up with more blocks to manage and not adaptable to stream speed, some traffic speeds up and down in different period.

The issue is to ask improvement to make ingestion adaptive to stream speed and still maintain ideal block size, options are

• keep latest block open for both query and append
• make a copy of a progressive block until it's sealed.
• open for other designs