Quick/ slow read isolation #471
Description
Now
Because reading data from S3 is slow (> 200ms), Kafka currently does not distinguish between fast and slow reads, and all blocking reads share a single thread pool. When there are a large number of slow reads, it may saturate the thread pool, causing fast reads to be blocked as well.
Expectation
Distinguish between fast and slow reading, and set different concurrency limits for fast and slow reading.
How
- S3Stream adds ReadOption to provide the ability to read slowly and fail fast.
- Kafka first attempts quick reading, and if it fails, it enters the slow reading process.