Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Suggestion: provide a maxWait param for cases where flush isn't enough #1

Open
klayhb opened this issue Oct 1, 2020 · 1 comment
Open

Comments

@klayhb
Copy link

klayhb commented Oct 1, 2020

So, I was using this code (and thank you for making it) to convert a read Kafka stream into batches (to reduce concurrent load on the downstream destination) -

The issue is that it seems like some ReadableStreams don't actually ever invoke flush, making the strictMode irrelevant
as a replacement, i added a "maxWait" parameter that records the timestamp of the last update, and upon receiving a put into the BatchStream, checks whether the required batch size has been reached OR the maxWait has elapsed since the last put.

This way, there is an occasional internal "flushing" of the stream without waiting for an externally invoked flush...

wdyt?

@roccomuso
Copy link
Owner

Can you paste code that replicate the issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants