-
Notifications
You must be signed in to change notification settings - Fork 69
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bug: If one stream has an error, then all streams fail and no STATE messages are ever sent #1064
Comments
I can mention that I'm using a |
I propose changing the default |
@edgarrmondragon and @kgpayne thoughts here since you've been working on BATCH support and implementation |
@tayloramurphy this is independent from @spacecowboy what's the target you're working with and are you able to override the value there? It might be a good idea to get heuristics or make the target batch size and wait limit configurable, similar to |
I have my own But in my opinion it just makes sense for the default value to be a lot smaller. |
@spacecowboy - re:
Yes, this sounds like a good approach to me as well. 10 or 15 minutes is not often enough that we're spending a large percentage of time in the batch flush operations. Even 5 minutes might be okay here. If we find that the max record age is not triggering as expected, that would be a separate bug we'd want to also address.
Nm. I see above you've already confirmed. Yes, I'm in favor. 👍 |
@aaronsteers what's your view on priority here? Would love a weight estimate as well. |
@tayloramurphy and @spacecowboy - some scenarios affected here would be resolved by this which is merged and soon-to-be released: The relative priority for this would be low, since all is functioning as designed - or will be when the above releases. That said, I've added the Weight is probably |
See #1192 |
Even better with the PR @spacecowboy 😄 @aaronsteers I've added the community contributed PR label and added it to the engineering board as well. |
Singer SDK Version
0.11.1
Python Version
3.9
Bug scope
Targets (data type handling, batching, SQL object generation, etc.)
Operating System
Linux
Description
Scenario
Sink
Target
and allSink
s stop.What is the actual result
Not a single
STATE
message is emitted by the target during this process even though it might have spent hours processing data already. Which means you need to start over from scratch. For ALL streams.What is the expected result
The
TARGET
orSINK
s should emit state messages periodically so state is not lost.Code
No response
The text was updated successfully, but these errors were encountered: