-
Notifications
You must be signed in to change notification settings - Fork 66
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
bug: possible regression in state messages and progress_markers handling #1704
Comments
@laurentS - it's possible that this is functioning as designed and that the test should be modified... it will depend on the desired output of the test. Essentially, the key question is whether the test actually did "complete" the steam that is being tested - or if the stream sync was "aborted abnormally" due to max record abort signal. If the latter, then the correct result would be an unfinalized state. If you want to update the stream to deal with abort scenarios cleanly, you can mark the stream as sorted/resumable, and/or you can add handling to deal with abort-type exceptions. Probably there are other ways to address as well, and it's possible that there's another issue going on, but this at least should kick off the conversation... 😁 |
Thanks for clarifying @aaronsteers! My understanding so far:
|
@laurentS @aaronsteers @edgarrmondragon I think the Looking into this now 👍 |
Thanks @kgpayne! |
Yeah, makes sense. Your draft PR #1708 looks like it addresses the issue. 👍 I added a comment there in the PR about some internals, but nothing that would block your PR. |
@edgarrmondragon reassigning this to you since Ken is out. Can you take a look and let me know if we need to make progress on this? |
|
Singer SDK Version
0.23.0
Is this a regression?
Python Version
NA
Bug scope
Taps (catalog, state, etc.)
Operating System
linux
Description
CI checks in MeltanoLabs/tap-github#209 are now failing on a test related to
progress_markers
.I've narrowed down the regression (?) to between sdk
v0.22.1
(pass) andv0.23.0
(fail), and I think it is specifically #1436 which caused the regression. Tagging @aaronsteers as the author of that PR.The test that fails was added to
tap-github
in this PR after this related PR was merged in the SDK. In short,progress_markers
now appears in the last state message issued by thesdk
, which used to signify that the message wasn't "final".It looks like there are some changes in how state is handled, so I'm opening this as a discussion to understand if there was indeed a regression, or if the test in
tap-github
needs to be modified to accomodate the new behaviour in the sdk.Code
No response
The text was updated successfully, but these errors were encountered: