destination-async-framework: use the value from stats counter for global state conversion #35111
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…bal state conversion
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
edgao
approved these changes
Feb 9, 2024
| stateManager.decrement(stateId, 2); | ||
| List<PartialStateWithDestinationStats> stateAfterAllRecordsAreFlushed = stateManager.flushStates(); | ||
| assertEquals(1, stateAfterAllRecordsAreFlushed.size()); | ||
| assertEquals(6.0, stateAfterAllRecordsAreFlushed.get(0).stats().getRecordCount()); |
There was a problem hiding this comment.
checking my understanding: without this PR, we would get 0 instead of 6 here?
There was a problem hiding this comment.
nope, we would get 2
gisripa
approved these changes
Feb 10, 2024
octavia-squidington-iii
added
area/connectors
|
/publish-java-cdk
|
octavia-squidington-iii
added
the
area/documentation
xiaohansong
pushed a commit
that referenced
this pull request
Feb 13, 2024
…bal state conversion (#35111)
jatinyadav-cc
pushed a commit
to ollionorg/datapipes-airbyte
that referenced
this pull request
Feb 21, 2024
…bal state conversion (airbytehq#35111)
jatinyadav-cc
pushed a commit
to ollionorg/datapipes-airbyte
that referenced
this pull request
Feb 26, 2024
…bal state conversion (airbytehq#35111)
jatinyadav-cc
pushed a commit
to ollionorg/datapipes-airbyte
that referenced
this pull request
Feb 26, 2024
…bal state conversion (airbytehq#35111)
jatinyadav-cc
pushed a commit
to ollionorg/datapipes-airbyte
that referenced
this pull request
Feb 26, 2024
…bal state conversion (airbytehq#35111)
xiaohansong
pushed a commit
that referenced
this pull request
Feb 27, 2024
…bal state conversion (#35111)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No issue for this. The GlobalAsyncStateManager assumes by default that the state message is by default PER-STREAM until an actual state message arrives. Meanwhile as records arrive, they are also being flushed and the record counter is decremented as records are flushed. Once a state message finally arrives and is identified as GLOBAL state, the previous counters need to be converted to GLOBAL type. We had a bug in conversion where we were using the decrementing counter instead of the stats counter.