SAMZA-2734: [Elasticity] Update last processed offset after an envelopepe is finished processing when elasticity is enabled#1607
Merged
rayman7718 merged 1 commit intoapache:masterfrom May 17, 2022
Conversation
…pe is finished processing when elasticity is enabled
mynameborat
pushed a commit
that referenced
this pull request
Aug 4, 2022
Symptom: Broadcast input ssp is not consumed by all containers of the job. Cause: Elasticity code changing systemconsumers and samza-core Changes: reverting all elasticity commits post SAMZA-2719: [Elasticity] fix container level metrics when elasticity is enabled #1585 after which this issue was detected. updating the checkpointv1 serde to accept checkpoints written with SAMZA-2743: [Elasticity] Add keybucket into SSP serde for checkpoint #1608 list of elasticity PRs being reverted - #1625 #1610 #1608 #1607 #1603 #1598 #1597 #1596 #1589
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Feature: Elasticity (SAMZA-2687) for a Samza job allows job to have more tasks than the number of input SystemStreamPartition(SSP). Thus, a job can scale up beyond its input partition count without needing the repartition the input stream.
This current PR is to update the last processed offsets maintained by the OffsetManager during processing stage of the container correctly when elasticity is enabled.
Changes:
Tests: added unit test
API changes: no public api change
Upgrade instructions: none
Usage instructions: None.
Backwards compatible: yes. does not affect the existing flow