SAMZA-2743: [Elasticity] Add keybucket into SSP serde for checkpoint#1608
Merged
rayman7718 merged 2 commits intoapache:masterfrom May 27, 2022
Merged
SAMZA-2743: [Elasticity] Add keybucket into SSP serde for checkpoint#1608rayman7718 merged 2 commits intoapache:masterfrom
rayman7718 merged 2 commits intoapache:masterfrom
Conversation
mynameborat
pushed a commit
that referenced
this pull request
Aug 2, 2022
…1625) Symptom: Rolling back to versions which dont accept 4 parts in checkpoint serde can throw NPE exceptions. Cause: As part of elasticity, #1608 introduced keyBucket into checkpoint serde. Fix: Remove keyBucket from checkpoint serde - aka do not add it when serializing checkpoint. Backwards Compatible: yes. Though checkpoints written after #1608 and before this PR will have 4 part SSP in checkpoint, serde, they can still be read by code in this PR and vice versa. elasticity will not work completely once this pr is merged.
lakshmi-manasa-g
added a commit
to lakshmi-manasa-g/samza
that referenced
this pull request
Aug 4, 2022
mynameborat
pushed a commit
that referenced
this pull request
Aug 4, 2022
Symptom: Broadcast input ssp is not consumed by all containers of the job. Cause: Elasticity code changing systemconsumers and samza-core Changes: reverting all elasticity commits post SAMZA-2719: [Elasticity] fix container level metrics when elasticity is enabled #1585 after which this issue was detected. updating the checkpointv1 serde to accept checkpoints written with SAMZA-2743: [Elasticity] Add keybucket into SSP serde for checkpoint #1608 list of elasticity PRs being reverted - #1625 #1610 #1608 #1607 #1603 #1598 #1597 #1596 #1589
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Feature: Add KeyBucket into SSP serde for checkpoint. Elasticity (SAMZA-2687) allows consumption of part of SSP denoted by KeyBucket (introduced in #1576). This PR is to add the keybucket into SSP serde for checkpointv1 and when SSP is key in checkpointv2
changes:
Tests: updated and added new tests
API Changes: no public API changes
Upgrade/usage instructions: None
Backwards compatible: yes, new test added uses old serde and new serde together