Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

StreamManager.updateStream should scale stream to uphold minumum number of segments (or fixed number of segments) #2887

Closed
shiveshr opened this issue Sep 4, 2018 · 0 comments · Fixed by #6018
Closed

StreamManager.updateStream should scale stream to uphold minumum number of segments (or fixed number of segments) #2887

shiveshr opened this issue Sep 4, 2018 · 0 comments · Fixed by #6018
Assignees
@shiveshr
Labels
area/controller kind/bug Correctness issue kind/enhancement Improvements that should be made priority/P1 status/in-progress version/0.9.1 version/0.10.0

Comments

@shiveshr
Copy link
Contributor

shiveshr commented Sep 4, 2018 •
edited

Problem description
Currently updating minimum number of segments for a stream results in scaling policy update in stream configuration.
The policy enforcement happens during subsequent scale requests. This means if a stream had lesser number of segments than the minimum number of segments specified in the policy, then it will continue to remain below the newly configured value.

We should evaluate if we should automatically trigger a scale up in response to updateStream request to change the number of segments in the stream to at least meet minimum number of segments requirement.

Similarly, for a fixed sized stream, if the minNumberOfSegments are changed, the stream should automatically change the partitioning.

Problem location
UpdateStreamTask

Suggestions for an improvement
Scale the stream to minimum number of segments during update workflow.
@shiveshr shiveshr self-assigned this Sep 4, 2018
@kmgowda kmgowda mentioned this issue Sep 10, 2018
Closed
@shiveshr shiveshr changed the title Evaluate semantics of StreamManager.updateStream for updating minumum number of segments StreamManager.updateStream should scale stream to uphold minumum number of segments (or fixed number of segments) May 13, 2021
@shiveshr shiveshr mentioned this issue May 13, 2021
Merged
This was referenced May 14, 2021
Closed
Merged
sachin-j-joshi pushed a commit that referenced this issue May 14, 2021
@shiveshr
Issue 6026: Cherry pick fix for 2887 in r0.9 (#6027)
2e6112e 
Cherry pick #2887 into r0.9.

Applies configuration change and enforces minimum number of segments. Scales stream if the stream has lesser number of segments (for auto scaling) and different number of segments (for fixed policy).

Signed-off-by: Shivesh Ranjan <shivesh.ranjan@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@shiveshr shiveshr

Labels
area/controller kind/bug Correctness issue kind/enhancement Improvements that should be made priority/P1 status/in-progress version/0.9.1 version/0.10.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Issue 2887: AutoScale stream if configuration changes the minNumberOfSegments shiveshr/pravega-1
3 participants
@fpj @shiveshr @kevinhan88