-
Notifications
You must be signed in to change notification settings - Fork 466
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
storage: write status updates in the controller, and add the "paused" status #23222
storage: write status updates in the controller, and add the "paused" status #23222
Conversation
c216f05
to
b8f148d
Compare
This PR has higher risk. Make sure to carefully review the file hotspots. In addition to having a knowledgeable reviewer, it may be useful to add observability and/or a feature flag. What's This?
Buggy File Hotspots:
|
436a008
to
de6cfae
Compare
Test coverage looks good (build: https://buildkite.com/materialize/coverage/builds/317). |
a1609ba
to
24ef729
Compare
24ef729
to
7b90226
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks much better! A few more comments but that's almost ready to merge
7b90226
to
107fc3c
Compare
@petrosagg I think I resolved the last of your comments! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great, thanks!
107fc3c
to
c18c7cc
Compare
Implements #19792
@rkrishn7 thank you for you work on #20735 ! I have rebased and adapted that code here.
This pr shifts where we write status updates for sources and sinks into the storage controller; this reduces contention on those history shards, but also allows us to easily mark sources and sinks as "paused" when there exists no replica (note that we could have done this in a more hacky way, but chose not to)
this pr attempts to make the
paused
status as understandable as possible, but some edge cases will be improved when #23013 is implemented.Motivation
Checklist
$T ⇔ Proto$T
mapping (possibly in a backwards-incompatible way), then it is tagged with aT-proto
label.