spanconfig: mark reconciliation job as idle #76384
Conversation
|
This change is |
|
Sanity checking, shows up as |
There was a problem hiding this comment.
Change looks good to me. Have you considered the interactions marking this thing as "always idle" will have with the pagination story we hope to address soon? Might be worth filing an issue/capturing thoughts on the pagination issue itself about this.
Reviewable status:
complete! 0 of 0 LGTMs obtained (waiting on @ajwerner, @irfansharif, and @nvanbenschoten)
pkg/spanconfig/spanconfigjob/job.go, line 49 at r1 (raw file):
// The reconciliation job is a forever running background job. It's always // safe to wind the SQL pod down whenever it's running -- something
"-- something"?
Fixes cockroachdb#70538. We have a forever running background AUTO SPAN CONFIG RECONCILIATION job on tenant pods. To know when it's safe to wind down pods, we use the number of currently running jobs as an indicator. Given the job is forever running, we need an indicator to suggest that despite the job's presence, it's safe to wind down. In cockroachdb#74747 we added a thin API to the jobs subsystem to do just that, with the intent of using it for idle changefeed jobs. We just cargo-cult that same approach here to mark the reconciliation job as always idle. Release note: None
irfansharif
force-pushed
the
220210.spanconfig-job-idle
branch
from
09b84eb
to
1f75f55
Compare
There was a problem hiding this comment.
Thanks! I didn't realize there were any implications. Was there something specific you were thinking about?
bors r+
Reviewable status:
pkg/spanconfig/spanconfigjob/job.go, line 49 at r1 (raw file):
Previously, arulajmani (Arul Ajmani) wrote…
"-- something"?
woops, Done.
|
I was thinking about the case where we've got a bunch of IDs from the SQLWatcher to reconcile and the job gets torn down while we're batching through the updates. We could end up in a scenario where zone configuration A was applied before zone configuration B, but zone configuration B makes its way down into KV but zone configuration A does not because the job was shut down. Maybe that's okay given our invariant though. |
|
Build succeeded: |
There was a problem hiding this comment.
Yea, that should be fine -- we're not making any sort of ordering guarantees like that. Besides, if the pod is wound down, I don't think there are any observers left for the "inconsistent" state anyway.
Reviewable status:
|
KV will still be observing the "inconsistent" state, but you're right, we're not making such ordering guarantees so we should be fine. |
Fixes #70538.
We have a forever running background AUTO SPAN CONFIG RECONCILIATION job
on tenant pods. To know when it's safe to wind down pods, we use the
number of currently running jobs as an indicator. Given the job is
forever running, we need an indicator to suggest that despite the job's
presence, it's safe to wind down.
In #74747 we added a thin API to the jobs subsystem to do just that,
with the intent of using it for idle changefeed jobs. We just cargo-cult
that same approach here to mark the reconciliation job as always idle.
Release note: None