Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

server,ui: Add debugging for quiesced ranges #26269

Merged
merged 1 commit into from
May 31, 2018

Conversation

bdarnell
Copy link
Contributor

I suspect that #26257 is caused by the unquiescedReplicas map
introduced in #24956 getting out of sync with the per-replica
quiescent flag. Add debug pages to help us see if that's happening.

Release note: None

@bdarnell bdarnell requested review from andreimatei and a team May 31, 2018 02:32
@cockroach-teamcity
Copy link
Member

This change is Reviewable

@tbg
Copy link
Member

tbg commented May 31, 2018

LGTM, but you need to update replica_test.go.

@@ -8672,7 +8672,7 @@ func TestReplicaMetrics(t *testing.T) {
metrics := calcReplicaMetrics(
context.Background(), hlc.Timestamp{}, config.SystemConfig{},
c.liveness, &c.desc, c.raftStatus, LeaseStatus{},
c.storeID, c.expected.Quiescent, CommandQueueMetrics{}, CommandQueueMetrics{})
c.storeID, c.expected.Quiescent, !c.expected.Quiescent, CommandQueueMetrics{}, CommandQueueMetrics{})
if c.expected != metrics {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused, shouldn't c.expected need Ticking set on it as well?

@@ -449,6 +453,7 @@ export default class RangeTable extends React.Component<RangeTableProps, {}> {
problems: this.contentProblems(info.problems, awaitingGC),
raftState: raftState,
quiescent: info.quiescent ? rangeTableQuiescent : rangeTableEmptyContent,
ticking: this.createContent(info.ticking.toString()),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be nice to roll this in with the quiescent row. I'm imagining we'd show quiescent and empty as normal states and "quiescent but ticking" and "not ticking but not quiescent" as error states.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I just wanted to get something in quickly for cyan without having to fiddle with and test all the combinations.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine with that, we can just have @BramGruneir come through and add some niceties here.

I suspect that cockroachdb#26257 is caused by the unquiescedReplicas map
introduced in cockroachdb#24956 getting out of sync with the per-replica
quiescent flag. Add debug pages to help us see if that's happening.

Release note: None
@BramGruneir
Copy link
Member

:lgtm: Just with @couchand's nits.


Review status: all files reviewed at latest revision, 2 unresolved discussions, some commit checks pending.


Comments from Reviewable

@bdarnell
Copy link
Contributor Author

bors r+

@andreimatei
Copy link
Contributor

:lgtm:


Review status: all files reviewed at latest revision, 2 unresolved discussions, some commit checks pending.


pkg/server/serverpb/status.proto, line 142 at r2 (raw file):

  // If the replica's quiescent flag doesn't agree with the store's
  // list of replicas that are ticking, warn about it.
  bool quiescent_equals_ticking = 6;

Even with the comment I was confused about whether the true value is the good one or not. How about calling it ticking_disagreement and also adding a quiescent field to determine which way the disagreement goes.


Comments from Reviewable

craig bot pushed a commit that referenced this pull request May 31, 2018
26269: server,ui: Add debugging for quiesced ranges r=bdarnell a=bdarnell

I suspect that #26257 is caused by the unquiescedReplicas map
introduced in #24956 getting out of sync with the per-replica
quiescent flag. Add debug pages to help us see if that's happening.

Release note: None

Co-authored-by: Ben Darnell <ben@cockroachlabs.com>
@craig
Copy link
Contributor

craig bot commented May 31, 2018

Build succeeded

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants