healthcheck: update healthy tablets correctly when a stream returns an error or times out #7654

deepthi · 2021-03-10T01:47:08Z

Description

There are error conditions from a tablet health check that require the healthy tablet list to be updated.

Tablet host/port is taken over by a different tablet (Fixed in healthcheck: healthy list should be recomputed when a tablet is removed #7176)
Healthcheck stream returns an error (this can be for any reason including tablet being down) VTGate HealthCheckUpdate failing to update vttablet serving status #7472
Healthcheck times out GetHealthyTabletStats is not updated when healthcheck times out #7177
These conditions need to be handled for all tablet types (primary/non-primary)

Related Issue(s)

Fixes #7472
Fixes #7177

Checklist

Should this PR be backported?
Tests were added or are not required
Documentation was added or is not required

Impacted Areas in Vitess

Components that this PR will affect:

Testing

Unit tests were added for each case. For #7472 I also followed the provided testing procedure to reproduce the problem and confirmed that the fix works.

mysql> show vitess_tablets;
+-------+----------+-------+------------+---------+------------------+------------------+----------------------+
| Cell  | Keyspace | Shard | TabletType | State   | Alias            | Hostname         | MasterTermStartTime  |
+-------+----------+-------+------------+---------+------------------+------------------+----------------------+
| zone1 | commerce | 0     | MASTER     | SERVING | zone1-0000000100 | localhost        | 2021-03-10T01:24:33Z |
| zone1 | commerce | 0     | REPLICA    | SERVING | zone1-0000000101 | localhost        |                      |
| zone1 | commerce | 0     | RDONLY     | SERVING | zone1-0000000102 | localhost        |                      |
+-------+----------+-------+------------+---------+------------------+------------------+----------------------+
3 rows in set (0.00 sec)
----
kill rdonly vttablet
----
mysql> show vitess_tablets;
+-------+----------+-------+------------+-------------+------------------+------------------+----------------------+
| Cell  | Keyspace | Shard | TabletType | State       | Alias            | Hostname         | MasterTermStartTime  |
+-------+----------+-------+------------+-------------+------------------+------------------+----------------------+
| zone1 | commerce | 0     | MASTER     | SERVING     | zone1-0000000100 | localhost        | 2021-03-10T01:24:33Z |
| zone1 | commerce | 0     | REPLICA    | SERVING     | zone1-0000000101 | localhost        |                      |
| zone1 | commerce | 0     | RDONLY     | NOT_SERVING | zone1-0000000102 | localhost        |                      |
+-------+----------+-------+------------+-------------+------------------+------------------+----------------------+
3 rows in set (0.00 sec)

mysql> use @rdonly;
Database changed
mysql> select * from product;
ERROR 1105 (HY000): target: commerce.0.rdonly: no healthy tablet available for 'keyspace:"commerce" shard:"0" tablet_type:RDONLY '

…n error or times out Signed-off-by: deepthi <deepthi@planetscale.com>

…r timeout from healthcheck stream Signed-off-by: deepthi <deepthi@planetscale.com>

sougou · 2021-03-11T21:09:15Z

go/vt/discovery/healthcheck.go

@@ -404,17 +404,17 @@ func (hc *HealthCheckImpl) deleteTablet(tablet *topodata.Tablet) {
 	}
 }

-func (hc *HealthCheckImpl) updateHealth(th *TabletHealth, shr *query.StreamHealthResponse, currentTarget *query.Target, trivialNonMasterUpdate bool, isMasterUpdate bool, isMasterChange bool) {
+func (hc *HealthCheckImpl) updateHealth(th *TabletHealth, currentTarget *query.Target, trivialUpdate bool, isPrimaryUp bool) {


Rename this to isPrimaryUpdate? Otherwise, I was reading it as "is primary up".

It is supposed to be "is primary up". The old parameters (isMasterChange and isMasterUpdate) were actually unnecessary because all the information they pass in is available within the scope of this func. However, we do need to know whether the primary should be marked unhealthy (if there is an error/timeout on the healthcheck connection), so I introduced this parameter.

sougou · 2021-03-11T21:10:32Z

go/vt/discovery/healthcheck.go

-					topoproto.TabletAliasString(hc.healthy[targetKey][0].Tablet.Alias),
-					shr.TabletExternallyReparentedTimestamp,
-					hc.healthy[targetKey][0].MasterTermStartTime)
+	if th.Target.TabletType == topodata.TabletType_MASTER {


I would presume that isPrimaryUp would be true only if the tablet type was MASTER. Is there a case where this is not the case?

That is a good point. I suppose this is more of a consistency check - to make sure that we still behave correctly if a caller passes in isPrimaryUp true for a non-master tablet type.

sougou · 2021-03-11T21:14:01Z

go/vt/discovery/healthcheck.go

 			}
+		} else {


If this section was the reason you added the MASTER check, it may read better if you explicitly checked for it here. Something like th.Target.TabletType == topodata.TabletType_MASTER && !isPrimaryUp. But I'm not sure if that means the same thing.

The MASTER check was already there but in the form of a boolean (isMasterUpdate).
I can break this into two separate if blocks instead of an if .. if .. else if that is easier to read/understand. What you are suggesting will be equivalent to how it is written today.

The concern was the deep nesting which made it non-obvious. A cascading if (or switch) may read better. Try it. If it doesn't improve it, we can keep this as is.

Ok, I think the switch made it better.

sougou

Approving, in case no further improvements are possible.

sougou · 2021-03-11T22:12:15Z

go/vt/discovery/healthcheck.go

 			}
+		} else {


The concern was the deep nesting which made it non-obvious. A cascading if (or switch) may read better. Try it. If it doesn't improve it, we can keep this as is.

Signed-off-by: deepthi <deepthi@planetscale.com>

5antelope · 2021-04-20T19:49:01Z

go/vt/discovery/healthcheck.go

 					hc.healthy[targetKey][0].MasterTermStartTime)
 			} else {
 				// Just replace it.
 				hc.healthy[targetKey][0] = th
 			}
 		}
+	case isPrimary && !isPrimaryUp:
+		// No healthy master tablet
+		hc.healthy[targetKey] = []*TabletHealth{}


@deepthi / @sougou I think this might not behave correctly in case of an unplanned failover with an external reparent, please let me know if my thinking is correct here #7906

Thanks!

that makes sense. This should be fixed.

healthcheck: update healthy tablets correctly when a stream returns a…

a1b7a1f

…n error or times out Signed-off-by: deepthi <deepthi@planetscale.com>

deepthi requested review from sougou and systay March 10, 2021 01:48

healthcheck: remove PRIMARY from healthy list when there's an error o…

7eb7228

…r timeout from healthcheck stream Signed-off-by: deepthi <deepthi@planetscale.com>

sougou reviewed Mar 11, 2021

View reviewed changes

sougou approved these changes Mar 11, 2021

View reviewed changes

healthcheck: reorganize code to be more readable

f3a432e

Signed-off-by: deepthi <deepthi@planetscale.com>

deepthi added this to In progress in Cluster Management via automation Mar 13, 2021

deepthi moved this from In progress to Reviewer approved in Cluster Management Mar 13, 2021

deepthi merged commit 58fc9a1 into vitessio:master Mar 13, 2021

Cluster Management automation moved this from Reviewer approved to Done Mar 13, 2021

deepthi deleted the ds-fix-7472-7177 branch March 13, 2021 01:09

askdba added the Component: Query Serving label Mar 18, 2021

askdba added this to the v10.0 milestone Mar 18, 2021

askdba added Component: Cluster management and removed Component: Query Serving labels Mar 18, 2021

deepthi mentioned this pull request Mar 24, 2021

Healthcheck: update healthy tablets correctly when a stream returns an error or times out #7732

Merged

8 tasks

5antelope reviewed Apr 20, 2021

View reviewed changes

ajm188 mentioned this pull request Jul 15, 2021

slack vitess v10.pre tinyspeck/vitess#228

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

healthcheck: update healthy tablets correctly when a stream returns an error or times out #7654

healthcheck: update healthy tablets correctly when a stream returns an error or times out #7654

deepthi commented Mar 10, 2021 •

edited

sougou Mar 11, 2021

deepthi Mar 11, 2021

sougou Mar 11, 2021

deepthi Mar 11, 2021 •

edited

sougou Mar 11, 2021

deepthi Mar 11, 2021

sougou Mar 11, 2021

deepthi Mar 12, 2021

sougou left a comment

sougou Mar 11, 2021

5antelope Apr 20, 2021

deepthi Apr 20, 2021

healthcheck: update healthy tablets correctly when a stream returns an error or times out #7654

healthcheck: update healthy tablets correctly when a stream returns an error or times out #7654

Conversation

deepthi commented Mar 10, 2021 • edited

Description

Related Issue(s)

Checklist

Impacted Areas in Vitess

Testing

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

deepthi Mar 11, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sougou left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

deepthi commented Mar 10, 2021 •

edited

deepthi Mar 11, 2021 •

edited