ddl: limit the number of DDL job retries #7474

ciscoxll · 2018-08-23T16:58:37Z

What problem does this PR solve?

This PR solves the problem of infinite retry when a DDL job error occurs.

What is changed and how it works?

If the current DDL job fails, it will be retried indefinitely, so we need to limit the number of retries.
Avoid error jobs blocking the entire DDL.
Add some logging and metrics.
Set the DDL job error retry number set @@global.tidb_ddl_error_retry_limit = 10000.
Fix issue limit the number of retries when a DDL job error occurs #7517

Check List

Tests

Unit test

PTAL @tiancaiamao @winkyao @zimulala .

This change is

zimulala · 2018-08-24T03:12:02Z

ddl/ddl_worker_test.go

@@ -521,6 +522,54 @@ func (s *testDDLSuite) TestBuildJobDependence(c *C) {
 	})
 }

+func (s *testDDLSuite) TestErrorCountlimit(c *C) {


Please put it to the file of fail test.

winkyao · 2018-08-25T03:00:13Z

ddl/ddl_worker.go

@@ -460,6 +463,7 @@ func (w *worker) runDDLJob(d *ddlCtx, t *meta.Meta, job *model.Job) (ver int64,
 			job.State = model.JobStateCancelled
 			job.Error = errCancelledDDLJob
 			job.ErrorCount++
+			metrics.DDLWorkerHistogram.WithLabelValues(metrics.WorkerCancelDDLJob, metrics.RetLabel(err)).Observe(time.Since(startTime).Seconds())


Is this metric valuable?

winkyao · 2018-08-25T03:00:49Z

ddl/ddl_worker.go

@@ -51,6 +51,8 @@ const (
 	waitDependencyJobInterval = 200 * time.Millisecond
 	// noneDependencyJob means a job has no dependency-job.
 	noneDependencyJob = 0
+	// errorCountlimit limit the number of retries.
+	errorCountlimit = 10000


10000 is too large, maybe 512 is enough.

Could we add a config item or system variable to set it?

winkyao · 2018-08-25T03:02:24Z

ddl/ddl_worker.go

 		} else {
-			log.Infof("[ddl-%s] the DDL job is normal to cancel because %v", w, errors.ErrorStack(err))
+			log.Infof("[ddl-%s] the DDL job is normal to cancel because %v job query %s", w, errors.ErrorStack(err), job.Query)


, job query

winkyao · 2018-08-25T03:02:31Z

ddl/ddl_worker.go

@@ -519,13 +523,18 @@ func (w *worker) runDDLJob(d *ddlCtx, t *meta.Meta, job *model.Job) (ver int64,
 	if err != nil {
 		// If job is not cancelled, we should log this error.
 		if job.State != model.JobStateCancelled {
-			log.Errorf("[ddl-%s] run DDL job err %v", w, errors.ErrorStack(err))
+			log.Errorf("[ddl-%s] run DDL job err %v job query %s ", w, errors.ErrorStack(err), job.Query)


, job query

crazycs520 · 2018-08-27T01:31:51Z

ddl/ddl_worker.go

 		}

 		job.Error = toTError(err)
 		job.ErrorCount++
+		if job.ErrorCount > errorCountlimit {
+			job.State = model.JobStateCancelling
+			metrics.DDLWorkerHistogram.WithLabelValues(metrics.WorkerCancelDDLJob, metrics.RetLabel(err)).Observe(time.Since(startTime).Seconds())


startTime is the job start execute time or start time of the last execute the job?

winkyao · 2018-08-27T02:26:56Z

ddl/ddl_worker.go

@@ -51,6 +51,8 @@ const (
 	waitDependencyJobInterval = 200 * time.Millisecond
 	// noneDependencyJob means a job has no dependency-job.
 	noneDependencyJob = 0
+	// errorCountlimit limit the number of retries.
+	errorCountlimit = 512


Make this be configurable.

@winkyao PTAL.

winkyao · 2018-08-27T07:23:37Z

sessionctx/variable/varsutil.go

+// SetDDLErrorRetryLimit sets ddlErrorRetryLimit count.
+func SetDDLErrorRetryLimit(cnt int32) {
+	if cnt < minDDLErrorRetryLimit {
+		cnt = ddlErrorRetryLimit


Why not set to minDDLErrorRetryLimit?

winkyao · 2018-08-27T07:25:08Z

sessionctx/variable/varsutil_test.go

@@ -223,4 +223,8 @@ func (s *testVarsutilSuite) TestVarsutil(c *C) {
 	c.Assert(err, IsNil)
 	c.Assert(val, Equals, "1")
 	c.Assert(v.EnableTablePartition, IsTrue)
+
+	c.Assert(GetDDLErrorRetryLimit(), Equals, int32(DefTiDBDDLErrorRetryLimit))


Add more test case like set -1 to DefTiDBDDLErrorRetryLimit. Please make your unit tests cover more cases.

@winkyao Done.

crazycs520 · 2018-08-27T07:33:22Z

ddl/ddl_worker.go

 		}

 		job.Error = toTError(err)
 		job.ErrorCount++
+		if job.ErrorCount > int64(variable.GetDDLErrorRetryLimit()) {
+			startTime := time.Now()


why startTime is not job.StartTS

zimulala · 2018-08-27T09:41:45Z

ddl/ddl_worker.go

@@ -519,13 +520,19 @@ func (w *worker) runDDLJob(d *ddlCtx, t *meta.Meta, job *model.Job) (ver int64,
 	if err != nil {
 		// If job is not cancelled, we should log this error.
 		if job.State != model.JobStateCancelled {
-			log.Errorf("[ddl-%s] run DDL job err %v", w, errors.ErrorStack(err))
+			log.Errorf("[ddl-%s] run DDL job err %v, job query %s ", w, errors.ErrorStack(err), job.Query)


Please add metrics(WorkerCancelDDLJob) here.

@zimulala Done.

zimulala · 2018-08-27T11:21:26Z

sessionctx/variable/tidb_vars.go

@@ -214,6 +217,7 @@ const (
 	DefTiDBDDLReorgWorkerCount       = 16
 	DefTiDBHashAggPartialConcurrency = 4
 	DefTiDBHashAggFinalConcurrency   = 4
+	DefTiDBDDLErrorRetryLimit        = 512


Will this value be too small?

zimulala · 2018-08-28T06:22:59Z

ddl/ddl_worker.go

 		}

 		job.Error = toTError(err)
 		job.ErrorCount++
+		if job.ErrorCount > int64(variable.GetDDLErrorRetryLimit()) {


Do we need to add a log here？

zimulala · 2018-08-28T06:29:43Z

sessionctx/variable/session.go

@@ -293,6 +293,9 @@ type SessionVars struct {
 	EnableStreaming bool

 	writeStmtBufs WriteStmtBufs
+
+	// DDLErrorRetryLimit limits the number of error retries that occur in ddl job.
+	DDLErrorRetryLimit int64


Is this variable useful?

ciscoxll · 2018-09-03T07:03:51Z

/run-all-tests

ciscoxll · 2018-09-04T02:17:20Z

@crazycs520 @winkyao @zimulala PTAL .

ciscoxll · 2018-09-05T02:14:38Z

@crazycs520 @winkyao @zimulala PTAL .

zimulala · 2018-09-05T09:13:47Z

ddl/ddl_worker.go

 		}

 		job.Error = toTError(err)
 		job.ErrorCount++
+		if job.ErrorCount > int64(variable.GetDDLErrorRetryLimit()) && job.Type != model.ActionAddIndex {
+			log.Infof("[ddl-%s] DDL job over maximum retry count is canceled because %v, job query %s", w, errors.ErrorStack(err), job.Query)
+			job.State = model.JobStateCancelling


Is there a problem with the following scenario？
If the operation of “Add column” updates to “Delete only”, then we cancel this job.

@zimulala I write a test and canceled this job, but no error.

@ciscoxll
If the column is “drop column” updates to “delete only”， but this job occurs many errors then this job cancel successfully. Then we return an error to the client, but this column's state is "delete only".
I think this will be a problem.

@zimulala I will write a rollback in another PR.

Handle it on #9295

ciscoxll · 2018-09-06T09:51:34Z

@winkyao PTAL.

crazycs520 · 2018-09-13T07:47:15Z

ddl/ddl_worker.go

+		if job.ErrorCount > int64(variable.GetDDLErrorRetryLimit()) && job.Type != model.ActionAddIndex {
+			log.Infof("[ddl-%s] DDL job over maximum retry count is canceled because %v, job query %s", w, errors.ErrorStack(err), job.Query)
+			job.State = model.JobStateCancelling
+			metrics.DDLWorkerHistogram.WithLabelValues(metrics.WorkerCancelDDLJob, job.Type.String(), metrics.RetLabel(err)).Observe(time.Since(model.TSConvert2Time(job.StartTS)).Seconds())


Repeat with line#529?

ciscoxll added the component/DDL-need-LGT3 label Aug 23, 2018

zimulala reviewed Aug 24, 2018

View reviewed changes

winkyao reviewed Aug 25, 2018

View reviewed changes

crazycs520 reviewed Aug 27, 2018

View reviewed changes

winkyao reviewed Aug 27, 2018

View reviewed changes

ciscoxll force-pushed the retry-limit branch from fe7d452 to 47a28c4 Compare August 27, 2018 03:29

winkyao reviewed Aug 27, 2018

View reviewed changes

crazycs520 reviewed Aug 27, 2018

View reviewed changes

zimulala reviewed Aug 27, 2018

View reviewed changes

zimulala reviewed Aug 28, 2018

View reviewed changes

ciscoxll force-pushed the retry-limit branch 5 times, most recently from b6d83fa to 1f19c1b Compare August 30, 2018 06:58

ddl: limit the number of ddl job retries

71acda4

ciscoxll force-pushed the retry-limit branch from fb9c447 to 71acda4 Compare August 30, 2018 12:25

fix ci error

aec7610

ciscoxll added the status/all tests passed label Sep 3, 2018

zimulala reviewed Sep 5, 2018

View reviewed changes

crazycs520 reviewed Sep 13, 2018

View reviewed changes

ciscoxll added the status/DNM label Sep 28, 2018

winkyao closed this Nov 23, 2018

sre-bot added the contribution This PR is from a community contributor. label Dec 18, 2019

you06 added the sig/sql-infra SIG: SQL Infra label Mar 4, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ddl: limit the number of DDL job retries #7474

ddl: limit the number of DDL job retries #7474

ciscoxll commented Aug 23, 2018 •

edited by c4pt0r

zimulala Aug 24, 2018

winkyao Aug 25, 2018

winkyao Aug 25, 2018

winkyao Aug 25, 2018

winkyao Aug 25, 2018

winkyao Aug 25, 2018

crazycs520 Aug 27, 2018

winkyao Aug 27, 2018

ciscoxll Aug 27, 2018

winkyao Aug 27, 2018

winkyao Aug 27, 2018

ciscoxll Aug 27, 2018

crazycs520 Aug 27, 2018

zimulala Aug 27, 2018

ciscoxll Aug 27, 2018

zimulala Aug 27, 2018

zimulala Aug 28, 2018

zimulala Aug 28, 2018

ciscoxll commented Sep 3, 2018

ciscoxll commented Sep 4, 2018

ciscoxll commented Sep 5, 2018

zimulala Sep 5, 2018

ciscoxll Sep 6, 2018

zimulala Sep 7, 2018

ciscoxll Sep 7, 2018

zimulala Feb 13, 2019 •

edited

ciscoxll commented Sep 6, 2018

crazycs520 Sep 13, 2018

ddl: limit the number of DDL job retries #7474

ddl: limit the number of DDL job retries #7474

Conversation

ciscoxll commented Aug 23, 2018 • edited by c4pt0r

What problem does this PR solve?

What is changed and how it works?

Check List

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ciscoxll commented Sep 3, 2018

ciscoxll commented Sep 4, 2018

ciscoxll commented Sep 5, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zimulala Feb 13, 2019 • edited

Choose a reason for hiding this comment

ciscoxll commented Sep 6, 2018

Choose a reason for hiding this comment

ciscoxll commented Aug 23, 2018 •

edited by c4pt0r

zimulala Feb 13, 2019 •

edited