ddl: support set explicit task type for ddl request #45789

glorv · 2023-08-03T09:43:11Z

What problem does this PR solve?

Issue Number: ref #44517

Problem Summary:

What is changed and how it works?

Set the ExplicitRequestSourceType for all ddl kv request and ingest request to support control ddl tasks as background tasks.
This PR depends on tikv/client-go#890, that is, only set the ExplicitRequestSourceType to ddl if RequestSourceType is not ddl. In both ways, tikv can parse the source component as ddl correctly.

NOTE: Background control for DDL distributed framework is not fully supported currently because the framework itself is not mature. We will see what need to do after the the framework supports handling ddl jobs.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

tiprow · 2023-08-03T09:43:35Z

Hi @glorv. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

glorv · 2023-08-03T09:44:03Z

@wjhuang2016 PTAL for the ddl and ddl ingest part, and @lance6716 PTAL at the changes for lightning part.

hawkingrei · 2023-08-03T10:26:00Z

/test all

codecov · 2023-08-03T10:36:13Z

Codecov Report

Merging #45789 (5540c1b) into master (a0cbff2) will decrease coverage by 0.6824%.
Report is 6 commits behind head on master.
The diff coverage is 23.0769%.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #45789        +/-   ##
================================================
- Coverage   73.3499%   72.6675%   -0.6825%     
================================================
  Files          1277       1298        +21     
  Lines        393392     399566      +6174     
================================================
+ Hits         288553     290355      +1802     
- Misses        86434      90739      +4305     
- Partials      18405      18472        +67

Flag	Coverage Δ
integration	`25.5025% <23.0769%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`54.0444% <ø> (ø)`
parser	`85.0602% <ø> (ø)`
br	`47.6274% <0.0000%> (-4.4038%)`	⬇️

glorv · 2023-08-04T03:25:18Z

/hold waiting for to be merged tikv/client-go#890

nolouch · 2023-08-09T03:11:01Z

ddl/column.go

@@ -1443,7 +1444,7 @@ func (w *updateColumnWorker) cleanRowMap() {
 // BackfillData will backfill the table record in a transaction. A lock corresponds to a rowKey if the value of rowKey is changed.
 func (w *updateColumnWorker) BackfillData(handleRange reorgBackfillTask) (taskCtx backfillTaskContext, errInTxn error) {
 	oprStartTime := time.Now()
-	ctx := kv.WithInternalSourceType(context.Background(), w.jobContext.ddlJobSourceType())
+	ctx := kv.WithInternalSourceAndTaskType(context.Background(), w.jobContext.ddlJobSourceType(), kvutil.ExplicitTypeDDL)


Why it cannot use WithInternalSourceType? it should also be ddl?

The ddlJobSourceType() returns something like "ddl_add_index", so in our current check logic, this can't be recognized as ddl.

…k-type

glorv · 2023-08-09T05:43:59Z

/unhold
@wjhuang2016 @nolouch PTAL again, thank you~

glorv · 2023-08-09T05:57:02Z

/test build

tiprow · 2023-08-09T05:57:37Z

@glorv: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test build

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

glorv · 2023-08-09T06:23:33Z

/test build

tiprow · 2023-08-09T06:24:04Z

@glorv: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test build

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

hawkingrei · 2023-08-09T06:32:02Z

/ok-to-test

tiprow · 2023-08-09T06:44:05Z

@glorv: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
tiprow_fast_test	`5540c1b`	link	true	`/test tiprow_fast_test`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

nolouch

lgtm

ti-chi-bot · 2023-08-09T07:56:51Z

[LGTM Timeline notifier]

Timeline:

2023-08-04 02:49:39.909604867 +0000 UTC m=+412863.851953403: ☑️ agreed by lance6716.
2023-08-09 07:56:49.808856999 +0000 UTC m=+99374.357872986: ☑️ agreed by nolouch.

glorv · 2023-08-09T08:00:51Z

/hold waiting for @wjhuang2016's review

ti-chi-bot · 2023-08-09T10:21:28Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lance6716, nolouch, wjhuang2016

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [lance6716,wjhuang2016]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

glorv · 2023-08-09T10:24:36Z

/unhold

glorv · 2023-08-09T10:30:27Z

/test unit-test

tiprow · 2023-08-09T10:30:58Z

@glorv: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test tiprow_fast_test

Use /test all to run all jobs.

In response to this:

/test unit-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

hawkingrei · 2023-08-09T12:31:39Z

/test all

support set explicit task type for ddl request

2246b87

ti-chi-bot bot added release-note-none size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 3, 2023

glorv requested review from lance6716 and wjhuang2016 August 3, 2023 09:43

update bazel

5fed1ab

lance6716 approved these changes Aug 4, 2023

View reviewed changes

ti-chi-bot bot added needs-1-more-lgtm approved labels Aug 4, 2023

ti-chi-bot bot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Aug 4, 2023

nolouch reviewed Aug 9, 2023

View reviewed changes

glorv added 2 commits August 9, 2023 13:33

update client-go

7fb0f47

Merge branch 'master' of https://github.com/pingcap/tidb into ddl-tas…

5540c1b

…k-type

ti-chi-bot bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 9, 2023

ti-chi-bot bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 9, 2023

ti-chi-bot bot added the ok-to-test label Aug 9, 2023

nolouch approved these changes Aug 9, 2023

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm labels Aug 9, 2023

ti-chi-bot bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 9, 2023

wjhuang2016 approved these changes Aug 9, 2023

View reviewed changes

ti-chi-bot bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 9, 2023

ti-chi-bot bot merged commit 1b7a17e into pingcap:master Aug 9, 2023
20 of 30 checks passed

glorv deleted the ddl-task-type branch December 15, 2023 08:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ddl: support set explicit task type for ddl request #45789

ddl: support set explicit task type for ddl request #45789

glorv commented Aug 3, 2023 •

edited

tiprow bot commented Aug 3, 2023

glorv commented Aug 3, 2023

hawkingrei commented Aug 3, 2023

codecov bot commented Aug 3, 2023 •

edited

glorv commented Aug 4, 2023

nolouch Aug 9, 2023

glorv Aug 9, 2023

glorv commented Aug 9, 2023

glorv commented Aug 9, 2023

tiprow bot commented Aug 9, 2023

glorv commented Aug 9, 2023

tiprow bot commented Aug 9, 2023

hawkingrei commented Aug 9, 2023

tiprow bot commented Aug 9, 2023

nolouch left a comment

ti-chi-bot bot commented Aug 9, 2023

glorv commented Aug 9, 2023

ti-chi-bot bot commented Aug 9, 2023

glorv commented Aug 9, 2023

glorv commented Aug 9, 2023

tiprow bot commented Aug 9, 2023

hawkingrei commented Aug 9, 2023

ddl: support set explicit task type for ddl request #45789

ddl: support set explicit task type for ddl request #45789

Conversation

glorv commented Aug 3, 2023 • edited

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

tiprow bot commented Aug 3, 2023

glorv commented Aug 3, 2023

hawkingrei commented Aug 3, 2023

codecov bot commented Aug 3, 2023 • edited

Codecov Report

glorv commented Aug 4, 2023

nolouch Aug 9, 2023

Choose a reason for hiding this comment

glorv Aug 9, 2023

Choose a reason for hiding this comment

glorv commented Aug 9, 2023

glorv commented Aug 9, 2023

tiprow bot commented Aug 9, 2023

glorv commented Aug 9, 2023

tiprow bot commented Aug 9, 2023

hawkingrei commented Aug 9, 2023

tiprow bot commented Aug 9, 2023

nolouch left a comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Aug 9, 2023

[LGTM Timeline notifier]

glorv commented Aug 9, 2023

ti-chi-bot bot commented Aug 9, 2023

glorv commented Aug 9, 2023

glorv commented Aug 9, 2023

tiprow bot commented Aug 9, 2023

hawkingrei commented Aug 9, 2023

glorv commented Aug 3, 2023 •

edited

codecov bot commented Aug 3, 2023 •

edited