disttask: support detectTask in parallel and do some updates for dispatcher

[zimulala]

What problem does this PR solve?

Issue Number: close #42383, close #42369

Problem Summary:

What is changed and how it works?

• detectTask in parallel

  • Every gTask using a goroutine when detecting a task state(call detectTask)

• Using spool.Pool instead of tidbutil.WaitGroupWrapper to new a goroutine.

• Handle the TODO of Consider get all unfinished tasks.

• If updating task/subtasks to the storage fails, ensure that the gTask state is not changed in cache.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• hawkingrei
• wjhuang2016

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[zimulala]

Unstable test: #42461

[zimulala]

/test unit-test

[GMHDBJD]

/retest

[lcwangchao]

A better way to handle the error is discard the task info in the memory and refetch it from storage because we do not know either the operation is succeed or not. Morever, the refetch operation still can return an error, so we need retry it until success.

[zimulala]

Generally, the reason for the error here is to update data to the storage layer error, the result is still to obtain information from the storage layer, feeling that the probability of failure is relatively large. It feels better to do a copy before this operation, and then do a copy after the operation is successful. But I'm only checking state so I'm just going to do it right now.

[lcwangchao]

We should consider one scene the network request succeed but failed to response to client. If we revert the state in memory, this reverted state is wrong.

[hawkingrei]

/test unit-test

[hawkingrei]

/test all

[zimulala]

/test check-dev2

[zimulala]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: 4afe50c