add index dist task can't resume after upgrade cluster

[aytrack]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. deploy a 7.5.0 cluster
2. set global tidb_ddl_enable_fast_reorg=1; set global tidb_enable_dist_task=1
3. do add index
4. duraing add index, upgrade the cluster to nightly (134d2bf)
5. check the add index ddl jobs

2. What did you expect to see? (Required)

ddl job will continue and add index success

3. What did you see instead (Required)

add index subtask in paused state and ddl jobs in running jobs for a long time

MySQL [(none)]> select id, task_key, type, state, step, exec_id, create_time, from_unixtime(start_time), end_time, cast(error as char) from mysql.tidb_background_subtask;
+----+----------+------+--------+------+------------------+-------------+---------------------------+----------+---------------------+
| id | task_key | type | state  | step | exec_id          | create_time | from_unixtime(start_time) | end_time | cast(error as char) |
+----+----------+------+--------+------+------------------+-------------+---------------------------+----------+---------------------+
|  4 | 2        |    0 | paused |    1 | tidb-3-peer:4000 | NULL        | 2024-02-01 15:54:04       | NULL     | NULL                |
|  5 | 2        |    0 | paused |    1 | tidb-2-peer:4000 | NULL        | 2024-02-01 15:54:05       | NULL     | NULL                |
|  6 | 2        |    0 | paused |    1 | tidb-2-peer:4000 | NULL        | 2024-02-01 15:54:05       | NULL     | NULL                |
+----+----------+------+--------+------+------------------+-------------+---------------------------+----------+---------------------+
3 rows in set (0.00 sec)

MySQL [(none)]> admin show ddl jobs;
+--------+---------+---------------------------------+---------------------------+----------------------+-----------+-----------------+-----------+---------------------+---------------------+---------------------+----------+
| JOB_ID | DB_NAME | TABLE_NAME                      | JOB_TYPE                  | SCHEMA_STATE         | SCHEMA_ID | TABLE_ID        | ROW_COUNT | CREATE_TIME         | START_TIME          | END_TIME            | STATE    |
+--------+---------+---------------------------------+---------------------------+----------------------+-----------+-----------------+-----------+---------------------+---------------------+---------------------+----------+
|    256 | sbtest1 | sbtest4                         | add index /* ingest */    | write reorganization |       102 |             225 |         0 | 2024-02-01 15:51:00 | 2024-02-01 15:54:03 | NULL                | running  |
|    257 | sbtest1 | sbtest1                         | add index /* txn */       | write only           |       102 |             242 |         0 | 2024-02-01 15:51:00 | 2024-02-01 15:54:33 | NULL                | running  |
|    258 | sbtest1 | sbtest3                         | add index                 | none                 |       102 |             228 |         0 | 2024-02-01 15:51:00 | 2024-02-01 15:54:33 | NULL                | queueing |
|    259 | sbtest1 | sbtest5                         | add index                 | none                 |       102 |             226 |         0 | 2024-02-01 15:51:00 | 2024-02-01 15:54:33 | NULL                | queueing |

4. What is your TiDB version? (Required)

v7.5.0 -> nightly (134d2bf)

[ywqzzy]

The resuming task can't start because the newer version of tidb have resource control, the legacy task takes more resource(actually controlled by task.concurrency) than the capacity.