Speed Up delete process from Web UI

[darrkz]

Purpose of the pull request

Speed Up delete process from Web UI

Brief change log

add an index on table t_ds_task_instance

create index idx_task_instance_process_instance_flag on t_ds_task_instance (process_instance_id, flag);

I use postgresql as my dolphinscheduler meta data base

Because I have some process run in each 5 minutes , so lead much process instance and task instance in the database , the most top instance of task node may reach 2000 and the total t_ds_task_instance table hold nearly 1000000 records in it.

I had try delete process or process instance from the web ui. But I found it too slow and lead my postgresql server cpu usage too high, and cost nearly about one hours I foud the sql is running long time is :

select                                                                                                             

                                                                                                                   

        id, name, task_type, process_instance_id, task_code, task_definition_version, state, submit_time,          

        start_time, end_time, host, execute_path, log_path, alert_flag, retry_times, pid, app_link,                

        flag, retry_interval, max_retry_times, task_instance_priority, worker_group,environment_code , executor_id,

        first_submit_time, delay_time, task_params, var_pool, dry_run                                              

                                                                                                                   

        from t_ds_task_instance                                                                                    

        WHERE process_instance_id = $1                                                                             

        and flag = $2                                                                                              

        order by start_time desc                                                                                   

then I try to add the index , and try delete the related process definition or process instance , it Just finsh in few seconds !!!

Verify this pull request

then I try to add the index , and try delete the related process definition or process instance , it Just finsh in few seconds !!!

[mergeable]

⚠️ This PR do not change database DDL synchronize.

[codecov-commenter]

Codecov Report

Merging #12826 (68f541d) into dev (04c6b5e) will decrease coverage by 0.01%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##                dev   #12826      +/-   ##
============================================
- Coverage     39.13%   39.11%   -0.02%     
+ Complexity     4212     4211       -1     
============================================
  Files          1046     1046              
  Lines         39765    39765              
  Branches       4576     4576              
============================================
- Hits          15562    15555       -7     
- Misses        22433    22440       +7     
  Partials       1770     1770              

Impacted Files	Coverage Δ
...erver/master/processor/queue/TaskEventService.java	69.64% <0.00%> (-10.72%)	⬇️
...r/plugin/task/sqoop/parameter/SqoopParameters.java	51.19% <0.00%> (-1.20%)	⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[zhongjiajie]

It is only need to change postgresql, or did we miss adding to h2 and mysql?

[zhongjiajie]

Maybe we should also add to specific update directory

[darrkz]

I just use postgresql

and I found there are some foreign key in mysql
I think it could be removed to improve

[zhongjiajie]

I just use postgresql

and I found there are some foreign key in mysql I think it could be removed to improve

But as a PR, you should also add your change to other type of database, such as mysql and H2 to keep synchronous.
Also, you should add the index add in https://github.com/apache/dolphinscheduler/blob/489e7fe4e2d980031c91eac2a4823b9f1713659a/dolphinscheduler-dao/src/main/resources/sql/upgrade/3.2.0_schema/postgresql/dolphinscheduler_ddl.sql and https://github.com/apache/dolphinscheduler/blob/489e7fe4e2d980031c91eac2a4823b9f1713659a/dolphinscheduler-dao/src/main/resources/sql/upgrade/3.2.0_schema/mysql/dolphinscheduler_ddl.sql

[davidzollo]

@darkz1984 you did a good job,

can you add these changes to mysql and H2 to keep the db file sync, I think it's not difficult for you ^_^

[darrkz]

And add an index for a big t_ds_task_instance with much rows online when dolphinscheduler is running in PostgreSQL is very easy and just cost few milliseconds.
But this action in MySQL is very danger, It may lead table lock and long time cost, My be an production accident.

[darrkz]

@darkz1984 you did a good job,

can you add these changes to mysql and H2 to keep the db file sync, I think it's not difficult for you ^_^

I had try mysql :
#13080

[darrkz]

#13081
the post upgrade ddl

[darrkz]

@davidzollo

[darrkz]

And I think H2 database has no index, so, no need to add index for it.

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information

[ruanwenjun]

Every workflow instance has a distinct id, you will rerun the workflow instance in every 5 minutes?

In general, the index process_instance_id is enough.

[darrkz]

Every workflow instance has a distinct id, you will rerun the workflow instance in every 5 minutes?

In general, the index process_instance_id is enough.

But delete process instance will delete the task instance casde, and the task instance may be with a huge number of rows in it. In my system, it reach 10,000 of rows one day, and total reach 4,000,000 rows. So it need to add this index to speed up delete the task instance with this index

[ruanwenjun]

Every workflow instance has a distinct id, you will rerun the workflow instance in every 5 minutes?
In general, the index process_instance_id is enough.

But delete process instance will delete the task instance casde, and the task instance may be with a huge number of rows in it. In my system, it reach 10,000 of rows one day, and total reach 4,000,000 rows. So it need to add this index to speed up delete the task instance with this index

We delete the task instance by process_instance_id, I am not clear what is your problem.

[EricGao888]

@darkz1984 I don't doubt that creating an index on the combination of (process_instance_id, flag) for table t_ds_task_instance could significantly improve the querying efficiency of the sql sentence you post in this issue, which you used to manually deleted the stuff from db. But I don't think this index could improve any of the queries in TaskInstanceMapper.xml. It is not meaningful for the upstream code.

https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-dao/src/main/resources/org/apache/dolphinscheduler/dao/mapper/TaskInstanceMapper.xml

[darrkz]

@EricGao888 @ruanwenjun
I detect this problem in my dolphinscheduler, and delete the process instance too slow, may be more than 10 minutes.
And i follow the database process of dolphinscheduler, see this slow database query.
Then I add this index, the action end in some seconds.

I have some process in dolphinscheduler run each 5 minutes, so some process definition has much process instance and mo task instance.
About 20,000+ task instance of the process definition. So the delete of process definition too slow.
if the process in dolphinscheduler just run every day, delete of the process definition is fast.

[ruanwenjun]

@darkz1984 There already exist process_instance_id in t_ds_task_instance, you don't need to add again.

The problem is we will query the validate task instance when delete, we don't need to query the validate task instance, we need to directly delete by process_instance_id.

[ruanwenjun]

@darkz1984 @EricGao888 I close this PR since the index(process_instance_id, flag) is not needed, and I find the logic in our code has a bug, when we delete the task instance by workflow instance id, we will query the validate task by tag, and then delete the query result, this will cause the invalidate task instance will never be deleted.

I submit #13091 to fix this.

[EricGao888]

@darkz1984 @EricGao888 I close this PR since the index(process_instance_id, flag) is not needed, and I find the logic in our code has a bug, when we delete the task instance by workflow instance id, we will query the validate task by tag, and then delete the query result, this will cause the invalidate task instance will never be deleted.

I submit #13091 to fix this.

Sounds good to me. Thanks : )

[darrkz]

@darkz1984 @EricGao888 I close this PR since the index(process_instance_id, flag) is not needed, and I find the logic in our code has a bug, when we delete the task instance by workflow instance id, we will query the validate task by tag, and then delete the query result, this will cause the invalidate task instance will never be deleted.

I submit #13091 to fix this.

OK, good news