handle ddl event failed while reorganize partition table

[aytrack]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. create partition table with some data
2. reorganize partition table

2. What did you expect to see? (Required)

3. What did you see instead (Required)

there are some stats error log

[2023/03/13 12:58:53.376 +00:00] [ERROR] [domain.go:2167] ["handle ddl event failed"] [event="(Event Type: alter table reorganize partition, Table ID: 720, Table Name xxx, Partition IDs: [721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781]"] [error="[kv:1062]Duplicate entry '721' for key 'stats_meta.tbl'"]

while penw is the new reorganized partition

[2023/03/13 12:50:31.912 +00:00] [INFO] [backfilling.go:371] ["[ddl] backfill worker finish task"] [worker="backfill-worker 3, tp add index"] [task="taskID: 7, physicalTableID: 788, range: [7480000000000003145f72a9
[2023/03/13 13:01:46.675 +00:00] [ERROR] [analyze_global_stats.go:79] ["merge global stats failed"] [info="merge global stats for xxx.xxx's index idx_xxx_i003"] [error="[types:8131]Build global-level stats failed due to missing partition-level stats: table `xxx` partition `pnew`"] [tableID=720]
[2023/03/13 13:01:48.084 +00:00] [INFO] [2pc.go:677] ["[BIG_TXN]"] [session=0] ["key sample"=7480000000000000165f69800000000000000104061b6c0c5c38000403800000000000758a] [size=3531362] [keys=95521] [puts=3] [dels=95517] [locks=1] [checks=0] [txnStartTS=440064411344306179]
[2023/03/13 13:01:48.489 +00:00] [INFO] [gc.go:88] ["remove stats in GC due to dropped table"] [table_id=732]
[2023/03/13 13:01:49.872 +00:00] [INFO] [2pc.go:677] ["[BIG_TXN]"] [session=0] ["key sample"=7480000000000000165f69800000000000000104061b6c207330000303800000000000758b] [size=3535580] [keys=95635] [puts=3] [dels=95631] [locks=1] [checks=0] [txnStartTS=440064411816427530]
[2023/03/13 13:01:50.438 +00:00] [ERROR] [analyze_global_stats.go:79] ["merge global stats failed"] [info="merge global stats for xxx.xxx's index idx_xxxl_i004"] [error="[types:8131]Build global-level stats failed due to missing partition-level stats: table `xxx` partition `pnew`"] [tableID=720]
[2023/03/13 13:01:50.439 +00:00] [INFO] [gc.go:88] ["remove stats in GC due to dropped table"] [table_id=734]
[2023/03/13 13:01:51.566 +00:00] [INFO] [advancer.go:190] ["[log backup advancer hint] current last region"] [min="([6D, 6D44423A3531380000FD0000000000000068), 440064388656005121)"] [for-polling=0] [min-ts=2023-03-13T13:00:20Z]
[2023/03/13 13:01:51.568 +00:00] [INFO] [advancer.go:410] ["updated log backup GC safe point."] [checkpoint=440062836632190976] [target=440064388656005120]
[2023/03/13 13:01:52.257 +00:00] [INFO] [2pc.go:677] ["[BIG_TXN]"] [session=0] ["key sample"=7480000000000000165f69800000000000000104061b6ba36840000403800000000000758d] [size=3541426] [keys=95793] [puts=3] [dels=95789] [locks=1] [checks=0] [txnStartTS=440064412327346184]
[2023/03/13 13:01:52.644 +00:00] [INFO] [gc_worker.go:1199] ["[gc worker] start resolve locks"] [uuid=61b6683340c0013] [safePoint=0] [try-resolve-locks-ts=440064334261125128] [concurrency=1]
[2023/03/13 13:01:52.645 +00:00] [INFO] [range_task.go:137] ["range task started"] [name=resolve-locks-runner] [startKey=] [endKey=] [concurrency=1]
[2023/03/13 13:01:52.645 +00:00] [INFO] [gc.go:88] ["remove stats in GC due to dropped table"] [table_id=735]
[2023/03/13 13:01:53.298 +00:00] [INFO] [range_task.go:246] ["range task finished"] [name=resolve-locks-runner] [startKey=] [endKey=] ["cost time"=653.839132ms] ["completed regions"=801]
[2023/03/13 13:01:53.299 +00:00] [INFO] [gc_worker.go:1221] ["[gc worker] finish resolve locks"] [uuid=61b6683340c0013] [safePoint=0] [try-resolve-locks-ts=440064334261125128] [regions=801]
[2023/03/13 13:01:54.258 +00:00] [INFO] [2pc.go:677] ["[BIG_TXN]"] [session=0] ["key sample"=7480000000000000165f69800000000000000104061b6ba43bf8000503800000000000758e] [size=3533952] [keys=95591] [puts=3] [dels=95587] [locks=1] [checks=0] [txnStartTS=440064412930277378]
[2023/03/13 13:01:54.565 +00:00] [INFO] [gc.go:88] ["remove stats in GC due to dropped table"] [table_id=736]
[2023/03/13 13:01:54.588 +00:00] [ERROR] [analyze_global_stats.go:79] ["merge global stats failed"] [info="merge global stats for xxx.xxx's index idx_xxx_h013"] [error="[types:8131]Build global-level stats failed due to missing partition-level stats: table `xxx` partition `pnew`"] [tableID=720]
[2023/03/13 13:01:56.088 +00:00] [INFO] [2pc.go:677] ["[BIG_TXN]"] [session=0] ["key sample"=7480000000000000165f69800000000000000104061b6ba5c8d8000503800000000000758f] [size=3540538] [keys=95769] [puts=3] [dels=95765] [locks=1] [checks=0] [txnStartTS=440064413415243787]
[2023/03/13 13:01:56.405 +00:00] [INFO] [gc.go:88] ["remove stats in GC due to dropped table"] [table_id=737]
[2023/03/13 13:01:57.933 +00:00] [INFO] [2pc.go:677] ["[BIG_TXN]"] [session=0] ["key sample"=7480000000000000165f69800000000000000104061b6ba50a380006038000000000007590] [size=3529882] [keys=95481] [puts=3] [dels=95477] [locks=1] [checks=0] [txnStartTS=440064413913579524]
[2023/03/13 13:01:58.244 +00:00] [ERROR] [analyze_global_stats.go:79] ["merge global stats failed"] [info="merge global stats for rtd

4. What is your TiDB version? (Required)

| tidb_version()                                            |
+-----------------------------------------------------------+
| Release Version: v6.7.0-alpha                             |
| Edition: Community                                        |
| Git Commit Hash: 145b7cdf72feeecf7f2b0b3e01693c603712dce5 |
| Git Branch: heads/refs/tags/v6.7.0-alpha                  |
| UTC Build Time: 2023-03-11 11:42:07                       |
| GoVersion: go1.20.2                                       |
| Race Enabled: false                                       |
| TiKV Min Version: 6.2.0-alpha                             |
| Check Table Before Drop: false                            |
| Store: tikv                                               |
+-----------------------------------------------------------+

[aytrack]

In HandleDDLEvent we don't handle ActionReorganizePartition correctly. Some partitions exist before and after reorg, we should not call insertTableStats2KV for them. This is why [kv:1062]Duplicate entry '721' for key 'stats_meta.tbl' occurs.

[aytrack]

Besides, we cannot simply call insertTableStats2KV for newly-created partitions. We need to evaluate the numbers of rows for newly-created partitions and use them to set modify_count and row_count, which would be used to decide whether to trigger auto analyze.

[aytrack]

fixed by #42523