[Feature] Support enabling partition aggregation for tables created in older versions

[sevev]

Why I'm doing:

In previous PRs, we implemented table-level enable_partition_aggregation property to reduce API calls in StarRocks shared-data mode. However, this property currently can only be specified during table creation. As a result, even after users upgrade to the latest version, they cannot enable this feature for existing tables unless they create new ones.

Therefore, we need to develop a method to activate this functionality for historical tables.

What I'm doing:

Users can enable/disable this feature using the following SQL command when they upgrade to new version:

Alter table $table_name set ('enable_partition_property' = 'true');
Alter table $table_name set ('enable_partition_property' = 'false');

But there is one critical limitation:consecutive modifications to enable_partition_aggregation are prohibited.

e.g.

1. If you successfully execute:

   ALTER TABLE test_tbl SET ('enable_partition_aggregation' = 'true');

2. You cannot immediately follow it with:

   ALTER TABLE test_tbl SET ('enable_partition_aggregation' = 'false');

   Otherwise, you may encounter errors like:

    Property enable_partition_aggregation cannot be updated now because this table contains mixed metadata types (both split and aggregate). Please wait until old metadata versions are vacuumed);
   

Why:

1. enable_partition_aggregation is designed as a non-frequently toggled property. Modification support is primarily intended to allow enabling the feature on tables created in old version.
2. Modify enable_partition_aggregation will ​​modify the table's metadata storage format​​. So the system must support ​​coexistence of both metadata formats​​ during the transition period. After modify enable_partition_aggregation, the table metadata will keep a metadataSwitchVersion field to track the version of the format change. Frequent modifications will increased complexity in compatibility logic, So the modification is prohibited​​ when metadataSwitchVersion is not 0.

When we can do the next modification of enable_partition_aggregation property?

If you want to modify enable_partition_aggregation, you can check the metadataSwitchVersion first. You can run show '/proc/dbs/$db/$table/partitions' to check the metadataSwitchVersion.

How can we efficiently reset the metadataSwitchVersion to 0 after completing the metadata format transition?

​​metadataSwitchVersion is reset to 0 when the version before metadataSwitchVersion is vacuumed. So we can decrease lake_autovacuum_grace_period_minutes to speed up vacuum.

Fixes #58316

What type of PR is this:

• BugFix
• Feature
• Enhancement
• Refactor
• UT
• Doc
• Tool

Does this PR entail a change in behavior?

• Yes, this PR will result in a change in behavior.
• No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

• Interface/UI changes: syntax, type conversion, expression evaluation, display information
• Parameter changes: default values, similar parameters but with different default values
• Policy changes: use new policy to replace old one, functionality automatically enabled
• Feature removed
• Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

• I have added test cases for my bug fix or my new feature
• This pr needs user documentation (for new or modified features or behaviors)
  • I have added documentation for my new feature or new function
• This is a backport pr

Bugfix cherry-pick branch check:

• I have checked the version labels which the pr will be auto-backported to the target branch
  • 3.5
  • 3.4
  • 3.3
  • 3.2
  • 3.1

[sonarqubecloud]

Quality Gate passed

Issues
16 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[github-actions]

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

[github-actions]

[FE Incremental Coverage Report]

✅ pass : 80 / 89 (89.89%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/service/InformationSchemaDataSource.java	0	1	00.00%	[491]
🔵	com/starrocks/catalog/OlapTable.java	4	7	57.14%	[3819, 3820, 3821]
🔵	com/starrocks/sql/analyzer/AlterTableClauseAnalyzer.java	11	15	73.33%	[275, 281, 292, 296]
🔵	com/starrocks/alter/LakeTableAlterMetaJob.java	12	13	92.31%	[80]
🔵	com/starrocks/alter/LakeTableAlterMetaJobBase.java	6	6	100.00%	[]
🔵	com/starrocks/catalog/PhysicalPartition.java	7	7	100.00%	[]
🔵	com/starrocks/catalog/system/information/PartitionsMetaSystemTable.java	1	1	100.00%	[]
🔵	com/starrocks/alter/LakeTableAsyncFastSchemaChangeJob.java	2	2	100.00%	[]
🔵	com/starrocks/alter/AlterJobExecutor.java	2	2	100.00%	[]
🔵	com/starrocks/task/TabletMetadataUpdateAgentTaskFactory.java	15	15	100.00%	[]
🔵	com/starrocks/alter/SchemaChangeHandler.java	9	9	100.00%	[]
🔵	com/starrocks/lake/vacuum/AutovacuumDaemon.java	8	8	100.00%	[]
🔵	com/starrocks/common/proc/PartitionsProcDir.java	3	3	100.00%	[]

[github-actions]

[BE Incremental Coverage Report]

✅ pass : 5 / 5 (100.00%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	be/src/exec/schema_scanner/schema_partitions_meta_scanner.cpp	2	2	100.00%	[]
🔵	be/src/storage/lake/txn_log_applier.cpp	1	1	100.00%	[]
🔵	be/src/storage/lake/schema_change.cpp	2	2	100.00%	[]