fix: Avoid deleting metadata table with MOR during upgrade / downgrade

[linliu-code]

Describe the issue this Pull Request addresses

#14171

Currently while performing rollback or compaction, metadata table gets deleted since the metadata enablement configs are set to false by default. We should avoid deleting the metadata table by removing the metadata table config during compaction.

Summary and Changelog

Before we do the rollback and/or compaction, we do:

1. Get the metadata partitions based on table config.
2. For column_stats, partition_stats, bloom_filters, and record_index, if they are specified in table config, we set their enablement configs accordingly.
3. Meanwhile, we check the if index.json exists. If so, we try to get the column list for column_stats, and differentiate partitioned and non-partitioned record index based on the options.

Impact

Low.

Risk Level

Low

Documentation Update

Contributor's checklist

• Read through contributor's guide
• Enough context is provided in the sections above
• Adequate tests were added if applicable

[hudi-bot]

CI report:

• e22408f UNKNOWN
• 784d662 UNKNOWN
• 772fe6c UNKNOWN
• 20ebbed Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build