[HUDI-8883] Add ability to configure HoodieCompactionPlanGenerator using reflection

[vinishjail97]

Change Logs

Compaction Planners can make use of extraMetadata to decide which partitions/file groups to compact, eg: extraMetadata can be used to keep track of previous compaction results.

Impact

Provides ability to write compaction planner which can make use of extraMetadata to decide which partitions/file groups to compact.

Risk level (write none, low medium or high below)

Low

Documentation Update

  public static final ConfigProperty<String> COMPACTION_PLAN_GENERATOR = ConfigProperty
      .key("hoodie.compaction.plan.generator")
      .defaultValue(HoodieCompactionPlanGenerator.class.getName())
      .markAdvanced()
      .withDocumentation("Compaction plan generator for data files. Override with a custom plan generator "
          + "if there's a need to use extraMetadata in the compaction plan for optimizations, ignore otherwise");

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[danny0405]

We already have the stratefy classes why need to customize the plan generator?

[vinishjail97]

CompactionStrategy takes in the list of compaction operations and decides which ones to pick it up based on strategy logic. This is for the ability to keep track of metadata from previous compaction results in extraMetadata and then decide the plan.

[nsivabalan]

can you add PR desc please.

[nsivabalan]

does this reflect all latest changes.

[hudi-bot]

CI report:

• cda0165 Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[nsivabalan]