[HUDI-7776] Simplify HoodieStorage instance fetching

[yihua]

Change Logs

This PR simplifies HoodieStorage instance fetching to pass down the HoodieStorage instance from the meta client as much as possible, instead of using reflection, which may not work with a given file system instance like TrinoFileSystem instance.

The major changes in this PR include:

• Introduces a new config hoodie.storage.class for instantiation of HoodieStorage class in Spark engine only.
• Makes HoodieIOFactory to instantiate with HoodieStorage instance and use the instance for creating readers and writers; makes HoodieStorage to store the StorageConfiguration instance and adds a new API #newInstance to HoodieStorage.
• Removes usage of HoodieStorageUtils.getStorage on the read path in hudi-common and hudi-io modules. Before this PR, HoodieStorage instantiated through reflection with the path and storage configuration through HoodieStorageUtils.getStorage does not work with a provided file system instance like TrinoFileSystem. With this PR, HoodieStorage instance is passed down from the meta client as much as possible. For engines like Spark, we still have the reflection code for HoodieHadoopStorage to make it work on the executor side.
• Refactors the code so that the HoodieStorage is passed down to the places it is needed.

Impact

As above.

Risk level

low

Documentation Update

none

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[hudi-bot]

CI report:

• 24ea421 UNKNOWN
• e02d4a3 Azure: FAILURE
• 2006412 UNKNOWN

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[codope]

Do we need a separate config? Can we infer from HOODIE_IO_FACTORY_CLASS? Or maybe we can add infer function.

[yihua]

I think we can replace HOODIE_IO_FACTORY_CLASS with HOODIE_STORAGE_CLASS since wherever the HoodieIOFactory is instantiated, the HoodieStorage is needed. So we can add a new API getIOFactory() in HoodieStorage to get the HoodieIOFactory instance and only use HOODIE_STORAGE_CLASS if the instantiation through reflection is needed.

[codope]

makes sense

[yihua]

Rethinking about this, the reason we cannot directly use a getter to return the HoodieIOFactory instance from HoodieStorage instance is that HoodieIOFactory and related classes are in hudi-common module because they use Hudi concepts such as HoodieRecord, BloomFilter, etc., while the HoodieStorage is in hudi-io module (hudi-common module depends on hudi-io). For now, the best way is to keep two configs. For Hadoop-based implementation, no configs are required as the default are the HoodieHadoopIOFactory and HoodieHadoopStorage.

[yihua]

Created HUDI-7789 for future effort of moving HoodieIOFactory to hudi-io module and keeping one config only.

[yihua]

@hudi-bot run azure

[yihua]

Azure CI is green on another PR with the same changes (this PR could not trigger Azure CI somehow).