[HUDI-7587] Make hudi-hadoop-common module dependent on hudi-common module

[jonvex]

Change Logs

make hudi-hadoop-common depend on hudi-common instead of the reverse. This is the next step for removing hadoop from hudi-common. There is hardcoded reflection currently that will be mostly removed or will be configurable for the other filesystems.

Took some methods out of FileSystemTestUtils and moved to HoodieTestTable because they are only needed in hadoop tests, and are mostly only used by that single test class

Impact

make hudi-hadoop-common depend on hudi-common instead of the reverse

Risk level (write none, low medium or high below)

medium

Documentation Update

N/A

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[yihua]

Is this moved to somewhere else?

[jonvex]

Yeah, since we need to use reflection, I moved it into a new constructor for HoodieHadoopStorage:

  public HoodieHadoopStorage(StoragePath path,
                             StorageConfiguration<?> conf,
                             boolean enableRetry,
                             long maxRetryIntervalMs,
                             int maxRetryNumbers,
                             long initialRetryIntervalMs,
                             String retryExceptions,
                             ConsistencyGuard consistencyGuard) {    
    FileSystem fileSystem = getFs(path, conf.unwrapCopyAs(Configuration.class));

    if (enableRetry) {
      fileSystem = new HoodieRetryWrapperFileSystem(fileSystem,
          maxRetryIntervalMs, maxRetryNumbers, initialRetryIntervalMs, retryExceptions);
    }
    checkArgument(!(fileSystem instanceof HoodieWrapperFileSystem),
        "File System not expected to be that of HoodieWrapperFileSystem");
    this.fs = new HoodieWrapperFileSystem(fileSystem, consistencyGuard);
  }

[jonvex]

@yihua to respond later

[yihua]

Looks like we have to add a new constructor due to the usage of reflection. We'll simplify this later.

[yihua]

Is test-jar type required?

[jonvex]

IDK, I just copied from what the other poms did

[yihua]

Overall LGTM! Could you rename the PR as "Make hudi-hadoop-common module dependent on hudi-common" to be clear?

Once you address all the nits, you can merge the PR.

[hudi-bot]

CI report:

• 94063d1 UNKNOWN
• bf97332 UNKNOWN
• e8f82c4 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build