[HUDI-9400] - Included Caffeine jar to the Hudi Hive Sync Bundle

[rangareddy]

Change Logs

Got the following exception while running the Hudi Hive Sync Bundle inside Docker image.

Exception in thread "main" java.lang.NoClassDefFoundError: com/github/benmanes/caffeine/cache/Caffeine
	at org.apache.hudi.avro.AvroSchemaCache.<clinit>(AvroSchemaCache.java:36)
	at org.apache.hudi.common.table.log.block.HoodieDataBlock.<init>(HoodieDataBlock.java:115)
	at org.apache.hudi.common.table.log.block.HoodieHFileDataBlock.<init>(HoodieHFileDataBlock.java:79)
	at org.apache.hudi.common.table.log.HoodieLogFileReader.readBlock(HoodieLogFileReader.java:202)
	at org.apache.hudi.common.table.log.HoodieLogFileReader.next(HoodieLogFileReader.java:386)
	at org.apache.hudi.common.table.log.HoodieLogFormatReader.next(HoodieLogFormatReader.java:102)
	at org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner.scanInternalV1(AbstractHoodieLogRecordScanner.java:282)
	at org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner.scanInternal(AbstractHoodieLogRecordScanner.java:252)
	at org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.performScan(HoodieMergedLogRecordScanner.java:207)
	at org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.<init>(HoodieMergedLogRecordScanner.java:123)
	at org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner$Builder.build(HoodieMergedLogRecordScanner.java:490)
	at org.apache.hudi.metadata.HoodieMetadataLogRecordReader$Builder.build(HoodieMetadataLogRecordReader.java:230)
	at org.apache.hudi.metadata.HoodieBackedTableMetadata.getLogRecordScanner(HoodieBackedTableMetadata.java:517)
	at org.apache.hudi.metadata.HoodieBackedTableMetadata.openReaders(HoodieBackedTableMetadata.java:436)
	at org.apache.hudi.metadata.HoodieBackedTableMetadata.getOrCreateReaders(HoodieBackedTableMetadata.java:421)
	at org.apache.hudi.metadata.HoodieBackedTableMetadata.lookupKeysFromFileSlice(HoodieBackedTableMetadata.java:304)
	at org.apache.hudi.metadata.HoodieBackedTableMetadata.getRecordsByKeys(HoodieBackedTableMetadata.java:264)
	at org.apache.hudi.metadata.HoodieBackedTableMetadata.getRecordByKey(HoodieBackedTableMetadata.java:155)
	at org.apache.hudi.metadata.BaseTableMetadata.fetchAllPartitionPaths(BaseTableMetadata.java:316)
	at org.apache.hudi.metadata.BaseTableMetadata.getAllPartitionPaths(BaseTableMetadata.java:124)
	at org.apache.hudi.common.fs.FSUtils.getAllPartitionPaths(FSUtils.java:249)
	at org.apache.hudi.sync.common.HoodieSyncClient.getAllPartitionPathsOnStorage(HoodieSyncClient.java:123)
	at org.apache.hudi.hive.HiveSyncTool.syncAllPartitions(HiveSyncTool.java:469)
	at org.apache.hudi.hive.HiveSyncTool.validateAndSyncPartitions(HiveSyncTool.java:321)
	at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:261)
	at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:189)
	at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:177)
	at org.apache.hudi.hive.HiveSyncTool.main(HiveSyncTool.java:547)
Caused by: java.lang.ClassNotFoundException: com.github.benmanes.caffeine.cache.Caffeine
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 28 more

Impact

Unable to perform the Hudi Hive Sync inside docker image.

Needs to include caffeine jar to the Hudi Hive Sync bundle.

<include>com.github.ben-manes.caffeine:caffeine</include> 

Output:

% jar tvf packaging/hudi-hive-sync-bundle/target/hudi-hive-sync-bundle-1.1.0-SNAPSHOT.jar | grep 'caffeine' | head -5
     0 Sun May 02 19:46:48 IST 2021 com/github/benmanes/caffeine/
  2208 Sun May 02 19:46:48 IST 2021 com/github/benmanes/caffeine/SCQHeader$HeadAndTailRef.class
  1040 Sun May 02 19:46:48 IST 2021 com/github/benmanes/caffeine/SCQHeader$HeadRef.class
  2406 Sun May 02 19:46:48 IST 2021 com/github/benmanes/caffeine/SCQHeader$PadHead.class
  2498 Sun May 02 19:46:48 IST 2021 com/github/benmanes/caffeine/SCQHeader$PadHeadAndTail.class

Risk level (write none, low medium or high below)

Medium

Documentation Update

N/A

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[hudi-bot]

CI report:

• e10b656 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build