[SUPPORT] EMR 6.13.0 Hudi cleaning throws method not found for SIMS cache

[subash-metica]

Caused by: java.lang.IllegalStateException: com.github.benmanes.caffeine.cache.SIMS
at com.github.benmanes.caffeine.cache.LocalCacheFactory.loadFactory(LocalCacheFactory.java:90) ~[app.jar:?]
at com.github.benmanes.caffeine.cache.LocalCacheFactory.newBoundedLocalCache(LocalCacheFactory.java:40) ~[app.jar:?]
at com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalManualCache.(BoundedLocalCache.java:3947) ~[app.jar:?]
at com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalManualCache.(BoundedLocalCache.java:3943) ~[app.jar:?]
at com.github.benmanes.caffeine.cache.Caffeine.build(Caffeine.java:1051) ~[app.jar:?]
at org.apache.hudi.common.util.InternalSchemaCache.(InternalSchemaCache.java:72) ~[hudi-utilities-bundle_2.12-0.13.1-amzn-1.jar:0.13.1-amzn-1]
... 79 more
Caused by: java.lang.NoSuchMethodException: no such constructor: com.github.benmanes.caffeine.cache.SIMS.(Caffeine,AsyncCacheLoader,boolean)void/newInvokeSpecial
at java.lang.invoke.MemberName.makeAccessException(MemberName.java:974) ~[?:?]
at java.lang.invoke.MemberName$Factory.resolveOrFail(MemberName.java:1117) ~[?:?]
at java.lang.invoke.MethodHandles$Lookup.resolveOrFail(MethodHandles.java:3649) ~[?:?]
at java.lang.invoke.MethodHandles$Lookup.findConstructor(MethodHandles.java:2750) ~[?:?]
at com.github.benmanes.caffeine.cache.LocalCacheFactory.loadFactory(LocalCacheFactory.java:85) ~[app.jar:?]
at com.github.benmanes.caffeine.cache.LocalCacheFactory.newBoundedLocalCache(LocalCacheFactory.java:40) ~[app.jar:?]
at com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalManualCache.(BoundedLocalCache.java:3947) ~[app.jar:?]
at com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalManualCache.(BoundedLocalCache.java:3943) ~[app.jar:?]
at com.github.benmanes.caffeine.cache.Caffeine.build(Caffeine.java:1051) ~[app.jar:?]
at org.apache.hudi.common.util.InternalSchemaCache.(InternalSchemaCache.java:72) ~[hudi-utilities-bundle_2.12-0.13.1-amzn-1.jar:0.13.1-amzn-1]
... 79 more
Caused by: java.lang.NoSuchMethodError: com.github.benmanes.caffeine.cache.SIMS: method 'void (com.github.benmanes.caffeine.cache.Caffeine, com.github.benmanes.caffeine.cache.AsyncCacheLoader, boolean)' not found
at java.lang.invoke.MethodHandleNatives.resolve(Native Method) ~[?:?]
at java.lang.invoke.MemberName$Factory.resolve(MemberName.java:1085) ~[?:?]
To Reproduce

Steps to reproduce the behavior:

1. Create a COW Hudi table with 10 commits, and then perform delete. The cleaning kicks off but fails with error.

Expected behavior

Successful clean operation

Environment Description

• EMR Version: 6.13.0

• Hudi version : 0.13.1-amz

• Spark version : 3.3.2

• Storage (HDFS/S3/GCS..) : S3

• Running on Docker? (yes/no) : no

Additional context

Add any other context about the problem here.

Stacktrace

Caused by: java.lang.IllegalStateException: com.github.benmanes.caffeine.cache.SIMS
at com.github.benmanes.caffeine.cache.LocalCacheFactory.loadFactory(LocalCacheFactory.java:90) ~[app.jar:?]
at com.github.benmanes.caffeine.cache.LocalCacheFactory.newBoundedLocalCache(LocalCacheFactory.java:40) ~[app.jar:?]
at com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalManualCache.(BoundedLocalCache.java:3947) ~[app.jar:?]
at com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalManualCache.(BoundedLocalCache.java:3943) ~[app.jar:?]
at com.github.benmanes.caffeine.cache.Caffeine.build(Caffeine.java:1051) ~[app.jar:?]
at org.apache.hudi.common.util.InternalSchemaCache.(InternalSchemaCache.java:72) ~[hudi-utilities-bundle_2.12-0.13.1-amzn-1.jar:0.13.1-amzn-1]
... 79 more
Caused by: java.lang.NoSuchMethodException: no such constructor: com.github.benmanes.caffeine.cache.SIMS.(Caffeine,AsyncCacheLoader,boolean)void/newInvokeSpecial
at java.lang.invoke.MemberName.makeAccessException(MemberName.java:974) ~[?:?]
at java.lang.invoke.MemberName$Factory.resolveOrFail(MemberName.java:1117) ~[?:?]
at java.lang.invoke.MethodHandles$Lookup.resolveOrFail(MethodHandles.java:3649) ~[?:?]
at java.lang.invoke.MethodHandles$Lookup.findConstructor(MethodHandles.java:2750) ~[?:?]
at com.github.benmanes.caffeine.cache.LocalCacheFactory.loadFactory(LocalCacheFactory.java:85) ~[app.jar:?]
at com.github.benmanes.caffeine.cache.LocalCacheFactory.newBoundedLocalCache(LocalCacheFactory.java:40) ~[app.jar:?]
at com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalManualCache.(BoundedLocalCache.java:3947) ~[app.jar:?]
at com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalManualCache.(BoundedLocalCache.java:3943) ~[app.jar:?]
at com.github.benmanes.caffeine.cache.Caffeine.build(Caffeine.java:1051) ~[app.jar:?]
at org.apache.hudi.common.util.InternalSchemaCache.(InternalSchemaCache.java:72) ~[hudi-utilities-bundle_2.12-0.13.1-amzn-1.jar:0.13.1-amzn-1]
... 79 more
Caused by: java.lang.NoSuchMethodError: com.github.benmanes.caffeine.cache.SIMS: method 'void (com.github.benmanes.caffeine.cache.Caffeine, com.github.benmanes.caffeine.cache.AsyncCacheLoader, boolean)' not found
at java.lang.invoke.MethodHandleNatives.resolve(Native Method) ~[?:?]
at java.lang.invoke.MemberName$Factory.resolve(MemberName.java:1085) ~[?:?]

[subash-metica]

Upon looking at the error, it is getting triggered for MOR table - which only metadata table is MOR since the base table is COW in my example. Looks like the error is not in cleaning but while performing compaction of metadata table which is MOR. Any leads on how to fix this issue ?

[ad1happy2go]

@subash-metica Can you please try with OSS version - 0.13.1, This error looks to be related to AWS managed Hudi version.

[subash-metica]

Hi @ad1happy2go - Thanks for quick reply. I tried using the OSS bundle jar instead of EMR provided one.

While doing so, I get java.lang.ClassNotFoundException: org.apache.spark.sql.adapter.Spark2Adapter exception.

Spark in EMR is 3.4.0 but the uber jar is of version Spark 3.3.2 and I sent the bundle jar as --jars while submitting the job. If I use the EMR bundle jar it starts fine.

The problem might be due to conflicting dependencies in the uber jar of Caffeine dependency - I will try to fix it and start again to avoid it but I am not able to use OSS bundle jar.

[subash-metica]

I have fixed the java.lang.NoSuchMethodError: com.github.benmanes.caffeine.cache.SIMS - the issue is due to conflicting caffeine dependency. I shaded our dependency and it worked in EMR, so I will close this issue.

PS: Using OSS bundle is still not working in EMR job submission, which is not a blocker for me at this point. Thanks for quick help.