[SUPPORT]com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 36 #6621
Comments
|
Did you try configuring the serializer explicitly: |
|
@alexeykudinkin Do you have interest on this ? |
yihua
added
priority:critical
|
@dik111 did you get a chance to try out this suggestion #6621 (comment) ? |
|
Created HUDI-4959 to track this issue. TL;DR Problem is in the fact that we're using Kryo to ser/de HoodieKey (which fails b/c can't find class registration nor association with the encoded id) EDIT I'm taking back my hypothesis that the issue is in the class encoding, after writing a small test to validate the issue i confirmed that Kryo actually writes out full class-name for all classes registered implicitly (as it should). It seems that the problem is actually indeed in misalignment of the Avro versions as reported by @KnightChess: quick-checking i see that b/w Avro 1.8.2 and 1.10.2, Provided that we're relying on Kryo to generate serializer for |
|
we also encountered this problem, in our env, flink-bundle avro version is 1.10.0 and spark3-bundle avro version is 1.10.2, and after flink write mor table with deletelogblock spark: Caused by: com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: Index: 36, Size: 7
Serialization trace:
string (org.apache.hudi.org.apache.avro.util.Utf8)
orderingVal (org.apache.hudi.common.model.DeleteRecord)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:731)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:731)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:391)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:302)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:813)
at org.apache.hudi.common.util.SerializationUtils$KryoSerializerInstance.deserialize(SerializationUtils.java:104)
at org.apache.hudi.common.util.SerializationUtils.deserialize(SerializationUtils.java:78)
at org.apache.hudi.common.table.log.block.HoodieDeleteBlock.deserialize(HoodieDeleteBlock.java:106)
at org.apache.hudi.common.table.log.block.HoodieDeleteBlock.getRecordsToDelete(HoodieDeleteBlock.java:91)
at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.processQueuedBlocksForInstant(AbstractHoodieLogRecordReader.java:476)
at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scanInternal(AbstractHoodieLogRecordReader.java:346)
... 30 more
Caused by: java.lang.IndexOutOfBoundsException: Index: 36, Size: 7
at java.util.ArrayList.rangeCheck(ArrayList.java:659)
at java.util.ArrayList.get(ArrayList.java:435)
at com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:60)
at com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:857)
at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:780)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:132)
... 44 morepresto: resovle: |
|
Hello i also have this probleme when i try to read data with spark, same job works fine sometimes and sometimes failed : Steps to reproduce : 1.Use spark structured streaming to fetch data from kafka in realtime
|
|
@dik111 please check my updated comment: can you try remediation that @KnightChess called out (making sure that the versions of Avro used in Flink and Spark are identical)? In the meantime, i'm going to be putting up a fix to properly handle |
|
Please follow HUDI-4959 for the updates regarding the fix. Closing this one (in favor of the Jira) |
Describe the problem you faced
I used Flink to update data in realtime, and used spark to read data, it throws an error
com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 36I used Flink to read the data and no relevant exception occurred
To Reproduce
Steps to reproduce the behavior:
1.Use Flink to update data in realtime
2.Use Spark to read data
Environment Description
Hudi version : 0.12.0
Spark version :2.4.4 and 3.2.1
Flink version :1.13.3
Hive version :3.0.0
Hadoop version :3.0.0
Storage (HDFS/S3/GCS..) : HDFS
Running on Docker? (yes/no) : no
Additional context
Here is my code:
Stacktrace
The text was updated successfully, but these errors were encountered: