Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generated class can't be used for deserialization #111

Closed
Yotamho opened this issue Dec 27, 2018 · 1 comment
Closed

Generated class can't be used for deserialization #111

Yotamho opened this issue Dec 27, 2018 · 1 comment

Comments

@Yotamho
Copy link

Yotamho commented Dec 27, 2018

I have a Message case class compiled by avrohugger,
I saved some serialized Messages in a file (using a scala version of the avro example),
Now I try to deserialize the Message objects in the file:

def readMessages(file: File): Vector[Message] = {
  val reader = new SpecificDatumReader[Message](classOf[Message])
  val dataFileReader = new DataFileReader(file, reader)
  val result = Iterator.continually(dataFileReader.next).takeWhile(_ != null).toVector
  dataFileReader.close()
  result
}

Line 3 in the following code fails on org.apache.avro.AvroRuntimeException: Not a Specific class: class com.xxx.protogate.generatedsources.Message.
Checking on SpecificData makes a lot of sense, as the createSchema method uses reflection to get the declared "field" SCHEMA$, but this is a method in the avrohugger's compiled object, so AFAIU it will never work this way.
Is there any way to work around it for now?

Exception stack trace:

An exception or error caused a run to abort: avro.shaded.com.google.common.util.concurrent.UncheckedExecutionException: org.apache.avro.AvroRuntimeException: Not a Specific class: class com.elbitsystems.protogate.generatedsources.Message 
org.apache.avro.AvroRuntimeException: avro.shaded.com.google.common.util.concurrent.UncheckedExecutionException: org.apache.avro.AvroRuntimeException: Not a Specific class: class com.elbitsystems.protogate.generatedsources.Message
	at org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:227)
	at org.apache.avro.specific.SpecificDatumReader.<init>(SpecificDatumReader.java:37)
	at com.elbitsystems.protogate.TestUtils$.readMessages(TestUtils.scala:39)
	at com.elbitsystems.protogate.AppTest.<init>(AppTest.scala:17)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at java.lang.Class.newInstance(Class.java:442)
	at org.scalatest.tools.Runner$.genSuiteConfig(Runner.scala:1428)
	at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$8(Runner.scala:1236)
	at scala.collection.immutable.List.map(List.scala:287)
	at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1235)
	at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24(Runner.scala:1031)
	at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24$adapted(Runner.scala:1010)
	at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1506)
	at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1010)
	at org.scalatest.tools.Runner$.run(Runner.scala:850)
	at org.scalatest.tools.Runner.run(Runner.scala)
	at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2(ScalaTestRunner.java:131)
	at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:28)
Caused by: avro.shaded.com.google.common.util.concurrent.UncheckedExecutionException: org.apache.avro.AvroRuntimeException: Not a Specific class: class com.elbitsystems.protogate.generatedsources.Message
	at avro.shaded.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2234)
	at avro.shaded.com.google.common.cache.LocalCache.get(LocalCache.java:3965)
	at avro.shaded.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3969)
	at avro.shaded.com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4829)
	at org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:225)
	... 20 more
Caused by: org.apache.avro.AvroRuntimeException: Not a Specific class: class com.elbitsystems.protogate.generatedsources.Message
	at org.apache.avro.specific.SpecificData.createSchema(SpecificData.java:285)
	at org.apache.avro.specific.SpecificData$2.load(SpecificData.java:218)
	at org.apache.avro.specific.SpecificData$2.load(SpecificData.java:215)
	at avro.shaded.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
	at avro.shaded.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
	at avro.shaded.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
	at avro.shaded.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
	... 24 more
@Yotamho Yotamho changed the title Generated class is can't be used for deserialization Generated class can't be used for deserialization Dec 27, 2018
@julianpeeters
Copy link
Owner

Scala's fields can't be reflected by Java reflection, regardless of where SCHEMA$ is defined. Luckily there are other constructors we can use that will take a schema directly and preempt reflection: https://github.com/apache/avro/blob/fab3d4051c6bf5cacf6bd80895a72332ae5f362f/lang/java/avro/src/main/java/org/apache/avro/specific/SpecificDatumWriter.java#L38

Since avrohugger just generates the classes, how those classes are used is up to the user. You can find further examples in the scripted serialization tests, e.g.: https://github.com/apache/avro/blob/fab3d4051c6bf5cacf6bd80895a72332ae5f362f/lang/java/avro/src/main/java/org/apache/avro/specific/SpecificDatumWriter.java#L38

Thanks for letting me know the trouble you ran into. Please try those examples and reopen if you are still experiencing issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants