NoClassDefFoundError AbstractKafkaAvroSerDeConfig

[kevinwallimann]

Running the ingestor fails as of 01a1ecd with the following error message

13:26:56.095 [main] INFO  za.co.absa.hyperdrive.ingestor.implementation.manager.factories.OffsetManagerAbstractFactory$ - Going to load factory for configuration 'component.manager'.
13:26:56.100 [main] INFO  za.co.absa.hyperdrive.ingestor.implementation.manager.checkpoint.CheckpointOffsetManager$ - Going to create CheckpointOffsetManager instance using: topic='clickstream', checkpoint base location='/tmp/checkpoint-location'
13:26:56.102 [main] INFO  za.co.absa.hyperdrive.ingestor.implementation.decoder.factories.StreamDecoderAbstractFactory$ - Going to load factory for configuration 'component.decoder'.
Exception in thread "main" java.lang.NoClassDefFoundError: io/confluent/kafka/serializers/AbstractKafkaAvroSerDeConfig
	at za.co.absa.hyperdrive.ingestor.implementation.decoder.avro.confluent.ConfluentAvroKafkaStreamDecoder$.apply(ConfluentAvroKafkaStreamDecoder.scala:72)
	at za.co.absa.hyperdrive.ingestor.implementation.decoder.factories.StreamDecoderAbstractFactory$.build(StreamDecoderAbstractFactory.scala:39)
	at za.co.absa.hyperdrive.driver.IngestionDriver.getStreamDecoder(IngestionDriver.scala:63)
	at za.co.absa.hyperdrive.driver.IngestionDriver.ingest(IngestionDriver.scala:48)
	at za.co.absa.hyperdrive.driver.drivers.PropertiesIngestionDriver$.main(PropertiesIngestionDriver.scala:49)
	at za.co.absa.hyperdrive.driver.drivers.PropertiesIngestionDriver.main(PropertiesIngestionDriver.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: io.confluent.kafka.serializers.AbstractKafkaAvroSerDeConfig
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 18 more
19/11/28 13:26:56 INFO SparkContext: Invoking stop() from shutdown hook

io/confluent/kafka/serializers/AbstractKafkaAvroSerDeConfig is part of kafka-avro-serializer and was added as a test dependency for PR #59

The line in which it fails is
PARAM_SCHEMA_REGISTRY_URL -> getOrThrow(KEY_SCHEMA_REGISTRY_URL, configuration, errorMessage = s"Schema Registry URL not specified. Is '$KEY_SCHEMA_REGISTRY_URL' configured?")
Oddly enough, there is no reference to AbstractKafkaAvroSerDeConfig in the whole project.

[kevinwallimann]

io.confluent.kafka.serializers.AbstractKafkaAvroSerDeConfig is used by za.co.absa.abris.avro.read.confluent.SchemaManager.
The explicit dependency for kafka-avro-serializer introduced in PR #59 has scope test and shadows the scope of the dependency as declared in abris

[kevinwallimann]

Closed b/c PR merged

[DevarajKR]

which versions of dependencies i should use , i am getting this error with kafka-avro-serializer-6.0.0.jar
@kevinwallimann

[kevinwallimann]

Hi @DevarajKR
If you use the latest fat-jars for Hyperdrive, i.e. https://mvnrepository.com/artifact/za.co.absa.hyperdrive/hyperdrive-release_spark-2 for Spark 2.4.x or https://mvnrepository.com/artifact/za.co.absa.hyperdrive/hyperdrive-release_spark-3 for Spark 3.0.x / 3.1.x, then you shouldn't need to provide kafka-avro-serializer-6.0.0.jar because the io/confluent/kafka/serializers/AbstractKafkaAvroSerDeConfig is already included in the jar.

Could you share the spark-submit command or similar and the error message?