Using KafkaAvroDeserializer

[arey]

Hi,

I'm trying to declare a new Message Format by declaring the io.confluent.kafka.serializers.KafkaAvroDeserializer class and uploading the kafka-avro-serializer-4.1.0.jar
The declaration failed. Class is not found.
Do you have an idea? Do you know ho to read topics serialized with Avro?

Regards

Antoine

[Crim]

So when I attempt to upload kafka-avro-serializer-4.1.0.jar with the following setup shown below, I end up with a 500 error. The stack trace complains about an unknown class, which leads me to believe that the packaged kafka-avro-serializer-4.1.0.jar jar does not contain all of the required dependencies.

Error:

2018-06-19 00:56:25.002 ERROR 8 --- [nio-8080-exec-8] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Handler dispatch failed; nested exception is java.lang.NoClassDefFoundError: io/confluent/common/config/ConfigException] with root cause

java.lang.ClassNotFoundException: io.confluent.common.config.ConfigException
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[na:1.8.0_151]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_151]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_151]
	at java.lang.Class.getDeclaredConstructors0(Native Method) ~[na:1.8.0_151]
	at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671) ~[na:1.8.0_151]
	at java.lang.Class.getConstructor0(Class.java:3075) ~[na:1.8.0_151]
	at java.lang.Class.newInstance(Class.java:412) ~[na:1.8.0_151]
	at org.sourcelab.kafka.webview.ui.manager.plugin.PluginFactory.getPlugin(PluginFactory.java:124) ~[classes!/:1.0.4]
	at org.sourcelab.kafka.webview.ui.controller.configuration.messageformat.MessageFormatController.create(MessageFormatController.java:226) ~[classes!/:1.0.4]

[Crim]

Following the instructions here, you likely need to build a JAR that contains all the required dependencies for the avro schema registry deserializer.

You can do this relatively easily by cloning the Examples project and using the following pom.xml file:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.sourcelab</groupId>
    <artifactId>kafka-webview-examples</artifactId>
    <version>1.0.0</version>

    <properties>
        <avro-registry-version>4.1.0</avro-registry-version>
        <confluent.maven.repo>http://packages.confluent.io/maven/</confluent.maven.repo>
    </properties>

    <repositories>
        <repository>
            <id>confluent</id>
            <name>Confluent</name>
            <url>${confluent.maven.repo}</url>
        </repository>
    </repositories>


    <!-- Define Dependencies -->
    <dependencies>
        <!-- Use kafka-webview-plugin dependency -->
        <!-- Scope is provided -->
        <dependency>
            <groupId>org.sourcelab</groupId>
            <artifactId>kafka-webview-plugin</artifactId>
            <version>1.0.0</version>
            <scope>provided</scope>
        </dependency>

        <!-- Kafka Dependency for Deserializers -->
        <!-- Scope is provided -->
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>0.11.0.1</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>io.confluent</groupId>
            <artifactId>kafka-avro-serializer</artifactId>
            <version>${avro-registry-version}</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <!-- Set Source & Target JRE Version -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.6.1</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>

            <!-- Copy dependencies over into packaged jar -->
            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <configuration>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                    <archive>
                        <manifest>
                            <mainClass/>
                        </manifest>
                    </archive>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-deploy-plugin</artifactId>
                <version>2.7</version>
                <configuration>
                    <skip>true</skip>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

You can then issue the command mvn package and upload the resulting jar kafka-webview-examples-1.0.0-jar-with-dependencies.jar and use the same configuration posted in the above screen shot.

I believe that should work for you, let me know.

[arey]

Thanks a lot for you reply.
I've build the 10Mb fat jar kafka-webview-examples-1.0.0-jar-with-dependencies.jar but I still have the same error. I'm using the same docker image as yesterday. Should have to pull a new version?

[Crim]

It should work with your current docker image. The update I made yesterday just more gracefully handles ClassNotFoundException exceptions.

Are you able to paste a stack trace of the error you are getting?

[arey]

There is no stracktrace nor ERROR message in the logs:

2018-06-20 13:57:27.818 INFO 19860 --- [ main] o.s.kafka.webview.ui.Application : Started Application in 13.018 seconds (JVM running for 13.805)
2018-06-20 13:58:25.080 INFO 19860 --- [MessageBroker-1] o.s.w.s.c.WebSocketMessageBrokerStats : WebSocketSession[0 current WS(0)-HttpStream(0)-HttpPoll(0), 0 total, 0 closed abnormally (0 connect failure, 0 send limit, 0 transport error)], stompSubProtocol[processed CONNECT(0)-CONNECTED(0)-DISCONNECT(0)], stompBrokerRelay[null], inboundChannel[pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0], outboundChannelpool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0], sockJsScheduler[pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 0]
2018-06-20 14:00:39.011 INFO 19860 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring FrameworkServlet 'dispatcherServlet'
2018-06-20 14:00:39.011 INFO 19860 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : FrameworkServlet 'dispatcherServlet': initialization started
2018-06-20 14:00:39.043 INFO 19860 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : FrameworkServlet 'dispatcherServlet': initialization completed in 32 ms

Just the UI error message: Unable to find class io.confluent.kafka.serializers.KafkaAvroDeserializer

[Crim]

Odd, I wiped my examples project, started over from scratch following the above instructions, and it loaded without any issues:

[arey]

I'm sorry. I don't take time to retest. Thanks for your help.