Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Some questions #11
Thank you for contributing this step to the marketplace! I am writing up a blog entry as we speak demonstrating both the consumer and producer you've created. I have a few questions and wanted to see if I've missed something or if this all makes sense.
When I first run the plugins published on the marketplace, I get the following exception:
2015/12/23 11:58:18 - Apache Kafka Consumer.0 - ERROR (version 184.108.40.206-353, build 1 from 2015-10-07 13.27.43 by buildguy) : Error initializing step [Apache Kafka Consumer]
Is this expected? To resolve the issue I placed all the lib/jar files in Spoon's main classpath. Do you have another approach? I am running Kettle 6.0 on Windows.
After I resolved that issue, I wasn't able to get the consumer to read any messages from Kafka. I tried setting various properties like auto.offset.reset=smallest, but I was unable to get anything out of Kafka. Also the transformation would stop quickly after running it. I am running Kafka 0.9 so maybe that had something to do with it.
To resolve that issue I checked out the latest source and updated the plugin to include the latest code. I noticed the new feature "Stop on empty topic", so figured there were changes that might be good to try out.
With the latest update, I still needed to copy the lib jars into the main classpath, after doing that I was able to successfully run the step!
Also with those changes I was able to get the producer working as well, I didn't need to get the latest version of the producer from github, the marketplace version worked against Kafka 0.9 just fine.
Would it be possible to update the marketplace with the latest version of the plugin?
Thank you again for contributing these steps!
Thank you for you interest and for the great article!
The problem you describe regarding the need to copy jar files into the Kettle's lib/ directory is weird, I've never encountered it before. I guess it has something to do with the version of Kettle that you're using.
I noticed in the "plugin.xml" file for the Consumer Plugin that you had lib/kafka_2.10-0.8.1.jar in the xml file, but lib/kafka_2.10-0.8.1.1.jar in the actual folder. Notice the extra ".1" at the end of the file name. Correcting the entry in plugin.xml fixed this for me. However, as with others, I am having trouble getting data using the Consumer plugin. I'll keep an eye out for the updates in the Marketplace.
New plugin version depends on kafka_2.10-0.8.2.1.jar
On Mon, Jan 4, 2016 at 11:16 PM Greg Hayes firstname.lastname@example.org wrote:
Thanks for updating the marketplace! I just tried out the latest version in my environment. I am still encountering the jar issue, maybe this is a windows only issue? Also, there now is a conflict with the xml-apis in Kettle 6.0, Kettle ships with version 2.0.2 of the jar and the plugin has an older version. Once I commented out the older version from the plugin.xml the consumer worked. The producer doesn't have an issue with it even though the jar is included in that as well.