New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some questions #11

Closed
wgorman opened this Issue Dec 23, 2015 · 6 comments

Comments

Projects
None yet
3 participants
@wgorman

wgorman commented Dec 23, 2015

Thank you for contributing this step to the marketplace! I am writing up a blog entry as we speak demonstrating both the consumer and producer you've created. I have a few questions and wanted to see if I've missed something or if this all makes sense.

When I first run the plugins published on the marketplace, I get the following exception:

2015/12/23 11:58:18 - Apache Kafka Consumer.0 - ERROR (version 6.0.0.0-353, build 1 from 2015-10-07 13.27.43 by buildguy) : Error initializing step [Apache Kafka Consumer]
2015/12/23 11:58:18 - Apache Kafka Consumer.0 - ERROR (version 6.0.0.0-353, build 1 from 2015-10-07 13.27.43 by buildguy) : java.lang.NoClassDefFoundError: kafka/consumer/ConsumerConfig

Is this expected? To resolve the issue I placed all the lib/jar files in Spoon's main classpath. Do you have another approach? I am running Kettle 6.0 on Windows.

After I resolved that issue, I wasn't able to get the consumer to read any messages from Kafka. I tried setting various properties like auto.offset.reset=smallest, but I was unable to get anything out of Kafka. Also the transformation would stop quickly after running it. I am running Kafka 0.9 so maybe that had something to do with it.

To resolve that issue I checked out the latest source and updated the plugin to include the latest code. I noticed the new feature "Stop on empty topic", so figured there were changes that might be good to try out.

With the latest update, I still needed to copy the lib jars into the main classpath, after doing that I was able to successfully run the step!

Also with those changes I was able to get the producer working as well, I didn't need to get the latest version of the producer from github, the marketplace version worked against Kafka 0.9 just fine.

Would it be possible to update the marketplace with the latest version of the plugin?

Thank you again for contributing these steps!

Will

@wgorman

This comment has been minimized.

Show comment
Hide comment
@wgorman

wgorman Dec 23, 2015

Here's my writeup, thanks again for sharing these steps with the community!
http://www.willgorman.com/?p=61

wgorman commented Dec 23, 2015

Here's my writeup, thanks again for sharing these steps with the community!
http://www.willgorman.com/?p=61

@spektom

This comment has been minimized.

Show comment
Hide comment
@spektom

spektom Dec 24, 2015

Member

Hi Will,

Thank you for you interest and for the great article!
I've just pushed an update to the Marketplace repository, but it will take some time until it's available on the Marketplace itself.

The problem you describe regarding the need to copy jar files into the Kettle's lib/ directory is weird, I've never encountered it before. I guess it has something to do with the version of Kettle that you're using.
I'll try to run consumer/producer plug-ins with Kettle 6 soon.

Thanks!
Michael

Member

spektom commented Dec 24, 2015

Hi Will,

Thank you for you interest and for the great article!
I've just pushed an update to the Marketplace repository, but it will take some time until it's available on the Marketplace itself.

The problem you describe regarding the need to copy jar files into the Kettle's lib/ directory is weird, I've never encountered it before. I guess it has something to do with the version of Kettle that you're using.
I'll try to run consumer/producer plug-ins with Kettle 6 soon.

Thanks!
Michael

@ghayes

This comment has been minimized.

Show comment
Hide comment
@ghayes

ghayes Jan 4, 2016

I noticed in the "plugin.xml" file for the Consumer Plugin that you had lib/kafka_2.10-0.8.1.jar in the xml file, but lib/kafka_2.10-0.8.1.1.jar in the actual folder. Notice the extra ".1" at the end of the file name. Correcting the entry in plugin.xml fixed this for me. However, as with others, I am having trouble getting data using the Consumer plugin. I'll keep an eye out for the updates in the Marketplace.

Thanks,
Greg

ghayes commented Jan 4, 2016

I noticed in the "plugin.xml" file for the Consumer Plugin that you had lib/kafka_2.10-0.8.1.jar in the xml file, but lib/kafka_2.10-0.8.1.1.jar in the actual folder. Notice the extra ".1" at the end of the file name. Correcting the entry in plugin.xml fixed this for me. However, as with others, I am having trouble getting data using the Consumer plugin. I'll keep an eye out for the updates in the Marketplace.

Thanks,
Greg

@spektom

This comment has been minimized.

Show comment
Hide comment
@spektom

spektom Jan 6, 2016

Member

New plugin version depends on kafka_2.10-0.8.2.1.jar
Can you grab the latest version?

Thanks,
Michael

On Mon, Jan 4, 2016 at 11:16 PM Greg Hayes notifications@github.com wrote:

I noticed in the "plugin.xml" file for the Consumer Plugin that you had
lib/kafka_2.10-0.8.1.jar in the xml file, but lib/kafka_2.10-0.8.1.1.jar in
the actual folder. Notice the extra ".1" at the end of the file name.
Correcting the entry in plugin.xml fixed this for me. However, as with
others, I am having trouble getting data using the Consumer plugin. I'll
keep an eye out for the updates in the Marketplace.

Thanks,
Greg


Reply to this email directly or view it on GitHub
#11 (comment)
.

Member

spektom commented Jan 6, 2016

New plugin version depends on kafka_2.10-0.8.2.1.jar
Can you grab the latest version?

Thanks,
Michael

On Mon, Jan 4, 2016 at 11:16 PM Greg Hayes notifications@github.com wrote:

I noticed in the "plugin.xml" file for the Consumer Plugin that you had
lib/kafka_2.10-0.8.1.jar in the xml file, but lib/kafka_2.10-0.8.1.1.jar in
the actual folder. Notice the extra ".1" at the end of the file name.
Correcting the entry in plugin.xml fixed this for me. However, as with
others, I am having trouble getting data using the Consumer plugin. I'll
keep an eye out for the updates in the Marketplace.

Thanks,
Greg


Reply to this email directly or view it on GitHub
#11 (comment)
.

@wgorman

This comment has been minimized.

Show comment
Hide comment
@wgorman

wgorman Jan 11, 2016

Thanks for updating the marketplace! I just tried out the latest version in my environment. I am still encountering the jar issue, maybe this is a windows only issue? Also, there now is a conflict with the xml-apis in Kettle 6.0, Kettle ships with version 2.0.2 of the jar and the plugin has an older version. Once I commented out the older version from the plugin.xml the consumer worked. The producer doesn't have an issue with it even though the jar is included in that as well.

wgorman commented Jan 11, 2016

Thanks for updating the marketplace! I just tried out the latest version in my environment. I am still encountering the jar issue, maybe this is a windows only issue? Also, there now is a conflict with the xml-apis in Kettle 6.0, Kettle ships with version 2.0.2 of the jar and the plugin has an older version. Once I commented out the older version from the plugin.xml the consumer worked. The producer doesn't have an issue with it even though the jar is included in that as well.

@spektom

This comment has been minimized.

Show comment
Hide comment
@spektom

spektom Jun 28, 2017

Member

Please try the newer version, it uses annotation-based registration.

Member

spektom commented Jun 28, 2017

Please try the newer version, it uses annotation-based registration.

@spektom spektom closed this Jun 28, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment