java.lang.NoSuchMethodError: com.google.common.collect.Sets$SetView.iterator()Lcom/google/common/collect/UnmodifiableIterator #143

jiangtao7 · 2017-11-05T11:39:34Z

Follow https://sematext.com/blog/kafka-connect-elasticsearch-how-to/ to build elasticsearch connect in my local.

1, local kafka version : 0.11.0.1
2, command ./bin/connect-standalone libexec/config/connect-standalone.properties libexec/config/elasticsearch-connect.properties

Run into the following error 👍

jliu-macOS:0.11.0.1 root# ./bin/connect-standalone libexec/config/connect-standalone.properties libexec/config/elasticsearch-connect.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/Cellar/kafka/0.11.0.1/libexec/libs/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/Cellar/kafka/0.11.0.1/libexec/libs/slf4j-simple-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.collect.Sets$SetView.iterator()Lcom/google/common/collect/UnmodifiableIterator;
	at org.reflections.Reflections.expandSuperTypes(Reflections.java:380)
	at org.reflections.Reflections.<init>(Reflections.java:126)
	at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:221)
	at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:198)
	at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:159)
	at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:47)
	at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:68)
jliu-macOS:0.11.0.1 root#

The text was updated successfully, but these errors were encountered:

jiangtao7 · 2017-11-05T11:41:20Z

meanwhile, I also check google, fond one post run into similar issue like me .

I redouble check my case, it should be not caused by missing of dependencies jar copy ...

kkonstantine · 2017-11-06T04:19:21Z

Copying from my answer on the SO article you included above:

This error means that you are bringing an older version of guava earlier in your classpath. Connect worker requires guava >= 20 for org.reflections to work correctly.

kafka-connect-elasticsearch or any other connector that brings with it guava 18.0 or older will prohibit the worker from starting up. This error message means that the older guava jar was encountered first in the classpath.

Two solutions:

Using classloading isolation by setting your plugin.path in Connect worker's configuration will allow the connector to work as-is without interfering with the Connect framework.
If adding the connector to the CLASSPATH is your only option, make sure it's added after Kafka Connect's dependencies, so that the most recent guava will be picked up.

kccheung · 2017-11-23T07:14:15Z

hi, I got a similar problem when setting up kafka connect elasticsearch connector:

[2017-11-23 07:11:40,346] INFO Finished starting connectors and tasks (org.apache.kafka.connect.runtime.distributed.DistributedHerder:825)
[2017-11-23 07:11:40,369] INFO Using multi thread/connection supporting pooling connection manager (io.searchbox.client.JestClientFactory:206)
[2017-11-23 07:11:40,379] ERROR Task es_sink_ATTENDANCE_DAILY-0 threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:148)
java.lang.NoClassDefFoundError: Could not initialize class org.apache.http.impl.client.ProxyAuthenticationStrategy
	at org.apache.http.impl.client.HttpClientBuilder.build(HttpClientBuilder.java:1029)
	at io.searchbox.client.JestClientFactory.createHttpClient(JestClientFactory.java:98)
	at io.searchbox.client.JestClientFactory.getObject(JestClientFactory.java:55)
	at io.confluent.connect.elasticsearch.ElasticsearchSinkTask.start(ElasticsearchSinkTask.java:95)
	at io.confluent.connect.elasticsearch.ElasticsearchSinkTask.start(ElasticsearchSinkTask.java:53)
	at org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:246)
	at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:147)
	at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:146)
	at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:190)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
[2017-11-23 07:11:40,381] ERROR Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:149)
[2017-11-23 07:11:40,382] INFO Stopping ElasticsearchSinkTask. (io.confluent.connect.elasticsearch.ElasticsearchSinkTask:148)

I have already set plugin.path=/usr/share/java, and place the elasticsearch jars directory inside. I am starting up in distributed mode.

mohsenSy · 2017-12-02T10:23:10Z

I am having the same problem

kafka version 1.0.0

java version 8

kafka-connect-elasticsearch version 3.3.1

I modified plugin.path in config/connect-standalone.properties

plugin.path=/opt/kafka-connect-elasticsearch-v3.3.1/kafka-connect-elasticsearch/target/kafka-connect-elasticsearch-3.3.1-package/share/java/kafka-connect-elasticsearch

this directory contains the jars.

I get the following error:

[2017-12-02 10:19:03,460] INFO Scanning for plugin classes. This might take a moment ... (org.apache.kafka.connect.cli.ConnectStandalone:74)
[2017-12-02 10:19:03,493] INFO Loading plugin from: /opt/kafka-connect-elasticsearch-v3.3.1/kafka-connect-elasticsearch/target/kafka-connect-elasticsearch-3.3.1-package/share/java/kafka-connect-elasticsearch/gson-2.4.jar (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:179)
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.collect.Sets$SetView.iterator()Lcom/google/common/collect/UnmodifiableIterator;
	at org.reflections.Reflections.expandSuperTypes(Reflections.java:380)
	at org.reflections.Reflections.<init>(Reflections.java:126)
	at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:258)
	at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:201)
	at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:193)
	at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:153)
	at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:47)
	at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:75)

How can I solve it please? running in standalone mode.

kkonstantine · 2017-12-04T06:43:03Z

@mohsenSy if you set the plugin.path to be the directory that contains the connector jar with all the dependency jars, it'll think that every jar is an uber jar that contains a connector with dependencies.

In your example above you should set the plugin.path one level up. Example: plugin.path=/opt/kafka-connect-elasticsearch-v3.3.1/kafka-connect-elasticsearch/target/kafka-connect-elasticsearch-3.3.1-package/share/java/

In general, plugin.path accepts a comma separated list of paths. For example: plugin.path=/usr/share/java,/opt/connectors

kkonstantine · 2017-12-04T06:47:37Z

@kccheung similar in your case. To add the connector's jars under plugin.path=/usr/share/java you need to copy the whole directory, not each file individually.

E.g. Everything in /usr/share/java/kafka-connect-elasticsearch with plugin.path=/usr/share/java

In more detail, read here: https://docs.confluent.io/current/connect/userguide.html#connect-installing-plugins

kkonstantine · 2018-01-30T06:40:47Z

I believe we can close this issue at this point.

junaidrazzaq323 · 2018-02-17T14:52:09Z

I am facing the same error with Kafka-connect-cassandra.
I have setup plugin.path in connect-avro-distributed.properties file:

plugin.path=/home/ubuntu/kafka_2.11-1.0.0/plugins

And I have kafka-connect-cassandra-1.0.0-1.0.0-all.jar in:

/home/ubuntu/kafka_2.11-1.0.0/plugins/lib

This is the error:

[2018-02-17 14:50:03,152] INFO Scanning for plugin classes. This might take a moment ... (org.apache.kafka.connect.cli.ConnectDistributed:69)
[2018-02-17 14:50:03,163] INFO Loading plugin from: /home/ubuntu/kafka_2.11-1.0.0/plugins/lib (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:179)
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.collect.Sets$SetView.iterator()Lcom/google/common/collect/UnmodifiableIterator;
	at org.reflections.Reflections.expandSuperTypes(Reflections.java:380)
	at org.reflections.Reflections.<init>(Reflections.java:126)
	at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:258)
	at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:201)
	at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:193)
	at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:153)
	at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:47)
	at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:70)

What am I missing? @kkonstantine Kindly help me out.

kkonstantine · 2018-02-18T09:33:52Z

@junaidrazzaq323 check your classpath. An old guava is added there somehow.

Use the plugin.path for your connectors.

CLASSPATH is for the framework, and its dependencies are added there by the startup bin scripts.

https://stackoverflow.com/questions/48842850/kafka-connect-java-lang-nosuchmethoderror-com-google-common-collect-setssetvie/48850184#48850184

junaidrazzaq323 · 2018-02-18T13:17:11Z

We have setup confluent and and created plugins/lib inside home directory. Then we paste the kafka-connect-cassandra-1.0.0.jar inside it. We haven't setup CLASSPATH, we just set the plugin.path in connect-avro-distributed.properties file to /home/plugins. Next we are starting connect-distributed but we get that exception. You commented on stackoverflow that: *"The problem is that the cassandra connector uber jar has guava in it probably. Take it out of that directory, set your plugin.path to something that will have cassandra uber jar and Kafka connect, but not the one inside the other."* I don't understand this part *"* *set your plugin.path to something that will have cassandra uber jar and Kafka connect, but not the one inside the other."* Can you kindly explain a bit more? Sincerely, Junaid Razzaq Data Engineer | eMumba Pvt Ltd

…

On Sun, Feb 18, 2018 at 2:33 PM, Konstantine Karantasis < ***@***.***> wrote: @junaidrazzaq323 <https://github.com/junaidrazzaq323> check your classpath. An old guava is added there somehow. Use the plugin.path for your connectors. CLASSPATH is for the framework, and its dependencies are added there by the startup bin scripts. https://stackoverflow.com/questions/48842850/kafka-connect-java-lang- nosuchmethoderror-com-google-common-collect-setssetvie/48850184#48850184 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#143 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Ai0LF4B7wCVtAyPUMh-MqGiEsJfCpKeCks5tV-6CgaJpZM4QSWIC> .

unkrich · 2020-06-27T01:25:46Z

Just as a note for anyone that might find this later - I was building the kafka-connect-elasticsearch from scratch using maven and copied all over to the /libs folder without realizing this duplicated guava . Deleting the v18 copy resolved the issue.

kkonstantine closed this as completed Jan 30, 2018

kkonstantine mentioned this issue Jan 30, 2018

NoSuchMethod Error (Standalone) #119

Closed

whynick1 mentioned this issue Mar 20, 2018

Update build.gradle dependency version wepay/kafka-connect-bigquery#107

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

java.lang.NoSuchMethodError: com.google.common.collect.Sets$SetView.iterator()Lcom/google/common/collect/UnmodifiableIterator #143

java.lang.NoSuchMethodError: com.google.common.collect.Sets$SetView.iterator()Lcom/google/common/collect/UnmodifiableIterator #143

jiangtao7 commented Nov 5, 2017

jiangtao7 commented Nov 5, 2017

kkonstantine commented Nov 6, 2017

kccheung commented Nov 23, 2017

mohsenSy commented Dec 2, 2017

kkonstantine commented Dec 4, 2017

kkonstantine commented Dec 4, 2017 •

edited

kkonstantine commented Jan 30, 2018

junaidrazzaq323 commented Feb 17, 2018

kkonstantine commented Feb 18, 2018

junaidrazzaq323 commented Feb 18, 2018 via email

unkrich commented Jun 27, 2020

java.lang.NoSuchMethodError: com.google.common.collect.Sets$SetView.iterator()Lcom/google/common/collect/UnmodifiableIterator #143

java.lang.NoSuchMethodError: com.google.common.collect.Sets$SetView.iterator()Lcom/google/common/collect/UnmodifiableIterator #143

Comments

jiangtao7 commented Nov 5, 2017

jiangtao7 commented Nov 5, 2017

kkonstantine commented Nov 6, 2017

kccheung commented Nov 23, 2017

mohsenSy commented Dec 2, 2017

kkonstantine commented Dec 4, 2017

kkonstantine commented Dec 4, 2017 • edited

kkonstantine commented Jan 30, 2018

junaidrazzaq323 commented Feb 17, 2018

kkonstantine commented Feb 18, 2018

junaidrazzaq323 commented Feb 18, 2018 via email

unkrich commented Jun 27, 2020

kkonstantine commented Dec 4, 2017 •

edited