Failing to connect to Kafka #239

ShantiSumanth · 2022-05-04T12:16:59Z

Hi,
Trying to make a connection to Kafka and produce message(son) to a topic but failing to connect.

Whilst sending a message using JMeter below issues are seen
Kafka producer is closing with timeout=0ms
Login module control flag not specified in JAAS config.

Please find the login specified in jaas.conf

Have tried all the possible ways in defining the credentials in config file but didn't worked, would appreciate your help in sorting the issue?

Thanks

KloadGen Version

Kloadgen-3.5.3

Relevant log output

INFO o.a.k.c.p.KafkaProducer: [Producer clientId=perf-test] Closing the Kafka producer with timeoutMillis = 0 ms.
2022-04-28 11:04:20,547 INFO o.a.k.c.m.Metrics: Metrics scheduler closed
2022-04-28 11:04:20,547 INFO o.a.k.c.m.Metrics: Closing reporter org.apache.kafka.common.metrics.JmxReporter
2022-04-28 11:04:20,547 INFO o.a.k.c.m.Metrics: Metrics reporters closed
2022-04-28 11:04:20,547 INFO o.a.k.c.u.AppInfoParser: App info kafka.producer for perf-test unregistered
2022-04-28 11:04:20,548 ERROR o.a.j.p.j.s.AbstractJavaSamplerClient: Failed to construct kafka producer
org.apache.kafka.common.KafkaException: Failed to construct kafka producer
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:440) ~[kloadgen-3.5.3-kloadgen-plugin-3.5.3.jar:?]
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:291) ~[kloadgen-3.5.3-kloadgen-plugin-3.5.3.jar:?]
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:318) ~[kloadgen-3.5.3-kloadgen-plugin-3.5.3.jar:?]
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:303) ~[kloadgen-3.5.3-kloadgen-plugin-3.5.3.jar:?]
	at net.coru.kloadgen.sampler.KafkaSchemaSampler.setupTest(KafkaSchemaSampler.java:113) [kloadgen-3.5.3-kloadgen-plugin-3.5.3.jar:?]
	at org.apache.jmeter.protocol.java.sampler.JavaSampler.sample(JavaSampler.java:194) [ApacheJMeter_java.jar:5.4.1]
	at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:638) [ApacheJMeter_core.jar:5.4.1]
	at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:558) [ApacheJMeter_core.jar:5.4.1]
	at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:489) [ApacheJMeter_core.jar:5.4.1]
	at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:256) [ApacheJMeter_core.jar:5.4.1]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.lang.IllegalArgumentException: Login module control flag not specified in JAAS config
	at org.apache.kafka.common.security.JaasConfig.parseAppConfigurationEntry(JaasConfig.java:110) ~[kloadgen-3.5.3-kloadgen-plugin-3.5.3.jar:?]
	at org.apache.kafka.common.security.JaasConfig.<init>(JaasConfig.java:63) ~[kloadgen-3.5.3-kloadgen-plugin-3.5.3.jar:?]
	at org.apache.kafka.common.security.JaasContext.load(JaasContext.java:88) ~[kloadgen-3.5.3-kloadgen-plugin-3.5.3.jar:?]
	at org.apache.kafka.common.security.JaasContext.loadClientContext(JaasContext.java:82) ~[kloadgen-3.5.3-kloadgen-plugin-3.5.3.jar:?]
	at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:167) ~[kloadgen-3.5.3-kloadgen-plugin-3.5.3.jar:?]
	at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81) ~[kloadgen-3.5.3-kloadgen-plugin-3.5.3.jar:?]
	at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105) ~[kloadgen-3.5.3-kloadgen-plugin-3.5.3.jar:?]
	at org.apache.kafka.clients.producer.KafkaProducer.newSender(KafkaProducer.java:448) ~[kloadgen-3.5.3-kloadgen-plugin-3.5.3.jar:?]
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:429) ~[kloadgen-3.5.3-kloadgen-plugin-3.5.3.jar:?]
	... 10 more
2022-04-28 11:04:20,552 ERROR o.a.j.p.j.s.AbstractJavaSamplerClient: Failed to send message
java.lang.NullPointerException: Cannot invoke "org.apache.kafka.clients.producer.KafkaProducer.send(org.apache.kafka.clients.producer.ProducerRecord, org.apache.kafka.clients.producer.Callback)" because "this.producer" is null
	at net.coru.kloadgen.sampler.KafkaSchemaSampler.runTest(KafkaSchemaSampler.java:153) [kloadgen-3.5.3-kloadgen-plugin-3.5.3.jar:?]
	at org.apache.jmeter.protocol.java.sampler.JavaSampler.sample(JavaSampler.java:197) [ApacheJMeter_java.jar:5.4.1]
	at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:638) [ApacheJMeter_core.jar:5.4.1]
	at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:558) [ApacheJMeter_core.jar:5.4.1]
	at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:489) [ApacheJMeter_core.jar:5.4.1]
	at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:256) [ApacheJMeter_core.jar:5.4.1]
	at java.lang.Thread.run(Thread.java:833) [?:?]
2022-04-28 11:04:20,553 INFO o.a.j.t.JMeterThread: Thread is done: Thread Group - KLoadgen 1-1
2022-04-28 11:04:20,553 INFO o.a.j.t.JMeterThread: Thread finished: Thread Group - KLoadgen 1-1
2022-04-28 11:04:20,553 INFO o.a.j.e.StandardJMeterEngine: Notifying test listeners of end of test
2022-04-28 11:04:20,554 INFO o.a.j.g.u.JMeterMenuBar: setRunning(false, *local*)

Have you added your JMX Testplan or a replica?

I have added a JMX testplan

Have you added your Schema or a replica?

I have added the Schema

github-actions · 2022-05-04T12:17:42Z

Thank you for collaborating with the project by giving us feedback!!' You will help us a lot if you can attach an example AVRO and an example Test Plan in order to be able to reproduce the error, if that is the case. Cheers

cmunozgomez · 2022-05-05T05:53:28Z

Hi @ShantiSumanth

Thank you for your collaboration, we will review it and the changes/progress that we are making will inform you here.

ShantiSumanth · 2022-05-05T08:22:24Z

Thanks @cmunozgomez.

jgbriz · 2022-05-10T06:38:48Z

Hi @ShantiSumanth

We are currently reviewing the issue. We'll inform you as soon as we get a solution.

pedror19 · 2022-05-10T09:09:45Z

Hi @ShantiSumanth
We have problems reading the config from a jaas.conf file, try to set the kafka client auth from the jaas.conf file in the java.security.auth.login.config property of the java request. It must be a single line string like this:

KafkaClient { org.apache.kafka.common.security.plain.PlainLoginModule required username=(xxxx) password=(+dljlsjfl); };

ShantiSumanth · 2022-05-11T09:47:06Z

Hi @pedror19,

I've already included Kafka client auth in java request and added auth in a single line string but didn't work.
.

Could you please look into this again.

Thanks

pedror19 · 2022-05-11T10:55:19Z

Hi @ShantiSumanth,

We will review the problem again and you will be in formed as soon as we made some improvement.

pedror19 · 2022-05-11T11:14:34Z

Hi again @ShantiSumanth,
We figured out that you are still using the jass.conf file, you need to write the config like this.

ShantiSumanth · 2022-05-11T12:30:07Z

Hello @pedror19,
I've added the Kafka auth details in the java request and it is throwing java.io.ioexception

pedror19 · 2022-05-11T13:59:13Z

Hello again @ShantiSumanth.
We will review again your problem and inform you about any improvement as soon as posible

jgbriz · 2022-05-12T13:31:05Z

Hi @ShantiSumanth ,

We've noticed that you are using the 3.5.3 version of KLoadGen, can you update and use the lastest version? We will continue searching for the problem, but it'd be great if you can update so we can be sure the problem is not because of an old version.
One more thing, can you tell us where are you locating the jass.conf file?

jemacineiras · 2022-05-12T14:18:24Z

Hi @ShantiSumanth,

we were playing around a bit with Kafka Authentication, and notice you need to pass the absolute path in the java.security.auth.login.config property. It also works if you put the module configuration in a line like my colleague @pedror19 said, but, without the KafkaClient part, neither the '{'.

Can you please try after update to the latest version (if possible)?

pedror19 · 2022-05-12T14:56:41Z

Hi @ShantiSumanth,
Adding to what @jemacineiras said you need to add to the java.security.auth.login.config the serviceName field like this:

org.apache.kafka.common.security.plain.PlainLoginModule required serviceName="yourServiceName" username="(xxxxx)" password="(+dljlsjfl)";

ShantiSumanth · 2022-05-13T10:45:13Z

Thanks @pedror19 @jgbriz @jemacineiras .
I'm trying to get latest version of kloadgen plugin but failing to build the project. if possible would you mind providing me with latest version of kloadgen plugin please?

Thanks

pedror19 · 2022-05-13T10:58:48Z

Hi @ShantiSumanth,
Every version is deployed automatically to maven central repository, you can download the latest version from there.

ShantiSumanth · 2022-05-13T15:11:43Z

Thanks @pedror19.

Ran script with latest version but still failing with same error(java.io exception and login module control flag not specified).

@jgbriz , jaas.conf file is located in jmeter/bin folder so given config file name directly in the request.

Adding file name in java request is throwing error " java.lang.IllegalArgumentException: Login module control flag not specified in JAAS config"

giving authentication as a single string in the request is failing with exception, " java.io.IOException:org.apache.kafka.common.security.plain.PlainLoginModule required serviceName="yourServiceName" username="(xxxxx)" password="(+dljlsjfl)"; (No such file or directory)".

Thanks

jgbriz · 2022-05-16T07:59:25Z

Hello @ShantiSumanth ,
Thank you for the information, we'll look into it.

jgbriz · 2022-06-01T06:55:12Z

Hello @ShantiSumanth ,
We need more information about the error you are getting, so we can have a better look trying to get the cause of the problem.

ShantiSumanth · 2022-06-07T09:45:17Z

Hi @jgbriz,
Apologies, lost my project so building the script again. Will give detailed information on error by eod.

Thanks

jgbriz · 2022-06-07T13:46:28Z

Hi @ShantiSumanth
No problem. In that case, I think it's better option to close this issue, and to reopen one in case you have other problems.

ShantiSumanth · 2022-06-08T09:53:45Z

Hi @jgbriz,
Experiencing two issues now,

failing to open Value Schema File Load Generator config. fyi... Loaded json file (file content screenshot

attached) 2. throwing java.io.IOException: Configuration Error, config error , think it is complaining about json file but the file is valid without errors. error details below, Caused by: java.lang.SecurityException: java.io.IOException: Configuration Error: Line 14: expected [option key], found [null] at sun.security.provider.ConfigFile$Spi.(ConfigFile.java:139) ~[?:?] at sun.security.provider.ConfigFile.(ConfigFile.java:104) ~[?:?] at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?] at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?] at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?] at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?] at java.lang.reflect.ReflectAccess.newInstance(ReflectAccess.java:128) ~[?:?] at jdk.internal.reflect.ReflectionFactory.newInstance(ReflectionFactory.java:347) ~[?:?] at java.lang.Class.newInstance(Class.java:645) ~[?:?] at javax.security.auth.login.Configuration$2.run(Configuration.java:258) ~[?:?] at javax.security.auth.login.Configuration$2.run(Configuration.java:249) ~[?:?] at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]

Please let me know if further details required for investigating.

Thanks

jgbriz · 2022-06-08T10:33:56Z

Hi @ShantiSumanth ,
We'll look into it.

jemacineiras · 2022-06-09T14:49:31Z

Hi @ShantiSumanth,

please, can you provide the required files in order to reproduce the error?
We are going to need the .jmx and .avsc ( or json schema) file you are using, or similar, in order to investigate and understand where is the problem.

Cheers

ShantiSumanth · 2022-06-10T09:13:04Z

Hi @jemacineiras,
please find attached json and imx files
jsonfile.json.zip
CatalogueService-07June2022.jmx.zip
Thanks

jemacineiras · 2022-06-13T08:54:11Z

Hi @ShantiSumanth,

those files are a bit confusing. What you want to achieve?
I need the original avsc file and the jmx, and you send to us a json file with values on it. This is not a data definition, as the plugin requires. May be if you describe the test you want to build, we can help you better.

Cheers

ShantiSumanth · 2022-06-14T10:00:54Z

Hi @jemacineiras,
I've changed all the confidential info to dummy values.
Trying to produce/send a json file(shared earlier) to Kafka topic in confluent cloud. With the current script I'm able to successfully connect to confluent cloud and load subjects, subject values were loaded using Value Schema Load gen config.

Thanks

jemacineiras · 2022-06-14T11:40:59Z

Hi @ShantiSumanth,

that is OK, but the json you send to us is not a valid json schema it looks like a Json with values. If you are trying to use this file as schema origin, it will not work.
If you try to use a data in JSON format, as source of the generation, we do not know what to generate, because there is no type and structure reference there.
Cheers!

ShantiSumanth · 2022-06-15T16:01:00Z

Will speak to our Devs. Thanks @jemacineiras

ShantiSumanth · 2022-06-16T08:42:39Z

Hi @jemacineiras,

Thanks for your support. Isn't the subject loaded on 'Value Schema Load Generator Config' type and structure reference? please correct me if I understood it wrongly.

Thanks

jemacineiras · 2022-06-16T08:56:20Z

Hi @ShantiSumanth,

that’s right, but I'm talking about the files you sent to us. We need more information about your test and the files in order to reproduce, unless you give us clearance to deal with your files in the confluent cloud.

Cheers.

ShantiSumanth · 2022-06-20T08:21:02Z

Hi @jemacineiras,
PFA schema.

schemaDoc.docx

jgbriz · 2022-06-23T08:16:28Z

Hi @ShantiSumanth,

We've made a solution to some problems we've detected processing the schema you provided us. Please, check the draft given in this issue, and try with that version.

Cheers.

ShantiSumanth · 2022-06-27T10:15:10Z

Hi @jgbriz,

Unable pull the draft version due to authentication issues. would you mind sharing the version/jar please?

Thanks

jgbriz · 2022-06-27T13:37:37Z

Hi @ShantiSumanth ,
You should be able to checkout the draft's branch. No permissions needed.

jgbriz · 2022-07-04T14:39:33Z

Hi @ShantiSumanth ,
We've uploaded a new version with the fixes. Download the lastest version from maven-central and try it.

ShantiSumanth · 2022-07-07T13:52:42Z

Hi @jgbriz,

I've tried with draft and latest version of plugin but still throwing same error

Also throwed an exception(Uncaught Exception org.apache.avro.SchemaParseException: No type: {"customerId":"0023422","correlationId":") whilst loading json file in Value scheme file load gen config
Could you please take a look and let me know if I have missed anything.

Thanks

jemacineiras · 2022-07-11T12:38:04Z

Hi, @ShantiSumanth,

we are not able to reproduce the errors you comment to us about the Security Exception. Guessing you already had checkout all the properties with your dev team?

About the Schema Parse Exception... Why are you trying to parse a JSON into an AVO?
Is how is looks like reading that Also throwed an exception (Uncaught Exception org.apache.avro.SchemaParseException: No type: {"customerId":"0023422","correlationId":")

Cheers

jemacineiras · 2022-10-17T15:27:08Z

Hi, @ShantiSumanth,

have you tried what I said the last time? Any luck with your test?

Cheers

dianaGomezGar · 2022-11-30T09:32:10Z

I'm going to close due to inactivity. If you have any question you can open a new Issue.

cmunozgomez added the bug Something isn't working label May 5, 2022

cmunozgomez added the question Further information is requested label May 5, 2022

cmunozgomez assigned pedror19 and jgbriz and unassigned pedror19 May 5, 2022

pedror19 added the blocked Waiting for answer label May 10, 2022

pedror19 removed the blocked Waiting for answer label May 11, 2022

pedror19 added the blocked Waiting for answer label May 11, 2022

pedror19 removed the blocked Waiting for answer label May 11, 2022

pedror19 added the blocked Waiting for answer label May 12, 2022

pedror19 removed the blocked Waiting for answer label May 16, 2022

jgbriz added the blocked Waiting for answer label May 26, 2022

jgbriz removed the blocked Waiting for answer label Jun 8, 2022

jgbriz linked a pull request Jun 22, 2022 that will close this issue

#239 avro extractor union record #271

Merged

jgbriz removed a link to a pull request Jun 22, 2022

#239 avro extractor union record #271

Merged

jgbriz linked a pull request Jun 23, 2022 that will close this issue

#239 avro extractor union record #271

Merged

jgbriz removed a link to a pull request Jul 4, 2022

#239 avro extractor union record #271

Merged

jgbriz added a commit that referenced this issue Jul 4, 2022

#239 avro extractor union record (#271)

60d3700

dianaGomezGar closed this as completed Nov 30, 2022

Failing to connect to Kafka #239

Failing to connect to Kafka #239

Comments

ShantiSumanth commented May 4, 2022 • edited

KloadGen Version

Relevant log output

Have you added your JMX Testplan or a replica?

Have you added your Schema or a replica?

github-actions bot commented May 4, 2022

cmunozgomez commented May 5, 2022

ShantiSumanth commented May 5, 2022

jgbriz commented May 10, 2022

pedror19 commented May 10, 2022

ShantiSumanth commented May 11, 2022 • edited

pedror19 commented May 11, 2022

pedror19 commented May 11, 2022

ShantiSumanth commented May 11, 2022 • edited

pedror19 commented May 11, 2022

jgbriz commented May 12, 2022

jemacineiras commented May 12, 2022

pedror19 commented May 12, 2022

ShantiSumanth commented May 13, 2022

pedror19 commented May 13, 2022

ShantiSumanth commented May 13, 2022

jgbriz commented May 16, 2022

jgbriz commented Jun 1, 2022 • edited

ShantiSumanth commented Jun 7, 2022

jgbriz commented Jun 7, 2022

ShantiSumanth commented Jun 8, 2022

jgbriz commented Jun 8, 2022

jemacineiras commented Jun 9, 2022

ShantiSumanth commented Jun 10, 2022

jemacineiras commented Jun 13, 2022

ShantiSumanth commented Jun 14, 2022

jemacineiras commented Jun 14, 2022

ShantiSumanth commented Jun 15, 2022

ShantiSumanth commented Jun 16, 2022

jemacineiras commented Jun 16, 2022

ShantiSumanth commented Jun 20, 2022

jgbriz commented Jun 23, 2022

ShantiSumanth commented Jun 27, 2022

jgbriz commented Jun 27, 2022

jgbriz commented Jul 4, 2022

ShantiSumanth commented Jul 7, 2022 • edited

jemacineiras commented Jul 11, 2022

jemacineiras commented Oct 17, 2022

dianaGomezGar commented Nov 30, 2022

ShantiSumanth commented May 4, 2022 •

edited

ShantiSumanth commented May 11, 2022 •

edited

ShantiSumanth commented May 11, 2022 •

edited

jgbriz commented Jun 1, 2022 •

edited

ShantiSumanth commented Jul 7, 2022 •

edited