Failed to connect to InfluxDB

[LearningBuddy11]

HI,
I followed the steps and looks like my cluster and pods are running. ./initialize_cluster.sh also worked as expected and i see the correct output as posted on your github. I tried to run the test via ./start_test.sh and that seemed to work fine but looks my grafana data source (jmterdb) is showing up but failing to connect.
I tried the same in my local jmeter and it says Failed to connect to influx db. Please find the log from jmeter below. I did google stuff and i cant connect to InfluxDB via cluster ip and its shows no node port for infuxdb service when i run minikube service list. Any ideas how to troubleshoot this?

Endpoint: http://tqa-loadtest-influxdb:8086/write?db=jmeter
InfluxdConfig file: bind-address = "127.0.0.1:8088" (this is also not reachable via browser)

Log:
ava.util.concurrent.ExecutionException: java.net.UnknownHostException: tqa-loadtest-influxdb
at org.apache.http.concurrent.BasicFuture.getResult(BasicFuture.java:71) ~[httpcore-4.4.11.jar:4.4.11]
at org.apache.http.concurrent.BasicFuture.get(BasicFuture.java:95) ~[httpcore-4.4.11.jar:4.4.11]
at org.apache.http.impl.nio.client.FutureWrapper.get(FutureWrapper.java:75) ~[httpasyncclient-4.1.4.jar:4.1.4]
at org.apache.jmeter.visualizers.backend.influxdb.HttpMetricsSender.destroy(HttpMetricsSender.java:235) [ApacheJMeter_components.jar:5.1.1 r1855137]
at org.apache.jmeter.visualizers.backend.influxdb.InfluxdbBackendListenerClient.teardownTest(InfluxdbBackendListenerClient.java:413) [ApacheJMeter_components.jar:5.1.1 r1855137]
at org.apache.jmeter.visualizers.backend.BackendListener.testEnded(BackendListener.java:387) [ApacheJMeter_components.jar:5.1.1 r1855137]
at org.apache.jmeter.visualizers.backend.BackendListener.testEnded(BackendListener.java:395) [ApacheJMeter_components.jar:5.1.1 r1855137]
at org.apache.jmeter.engine.StandardJMeterEngine.notifyTestListenersOfEnd(StandardJMeterEngine.java:218) [ApacheJMeter_core.jar:5.1.1 r1855137]
at org.apache.jmeter.engine.StandardJMeterEngine.run(StandardJMeterEngine.java:492) [ApacheJMeter_core.jar:5.1.1 r18551

[infinitydon]

The Influxdb service is running as a ClusterIP service type, this is why you will not see any Nodeport that will be assigned to it, the loopback IP 127.0.0.1 is not referring to your system, this is referring to the influxdb POD..

Can you provide the output of the following command?

kubectl -n tqa get all

kubectl -n tqa describe svc tqa-loadtest-influxdb

[LearningBuddy11]

Hi,
Please find the output for the same below:

MacBook:jmeter-operator $ kubectl -n tqa get all
NAME READY STATUS RESTARTS AGE
pod/tqa-loadtest-grafana-7b8db9bdb5-b6f7l 1/1 Running 1 41h
pod/tqa-loadtest-influxdb-745b76b6d6-gwhhf 1/1 Running 1 41h
pod/tqa-loadtest-jmeter-master-6cd6d7df46-kbt7z 1/1 Running 1 41h
pod/tqa-loadtest-jmeter-slaves-768d8c4dc9-2ktwc 1/1 Running 1 41h
pod/tqa-loadtest-jmeter-slaves-768d8c4dc9-k4sf6 1/1 Running 1 41h

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/tqa-loadtest-grafana LoadBalancer 10.104.4.209 3000:32041/TCP 41h
service/tqa-loadtest-influxdb ClusterIP 10.107.35.203 8083/TCP,8086/TCP,2003/TCP 41h
service/tqa-loadtest-jmeter-slaves-svc ClusterIP None 1099/TCP,50000/TCP 41h

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/tqa-loadtest-grafana 1/1 1 1 41h
deployment.apps/tqa-loadtest-influxdb 1/1 1 1 41h
deployment.apps/tqa-loadtest-jmeter-master 1/1 1 1 41h
deployment.apps/tqa-loadtest-jmeter-slaves 2/2 2 2 41h

NAME DESIRED CURRENT READY AGE
replicaset.apps/tqa-loadtest-grafana-7b8db9bdb5 1 1 1 41h
replicaset.apps/tqa-loadtest-influxdb-745b76b6d6 1 1 1 41h
replicaset.apps/tqa-loadtest-jmeter-master-6cd6d7df46 1 1 1 41h
replicaset.apps/tqa-loadtest-jmeter-slaves-768d8c4dc9 2 2 2 41h

kubectl -n tqa describe svc tqa-loadtest-influxdb
Name: tqa-loadtest-influxdb
Namespace: tqa
Labels: app=tqa-loadtest-influxdb
Annotations:
Selector: app=tqa-loadtest-influxdb
Type: ClusterIP
IP: 10.107.35.203
Port: http 8083/TCP
TargetPort: 8083/TCP
Endpoints: 172.17.0.8:8083
Port: api 8086/TCP
TargetPort: 8086/TCP
Endpoints: 172.17.0.8:8086
Port: graphite 2003/TCP
TargetPort: 2003/TCP
Endpoints: 172.17.0.8:2003
Session Affinity: None
Events:

[infinitydon]

Looks Ok to me..

Can you please check the Influxdb logs also..

From your explanation both grafana and jmeter can't establish a connection with the Influxdb, can you try to use the service IP address of influxdb directly in the Grafana and the test script?

Change http://tqa-loadtest-influxdb:8086/write?db=jmeter (in cloudssky.jmx) to http://10.107.35.203:8086/write?db=jmeter

Also in the influxdb datasource in GRAFANA use the Inlfuxdb IP address 10.107.35.203 instead of tqa-loadtest-influxdb

[LearningBuddy11]

Hi,
So looks like i am able to get the data in the influx DB when i made the change you suggested above. The only issue i am seeing is when i start the test via ./start_test.sh is where to find the declaration of the number of users, duration etc. Is that something declared in the test plan? Also, how to stop the test? Thanks!!

jmeter-operator $ ./start_test.sh
Enter the Jmeter Namespace: tqa
Enter path to the jmx file cloudssky.jmx
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/jmeter/apache-jmeter-5.0/lib/log4j-slf4j-impl-2.11.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/jmeter/apache-jmeter-5.0/lib/ext/pepper-box-1.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Creating summariser

Created the tree successfully using cloudssky.jmx
Configuring remote engine: 172.17.0.2
Configuring remote engine: 172.17.0.4
Starting remote engines
Starting the test @ Fri Jun 07 19:06:05 UTC 2019 (1559934365966)

[infinitydon]

First of all, it looks like your kube-dns/coredns is not functioning properly, this should not be, I suggested that action as a way of troubleshooting why you are having issues with the inlfuxdb, I strongly suggest you fix this! DNS is a critical aspect of your kubernetes infrastructure..

The test plan (cloudssky.jmx) is where you configure all the necessary test parameters, please check the jmeter documentation for more information on this.

To stop an ongoing test, you can try the suggestion in issue #7

[LearningBuddy11]

Hi,
Yeah sure i am looking into fixing the issue. Thanks for your help with this issue.