send data from my java app with influxdb library using http protocol

[mehrdad2000]

Describe the bug

Hi
I'm using influxdb need to replace it with questdb.
in my app current use influxdb java library to send data to influx (over http protocol).

now question is is there any way to send data to questiondb in that way? i mean over http via influxdb java library?

FYI: read this: https://questdb.io/docs/reference/api/ilp/overview/

Difference from InfluxDB
QuestDB TCP Receiver uses InfluxDB Line Protocol as both serialization and the transport format. InfluxDB on other hand uses HTTP as the transport and InfluxDB Line Protocol as serialization format. For this reason the existing InfluxDB client libraries will not work with QuestDB.

[ideoma]

QuestDB has a library to use ILP with TCP in Java and other languages:

https://questdb.io/docs/reference/clients/java_ilp/

[mehrdad2000]

@ideoma is there any way to do this with influx library? i don't want to change the code.

[ideoma]

You can use Telegraf to receive using HTTP and send forward using TCP

[mehrdad2000]

@ideoma already use telegraf to send system and jolokia metrics, but i have some mbeans that generate in my app and telegraf can't send them, they will send directly from my app over HTTP via influx java client library!

Any other idea?

[ideoma]

Sorry, I don't understand. I suggest running an instance of Telegraf which accepts ILP HTTP traffic from your application and forwards it to QuestDB using TCP.

[mehrdad2000]

@ideoma

1-in my JAVA APP I've generate some metrics and using "influx java client library" to send these metric to influxdb.

2-also use telegraf to send "system" and "jvm" metrics to influxdb. (using output.influx.plugin of telegraf)

3-now telegraf send "system" and "jvm" metric to questdb correctly.

4-but i can't send my JAVA APP to questdb. because questdb not allow line protocol over http!

FYI: i don't want to change code. just need to send it via http protocol to questdb in line protocol format, just like what happen currently with influxdb!

FYI: my JAVA APP metric not pass through telegraf. because they are custom metrics.

Any idea?

[ideoma]

4. I propose that your Java application sends data via Telegraf and Telegraf forwards to QuestDB.

You can set up Telegraf to act like a proxy

# Accept metrics over InfluxDB 2.x HTTP API
[[inputs.influxdb_v2_listener]]
  ## Address and port to host InfluxDB listener on
  ## (Double check the port. Could be 9999 if using OSS Beta)
  service_address = ":8086"

[[outputs.socket_writer]]
  ## Address and port to write to
  address = "tcp://questdb.questdb.svc:9009"

[javier]

I just created a working example at https://github.com/javier/influx-http-telegraf-questdb/tree/master

The example starts a telegraf agent listening at port 8086, and will accept requests from your JAVA client. It will rely those requests to a QuestDB running at localhost 9009. The code I am using is the ingestion code (with no changes) found at https://github.com/influxdata/influxdb-client-java#writes-and-queries-in-influxdb-2x.

[mehrdad2000]

@ideoma @javier
ok try that and here is the result:

currently send metric from my java app via "influx java client library" to influxdb.
now need to send telegraf instead of influx , so add below parameter to telegraf.

[[inputs.influxdb_v2_listener]]
service_address = ":8086"

after reset telegraf got this error on my java app:
2023-10-17 09:11:34,731 ERROR [ApiWriter] Can not write to influx, writeApi is null!

here is the code:

import com.influxdb.client.*;
import com.influxdb.client.write.Point;

public void write(Point point) {
if (writeApi == null) {
LOGGER.error("Can not write to influx, writeApi is null!");
return;

Any idea?
Thanks

[javier]

Hey @mehrdad2000 . I am afraid your code is partial and I cannot see what's going on. For JAVA I already provided the example where you can see how to connect. In case it helps providing also a working python example, based on https://docs.influxdata.com/influxdb/cloud/api-guide/client-libraries/python/

import influxdb_client
from influxdb_client.client.write_api import SYNCHRONOUS

bucket = "ignore"
org = "ignore"
token = "ignore"
# Store the URL of your InfluxDB instance
url="http://localhost:8086"

client = influxdb_client.InfluxDBClient(
    url=url,
    token=token,
    org=org
)

# Write script
write_api = client.write_api(write_options=SYNCHRONOUS)

p = influxdb_client.Point("my_measurement").tag("location", "Prague").field("temperature", 25.3)
write_api.write(bucket=bucket, org=org, record=p)

And it worked flawlessly. Just added the python script to my demo repository https://github.com/javier/influx-http-telegraf-questdb/tree/master/python

[mehrdad2000]

@javier I have if condition that check influx with ping, now issue is, is there anything like that on telegraf that simulate this behavior?

if (influxDBClient.ping()) {
    writeApi = influxDBClient.makeWriteApi(WriteOptions.builder().batchSize(monitoringBatchSize.get()).flushInterval(monitoringFlushInterval.get()).build());

[javier]

To the. best of my knowledge, Telegraf supports health check only via their health check plugin, where you could ping it https://github.com/influxdata/telegraf/blob/master/plugins/outputs/health/README.md. Unfortunately, the agent needs to start on a different port, so port 8086 cannot be used for that ping.

[mehrdad2000]

@javier exactly influx data and health work on same port, but telegraf won’t able to run inputs.influxdb_v2_listener and outputs.health on same port!

i think it is good idea to add this feature to telegraf. Or questdb provide adapter that accept http request from influx!

[javier]

I guess it should be possible to change this plugin for Telegraf and hopefully make it work, but I have never written a plugin for Telegraf, and I also don't know what Influx sends exactly as a ping, so not sure how difficult this really would be https://github.com/influxdata/telegraf/blob/master/plugins/inputs/influxdb_v2_listener/influxdb_v2_listener.go