Frontend: Direct TCP connection to ingest *streaming* data with InfluxDB format

[vbmithr]

Like QuestDB does. When I played with Greptime the other day, I was unable to feed high frequency data. Doing a new HTTP connection for each packet was not acceptable for me.

Basically, I need a way where I can open a TCP connection and send data in, say, InfluxDB line protocol (since it is pretty much a standard and is simple). Websocket is also fine.

[evenyag]

I was unable to feed high frequency data

I guess the storage engine might be the bottleneck instead of the HTTP layer. We haven't focused on the performance now so the write throughput isn't too good. Many modules are still in prototype stage, including the storage engine. They are not production ready now. But we'd fix obvious performance issues in the beta or the GA version.

[vbmithr]

Okay, I get it. But overall, even if high frequency ingestion is not ready, it would be good to have the interfaces ready (i.e. Unix Domain Sockets / TCP (+Websocket) ready, so that the client work can be done pretty much once for all.

[MichaelScofield]

@vbmithr

Like QuestDB does. When I played with Greptime the other day, I was unable to feed high frequency data. Doing a new HTTP connection for each packet was not acceptable for me.

I've done a basic benchmark using TSBS last month, and achieve a "loaded 2592000 rows in 142.478sec with 10 workers (mean rate 18192.25 rows/sec)" result in http + InfluxDB line protocol.

The TSBS benchmark steps are:

• First generate benchmark data like this:

  tsbs_generate_data --use-case="cpu-only" --seed=123 --scale=100 --timestamp-start="2016-01-01T00:00:00Z" --timestamp-end="2016-01-04T00:00:00Z" --log-interval="10s" --format="influx" > /tmp/1

  It's a CSV file compatible with InfluxDB format; and it has 2592000 lines.
• Then run TSBS against with GreptimeDB:

  tsbs_load_influx --urls=http://localhost:4000/v1/influxdb --file=/tmp/1 --batch-size=100 --gzip=false --workers=10

You can try it yourself.

Basically, I need a way where I can open a TCP connection and send data in, say, InfluxDB line protocol (since it is pretty much a standard and is simple). Websocket is also fine.

GreptimeDB supports TCP connection, in various protocols. Please refer to our docs for details.

By the way, if you are using Java, we recommend our Java SDK, it's using Arrow Flight to ingest data.

[vbmithr]

Ok, thanks.
However my use case is streaming data. I don't have the data in advance. Using the HTTP endpoint incurs TCP+HTTP handshakes and I get each influxDB lines individually so I don't really want one handshake per line.

Unfortunately I use OCaml and don't have access to most fancy libraries.

It seems like a good way to batch insert data to Greptime though. Thanks for the tip. I'll export my historical data in influx format and will try this :)

[bcrant]

@vbmithr curious if you ended up doing the historical export to influxdb line protocol --> import greptimedb?

I am exploring to see if this is a viable solution to get a dozen 4TB+ Influxdb archives into parquet in object store. My initial experimentation was not promising on the influxdb side – creating the line protocol exports was very, very slow going.

On the greptimedb side, it is unclear to me whether or not the parquet files produced by the storage engine will be immediately useful outside of greptimedb.

It appears some sort of parquet export feature is underway so I am excited to see how that plays out. PR #1000

[vbmithr]

I ended up playing with DuckDB, QuestDB and ClickHouse. ClickHouse seems the most complete OLAP thing right now. If you are just interested in converting influxDB to Parquet, why not write an utility that just reads those files and output Parquet? (Golang has nice libraries for doing so). Doesn't seem you need a DB for this.