compare performance with s3 clickhouse and s3 tempo

[zdyj3170101136]

tempo can configured stored in s3.
clickhouse can configured stored all data into s3, too~.

and the clickhouse also have bloom filter, high performance zstd compression.

so what is the performance between them?

[joe-elliott]

I honestly don't know. I've been tempted in the past to spend time to setup a comparison, but I know an immense amount about tuning Tempo and basically nothing about Clickhouse. So I don't believe I could do anything honest in a reasonable amount of time.

Tempo stores traces contiguously using a nested columnar format and comes with a custom Trace focused query language. This allows us to do things like structural queries. Find a trace where service foo is a parent of service bar:

{ resource.service.name = "foo" } > { resource.service.name = "bar" }

In CH I believe you would be using SQL to query traces. One of these two directions may appeal to you or to be more appropriate for your org.

Tempo also is a hybrid key value/columnar store to allow fast trace by id lookup. Over ~27B traces (500 TB) stored in object storage we see low single digit second lookup times across the entire retention. But again, I do not know how this or search compares to CH.

My honest guess is that there are different tracing workloads as well as org needs that can make CH or Tempo a better fit. There is no straightforward answer.