[Feature] Support ClickHouse as a core storage backend

[haanhvu]

Summary

Build first-class support for ClickHouse  as an official Jaeger backend. ClickHouse is an open-source column-oriented database for OLAP use cases. It is highly efficient and performant for high volumes of ingestion and search making it a good database for tracing and logging data specifically. It can also do aggregates very quickly which will come in handy for several features in Jaeger.

Benefits to the users:

• Efficient backend
• Powerful search
• Analytics capability, e.g. the possibility to support the APM function (Monitoring tab in Jaeger) directly from ClickHouse

Existing Implementations

• Jaeger's own implementation via a remote storage plugin: https://github.com/jaegertracing/jaeger-clickhouse/
• OTEL's implementation: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/clickhouseexporter

Design

As part of the CNCG GSoC program, a blog post was published that benchmarked the aforementioned implementations and settled on a design for ClickHouse in Jaeger.

Proposal

• Benchmark existing design choices and make design decisions (schema, batch insert) (Benchmark report)
• Implement spans exporting ([WIP] Implement spans exporting for ClickHouse storage in Jaeger V2 #4941)
• Implement spans reading
• Implement dependency store
• Add necessary storage configs and options
• Support Monitoring tab directly from ClickHouse
• Implement sampling store

[haanhvu]

@yurishkuro pls check if I miss anything

[VaibhavMalik4187]

Hey @haanhvu, and @yurishkuro, I'm interested in contributing to this issue. However, I'm new to the codebase and do not have a deep understanding of the complexities of the tasks mentioned in this issue. While Haanhvu working on span export, do you think that I can start working on the Implement dependency store item? Looking forward to collaborating with you on these tasks. I'm open to working on other items on the checklist as well. Thank!

[haanhvu]

However, I'm new to the codebase and do not have a deep understanding of the complexities of the tasks mentioned in this issue.

These tasks are medium in complexity. You're of course welcome to work on these. But they would require more time and effort than good first issues, especially if you're new to Jaeger and ClickHouse.

[VaibhavMalik4187]

I'm all in, I'd appreciate it if you could share some pointers about the dependencies between each of these tasks. As a beginner, I'd like to tackle independent tasks first.

[haanhvu]

Yeah I think dependency store can be implemented independently from span store. Right @yurishkuro ?

[yurishkuro]

It can be but only as an extension of already established framework for Ch integration

[haanhvu]

It can be but only as an extension of already established framework for Ch integration

@yurishkuro So #4941 needs to be merged first right?

[varshith257]

Will look into it

[varshith257]

@yurishkuro Can I take up on this issue progress?
Or Would I prefer

• [Jaeger v2] Add Kafka exporter and receiver #4868

I think the Kafka exporter left to be work as part of our V2 roadmap.

[chenlujjj]

Regarding the "Implement dependency store" task, is a big data job like https://github.com/jaegertracing/spark-dependencies needed as well ?

[yurishkuro]

@chenlujjj I would say dependency store is much lower priority, but yes, ideally the spark job would be extended to support a different storage.