Roadmap to promote the opentelemetry tracing feature to be production ready

[FrankChen021]

The distributed tracing feature is heavily used in our environment to deal with problems in distributed environment and it has to be proven a powerful feature for a long time that helps us solve or find out many complicated problems.
Most of the work that are implemented in our own branch has been contributed back to the community.

The main features in the master branch now are as follows:

1. Trace on cluster DDLs
2. Trace a query on local node and its sub-queries on remote node
3. Trace async or sync INSERT on distributed table
4. Trace queries from HTTP/TCP/GRPC
5. Propagate tracing context to downstream servers via URL engine

But the status of this feature is still marked as experienmental.
From the community perspective, I think it's time for us to give a plan to promot it as a production ready feature.

Before that, here some things that I can come across to be completed:

1. Standardize the attribute names as defined in the opentelemetry specification.
   The attribute names now can be defined in anyway, it's better to use current specification to standardize some of them to allow the logs can be easily handled by some other external visulization tools. This is NOT backward compatible.
2. Investigate the root cause of Abort in OpenTelemetry::SpanHolder::finish() #49185
   Even though it occurs in Debug build, but it indicates that this may lead to incorrect logs in the Release build if it happens
3. Add trace_id column to system.query_log
   This will give a clear info in the query log that if a query is traced or not. And then it can be used to search/join the opentelemetry_span_logs table
4. Support Materialized View in the distributed tracing
   See: Add OpenTelemetry Support to Materialized View #41672
5. Propagate the tracing context to remote S3
   Some S3-compatible remote storages support this distributed tracing feature. It would give us the ability to deal with problems between ClickHouse and underlying S3 storage.

What do you think? @alexey-milovidov

[alexey-milovidov]

1. Yes, let's change the attribute names to match some standards until it is too late.
2. This error is related to the usage of fibers due to enabling hedged requests. @Avogar is investigating it.
3. Let's do it.
4. This pull request is still a draft, let's go ahead.
5. This is of low priority, but why not... Also check Sign all headers for s3 requests #34585, you can continue in another PR.

[mzealey]

I've just been playing with this via python asynch library which includes some support for it. There is what seems to be non-standard behaviour with traceparent in that the CH opentelemetry_span_log takes the span id as parent_span_id but does not take the trace_id (which in python at least is not a uuid). I think that what seems to be standard behaviour in otel ecosystem is that prior to going to production-ready status (and while these other changes are being made) it should probably be changed so that CH trace_id can be anything, and that it accepts this from the traceparent parameter?

[FrankChen021]

the CH opentelemetry_span_log takes the span id as parent_span_id but does not take the trace_id (which in python at least is not a uuid).

If you look at the code in ClickHouse, it follows the opentelemtry standard to parse the traceparent. I don't know how it happens, but you can give an example.

[mzealey]

Sure, but it seems like what the behaviour of other otel systems (eg python and java integrations) is is that trace-id is also inherited from traceparent, whereas with clickhouse it is not and there is a uuid type restriction on it which would need lifting for it to be allowed

[alexey-milovidov]

Depends on #52584

[maruthgoyal]

👋 @alexey-milovidov ooc, is there an update with regards to this issue? (since the dependent issues seem to be marked clsoed)

[alexey-milovidov]

If @FrankChen021 says it is production-ready, we can remove the experimental flag.

[FrankChen021]

I don't see there will be breaking changes of this feature in future.
Even some of the work in this issue have not been done, we can see them as improvements of this future.
So, let's mark it as production ready.