Enable OpenTelemetry Tracing in CRI-O, Etcd, Kube APIServer

[sallyom]

Is your feature request related to a problem? Please describe.
As of kube-1.22 and etcd 3.5, exporting OpenTelemetry trace data is experimental

• Etcd: https://etcd.io/docs/v3.5/op-guide/monitoring/#distributed-tracing
• APIServer: https://kubernetes.io/blog/2021/09/03/api-server-tracing/

And CRI-O also has opentelemetry tracing support:

• Configure OpenTelemetry Tracing cri-o/cri-o#4883

Describe the solution you'd like
Enabling tracing in CRI-O, APIServer and Etcd will benefit the K8s and OpenTelemetry communities:

• Document and automate the enabling of OpenTelemetry tracing in etcd and APIServer.
• Calculate performance overhead of exporting OpenTelemetry traces.
• Document the collection and visualization of trace data with various tracing backends. OpenTelemetry Collector and Jaeger are available through OperatorHub.
• Demonstrate how tracing can improve the performance and debugging of K8s, weighed against the performance overhead of exporting traces.
• Investigate how instrumented applications running on K8s interact with the instrumented core components (etcd, apiserver)?

Enabling tracing will provide valuable data for research:

• Would like to deploy varied workflows & collect traces from the platform as well as the applications running on it.
• Does the tracing instrumentation make sense for K8s architecture? For what use-cases?

Describe alternatives you've considered
Setup to enable tracing in a pseudo environment (kubeam single node): https://github.com/sallyom/otel-kubeadm

Additional context
Etcd, APIServer, and CRI-O tracing is experimental upstream, not enabled by default. In order to enable in Operate First I will be investigating what, if any, modifications to OpenShift operators are required. Kubelet tracing may be added in a follow-up issue, instrumentation is in progress upstream.

@durandom

[durandom]

@sallyom are you looking for ways to enable this in Etcd in our environment? We define cluster-scope configuration in https://github.com/operate-first/apps/tree/master/cluster-scope
Is this something we would have to do at boot time? I have no idea on how to pass cli arguments :/

@HumairAK @tumido

[sallyom]

@durandom first step is to enable the required flags, feature-gate in OCP etcd, kube-apiserver. There's an enhancement for that and am working on getting reviews. The experimental features will not be enabled by default, but I'd like to enable & turn on the tracing features in an Operate First environment. This will provide information about the use-cases, benefits, performance-hits, etc. when tracing in a production cluster.
Upstream, to turn on in etcd requires flags added to the deployment and in APIServer, a tracing configuration file is required on the host, along with the feature-gate enabled.

[tumido]

Hey @sallyom can you please point us to any documentation what changes needs to be applied on the cluster to support that for you? Would you like us to apply the manifests from https://github.com/sallyom/otel-kubeadm ?

Do I understand correctly that this requires Openshift Service Mesh to be deployed first?

/kind feature
/priority important-longterm

[sallyom]

@tumido the otel-kubeadm manifests are only tested on a single node kubeadm cluster, not OpenShift. For OCP, there is an enhancement to add necessary changes in order to enable the experimental feature-gate/flags in etcd & kube-apiserver. (I should have included this in the issue description!) I will keep you posted here, but for now I don't believe it's possible to enable tracing in OCP-etcd, kube-apiserver.
OCP enhancement: openshift/enhancements#725

[sallyom]

@tumido to enable tracing in CRI-O, once the version with opentelemetry tracing PR is deployed (or, can we deploy a cluster with a latest-from-main-branch CRI-O?), all that is required is a tracing configuration file on each node's filesystem:

cat /etc/crio/crio.conf.d/tracing.conf
[crio.tracing]
tracing_sampling_rate_per_million=999999
enable_tracing=true

then

systemctl daemon-reload
systemctl restart crio.service

we should start with enabling tracing in CRI-O!

[tumido]

@sallyom ok, how do we know then the opentelemetry tracing PR is deployed?

can we deploy a cluster with a latest-from-main-branch CRI-O?

I'm not sure. We've never done that, if you can help us customizing OCP install we can try doing it. Can such modification survive OCP upgrades/cluster scaling?

@larsks do you have any thoughts/knowledge how to do these things?

[sallyom]

@tumido I missed that! sorry. Can you tell me what version of OCP is running? Do you run w/ latest stable release? So, 4.9?

Enabling tracing in CRI-O requires a drop-in file on the host, that might survive upgrade, I'll check on that. This will be easiest to start with, not sure how much benefit we'll get, but we'll be able to get some information about the overhead of tracing, so that will be good.

[tumido]

/unassign @tumido