Jaeger: Jose + Souheil investigation + jaeger-operator work

[chuckbelisle]

Monitoring Epic: #839
Epic: #914
relates to #916 - Jaeger - MVP implementation planning

[Jose-Matsuda]

This might be notes it might not be notes;
https://github.com/jaegertracing/helm-charts/tree/main/charts/jaeger-operator.
So we probably reference and use this helm chart in the prod infrastructure repo (monitoring-system.tf), and then maybe have the After the Helm installation inside the aaw-argocd-manifests repo.

What we need to do is verify the default settings each Jaeger instance will use, we can use jaege.spec inside the helm chart to define a baseline, and if we wanted to, we could also use jaeger.create and set that to true so that we don't even need to touch another repo (like aaw-argocd-manifests)

Jaeger CRD definition

[Jose-Matsuda]

Current Status: 2022/12/12 (will be edited)

Installation in Dev cluster

Working on getting the jaeger-operator functional and installed in dev, running into a few errors that are preventing the terraform from applying.

• jaeger-operator pod confirmed alive

Configuration in Dev cluster

Can take the minimal jaeger instance configuration and just expand off of it, we have the CRD definition linked above.

Here are settings that we will need to configure (as per Pat in the MVP planning ticket)

• Get istio traces
  Possibly related issues: values.global.tracer.zipkin.address is deprecated istio/istio.io#9668
  Might be a cloud thing? Look at the infrastructure example eventually leading to the gitlab repo that's not under DAaas Infrastructure
• Resource sizing, not required at the moment can fine-tune this
• Configure to use elastic search if it's in the same namespaces. Here are the configuration options or this as talked about in all options
• Configure a cleanup of elastic search
• Configure connection to prometheus options for monitoring of Jaeger
  • Configuration of elasticsearch and prometheus you can do from environment variables
• Networking which I myself am not 100% on yet
  • I think I understand that the collector port would go under the collector block like you can see here

---

I think we should split this out into smaller tickets as well; like one size s for the configuration of Prometheus with Jaeger, another size s for elasticsearch integration etc.

[Souheil-Yazji]

Agreed with @Jose-Matsuda. Enabling Jaeger and integrating it successfully with our monitoring eco-system is a multistep process. For documentation and traceability purposes, it would be best to have an Epic for Jaeger with the tasks mentioned above itemized in there.

[Jose-Matsuda]

Closing as a catch-up for Souheil and I to take a look at jaeger get familiar with it as well as put in the jaeger-operator in dev