Set up Elastic APM in dev

[Jose-Matsuda]

Parent issue: #1505 (comment)

Steps to set up APM: quick start via documentation

• Set up fleet for a self-managed Elastic cluster
• Configure the APM integration
• Install and Run Elastic Agent
• Elastic Agent should appear in the Fleet -> Agents list and should be deployed on aaw-dev-cc-00. This may eventually need to be added to argo manifests but for initial development can be a kubectl apply -f manifest.yaml.

The verification of the integration of Elastic APM and OTel will be in a separate task.

[Souheil-Yazji]

APM Configuration

Agent Policy

For this part, an added step of adding the Kubernetes Integration is required. This should allows Elastic Agent to log Kubernetes system traces.
More references available here.

The remaining work should be covered in this guide. This should deploy a manifest with the Elastic Agent, alongside the kubernetes integration, to the cluster and to managed by fleet. The following caveats apply.

Note that the manifests provided use the kube-system namespace. Changing the namespace value might cause discoverability issues with roles/bindings.

[Souheil-Yazji]

kube-state-metrics deployment info:
https://github.com/StatCan/terraform-statcan-kubernetes-core-platform/blob/main/prometheus.tf#L324

[Souheil-Yazji]

Looks like some elastic-agent rbac components are already deployed by pat's previous work, but it doesn't fully line up with the upstream manifests for the elastic-agent deployment. In particular, the Cluster Role permissions are vastly different.

[Souheil-Yazji]

The deployment offered by the ElasticSearch docs creates an Elastic agent daemonset, which creates an elastic agent container in each namespace for metrics retrieval.

When I initially read this, I was wondering if we don't have something like this already set up for promethues. After bringing it up to @vexingly and @Jose-Matsuda, Pat brought up the point that we have fluentd set up to do a similar thing in the same manner, and whether or not that could be leverged instead of having another per-namespace cointainer running in namespaces.

I haven't done enough research on fluentd's implementation and whether it's adaptable to perform a similar task or not. I think fluentd isn't capable of adding traces to log data, but it should be possible to distribute them to Elastic which seems to be the role of the Elastic agent (not the APM agent!). If it does collect trace data, then technically this should be jerry-riggable to use fluentd instead of elastic agent.

[Souheil-Yazji]

After talking to pat, will be puting this on hold while we explore a simpler trace mvp.