Server: Add support for shipping traces to an otel collector

[douglascamata]

What's done in the PR:

• Add support for telemeter/rhelemeter servers to ship traces in OTEL format to an OTEL collector.
  • Changes their integration tests to run an OTEL collector and ship traces to it.
• Log request details using trace and span ids in both Telemeter (used a randomly generated request id before) and Rhelemeter.
  • We don't need to generate yet another request identifier if we already have the trace and span ids there. The proposal is to standardize around the traces.

Took some extra time to...

• Upgrade Prometheus used in the integration tests from a release old enough that it didn't have a build for my OS/architecture to the most recent.
  • Gave up on this: it triggers failures in integration tests that are very difficult to fix due to a chain of dependency upgrades that are required.

Verification

Verified it works by running the integration tests with Rhelemeter on debug level and a local OTEL collector receiving the traces.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[douglascamata]

Seeing some weird CI failures and trying to investigate:

error: failed to push image registry.build01.ci.openshift.org/ci-op-lf1nky9g/release:latest: unable to upload new layer (0): Patch "https://registry.build01.ci.openshift.org/v2/ci-op-lf1nky9g/release/blobs/uploads/1dd4cd97-9b41-4af5-9227-96247ddba9c6?_state=VPkkGVZBjSd6-JxdpBszU2B6HnAClvm_D3B4wuTrO-F7Ik5hbWUiOiJjaS1vcC1sZjFua3k5Zy9yZWxlYXNlIiwiVVVJRCI6IjFkZDRjZDk3LTliNDEtNGFmNS05MjI3LTk2MjQ3ZGRiYTljNiIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMy0xMC0wOVQwOTo0MDoxMy43MzEyNjU4MjlaIn0%3D": operator "cluster-network-operator" contained an invalid image-references file: no input image tag named "kuryr-cni"
{"component":"entrypoint","error":"wrapped process failed: exit status 1","file":"k8s.io/test-infra/prow/entrypoint/run.go:84","func":"k8s.io/test-infra/prow/entrypoint.Options.internalRun","level":"error","msg":"Error executing test process","severity":"error","time":"2023-10-09T09:40:14Z"} 

[douglascamata]

Found the reason and solution to the build failures in openshift/cluster-network-operator#2056. Waiting for it to be merged.

[douglascamata]

/retest-required

[douglascamata]

/test e2e-aws-upgrade

[philipgough]

/lgtm

[openshift-ci]

@douglascamata: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[philipgough]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: douglascamata, philipgough

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [douglascamata,philipgough]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[moadz]

vendor files should be .gitignore'ed