proxy: OpenTelemetry tracing.

[hlinnaka]

Thanks to commit XXXX ("Refactor common parts of handle_client and handle_ws_client to function."), we now have separate tracing spans for the connection establishment phase and for the forwarring phase of each connection. This commit sets up OpenTelemetry tracing and exporter, so that they can be exported as OpenTelemetry traces as well.

This adds tracing to all outgoing HTTP requests. A separate (child) span is created for each outgoing HTTP request, and the tracing context is also propagated to the server in the HTTP headers. Use the 'reqwest-middleware' crate to do that.

If tracing is enabled in the control plane and compute node too, you can now get an end-to-end distributed trace of what happens when a new connection is established, starting from the handshake with the client, creating the 'start_compute' operation in the control plane, starting the compute node, all the way to down to fetching the base backup and the availability checks in compute_ctl.

Note: this is on top of PR #3433

[funbringer]

I don't understand why this can't be based on main instead of #3433. Adding opentelemetry tracing is one thing, but tuning the spans to your liking is the other. We could add the support right now as is and then do other refactorings.

[hlinnaka]

I don't understand why this can't be based on main instead of #3433. Adding opentelemetry tracing is one thing, but tuning the spans to your liking is the other. We could add the support right now as is and then do other refactorings.

Without having the spans right, the opentelemetry traces will be less useful. But sure, we could make these changes the other way round, too.

[funbringer]

Without having the spans right, the opentelemetry traces will be less useful.

Agreed. Luckily, #3331 has been merged, so we now have much better spans. I'll gladly merge this PR if you rebase it on top of main.

[LizardWizzard]

LGTM

Saw new env var OTEL_EXPORTER_OTLP_ENDPOINT and started thinking again about different config sources, some parameters come from cli flags, some from env. Would be nice to eventually merge different sources into one

[funbringer]

Ok, so now we can easily capture traces for the proxy's tests.

How to reproduce:

# https://www.jaegertracing.io/docs/1.42/getting-started/
docker run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  -p 14250:14250 \
  -p 14268:14268 \
  -p 14269:14269 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.42

# important: set otel endpoint
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318

# run proxy's tests
poetry run pytest test_runner/regress/test_proxy.py

# open jaeger's UI in a web browser
xdg-open http://localhost:16686

[LizardWizzard]

Ok, so now we can easily capture traces for the proxy's tests.

Nice! Worked for me

[kelvich]

Nice.

@lassizci I haven't followed closely story about traces, but I see that we have tempo source in Grafana now. What are the steps to looks at that traces in grafana?