chore: Integrate tracing tool

[mstrug]

Background

Currently we are using metrics and logs for the observability and analysis of any ongoing issues or alerts. These two solutions can be updated with 3rd one: decentralized tracing which can support us with quicker insights to debug issues, identifying performance bottlenecks, streamlining the debugging and troubleshooting process.

Details

How this can be achieved using decentralized tracing?

• Tracing application UI shows graph (Flamegraph + Gantt) of the flow of an request/events (depending what we want to use it for) across all microservices in context of execution time of each instrumented function. So it is possible to easily see where the request (order/quote) execution got stuck.

What it could improve?

• No need to manually search through logs with various ids on different service log formats - instead using one id (like request id, or other) will show the complete flow graph
• Verification of execution times - no need to look over timestamp calculating the time differences - this can be captured easily looking over graph

Proposed approach:

• We can integrate some simple tracing app (for instance Zipkin which just runs as a docker container with practically no setup) to use with Playground, instrument couple of crucial functions in quote-order scenario (this will required some coding to support app library, for Zipkin we could use opentelemetry tracing rust integration)
• Team can play with this setup and decide if this gives us any process improvements and if we want to integrate it with our backend on production and which tracing solution to use (there are many of them free and paid)

Acceptance criteria

Troubleshooting process improved - flow of a request can be analyzed more easily.

[MartinquaXD]

This suggestion was already made in the past and I think it still makes a lot of sense. One thing I'm slightly worried about is that most of the examples I saw integrated the open telemetry exported into the application itself.
Currently our setup works such that our pods simply log to stdout and a log collector forwards them to elasticsearch. I think ideally we'd have a similar setup with zipkin or jeager.

[MartinquaXD]

Looks like jaeger supports opensearch as a log storage backend. So it seems like we mostly have to switch to structured logs (which are supported by the current tracing_subscriber) and use the expected log format to integrate it nicely into our current logging infra.

[github-actions]

This issue has been marked as stale because it has been inactive a while. Please update this issue or it will be automatically closed.