🔊 Setup OpenTelemetry

[roma-glushko]

In order to collect metrics on different aspects of the system (e.g. how many times OpenAI failed/accepted requests), we need to setup OpenTelemetry.

Also, we could use OpenTelemetry to implemented distributed tracing, so it's seamlessly integrated into business services that communicates with it and instrumented by OTEL. Also, this should be useful for us to have a better debugging capabilities.

Requirements:

• setup OTEL
• make it possible to disable traces/metrics via config if that's unused by users
• emit gateway-specific metrics (instrument Fiber with OTEL)
• allow to control cardinality of metrics

Tasks

• 👷 [Telemetry] Setup local environment to view metrics and traces via Prometheus & Jaeger #219
• 🔧 [Telemetry] Instrument Glide HTTP API with OTEL #220
• 🔧 [Telemetry] Setup OTEL Tracer(s) and Meter(s) #221
• ✨ 🔧 [Telemetry] Add traces to lang chat workflows #222
• ✨ 🔧 [Telemetry] Add metrics to measure health, latency, request rate to model providers #223
• 🔧 [Telemetry] Add in-process gateway metrics #224