Export Prometheus Metrics

[alecmerdler]

Description

Use the Prometheus Python client to export metrics for monitoring the performance of Echo, which will be especially useful when running at scale in a multi-tenant environment (https://lightning.ai/echo).

Some metrics could include:

• Number of processing Echoes
• Total number of completed Echoes
• Current number of Recognizer/YouTuber Works running
• etc...

[manskx]

@alecmerdler If we create a route in configure_api() that exposes the needed metrics, would this be useful?

[alecmerdler]

@manskx I considered and half-implemented a /metrics endpoint which queried the database for some basic information, but I realized that we get a lot more value by adding Prometheus properly (Grafana, per-node metrics, FastAPI integration, etc). Let's just commit to Prometheus: the client looks easy enough to adopt and we can start with basic metrics and always expand later.

[tchaton]

@alecmerdler I was using this one: https://github.com/trallnag/prometheus-fastapi-instrumentator and started to build Prometheus and Graphana works, but we need to install python within their images, so the work can start.