Add support for opentelemetry

[varmar05]

This PR adds basic monitoring to our docker compose. It can be optionally run for both community and enterprise edition as additional docker compose stack.

Dependencies are always installed in dockerfile but telemetry is enabled with OTEL_ENABLED flag. We use Opentelemetry (OTEL) collector as centralized receiver for metrics, logs and traces. They are then passed to backends and can be visualized in grafana via prometheus, loki or tempo respectively.

Logging was set up in a way that it contains trace id in logs so can be linked via derived fields with traces.

Traces were instrumented for flask app, celery worker app, celery beat and sqlalchemy. For production there is an option to set up sampling OTEL_TRACES_SAMPLER_ARG with parent relationship (e.g. flask-celery worker, celery beat - celery worker). For celery metrics there is special flag OTEL_MANUAL_CELERY_TRACING to enable in case of threads/gevent workers.

Metrics are collected from gunicorn, flask and celery. We replaced statsd sidecar container with statsd receiver in OTEL to get basic metrics like worker count. Standard http requests metrics are then scraped from flask workers directly. We also defined custom metrics for celery tasks and manually provisioned both metrics and traces as auto-provisioning was not working properly. There is also an option to scrape redis metrics, but it is only system related, there is no way to get custom metrics like celery queue length without custom exporter.

New env variables used (there is probably bunch of others coming from otel by default)

OTEL_METRICS_EXPORTER=otlp
OTEL_TRACES_EXPORTER=otlp
OTEL_LOGS_EXPORTER=otlp
OTEL_EXPORTER_OTLP_ENDPOINT=http://mergin-otel-collector:4317
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=http://mergin-otel-collector:4317
OTEL_METRIC_EXPORT_INTERVAL=10000
OTEL_TRACES_SAMPLER_ARG
OTEL_SERVICE_NAME
-- custom ones
OTEL_ENABLED
OTEL_PYTHON_FLASK_EXCLUDED_URLS
OTEL_MANUAL_CELERY_TRACING

[coveralls]

Pull Request Test Coverage Report for Build 23599234188

Details

• 1 of 141 (0.71%) changed or added relevant lines in 4 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage decreased (-1.4%) to 93.234%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
server/mergin/otel/config.py	0	7	0.0%
server/mergin/otel/instrument.py	0	49	0.0%
server/mergin/otel/app.py	0	84	0.0%

Totals
Change from base Build 23292697212:	-1.4%
Covered Lines:	9109
Relevant Lines:	9770

---

💛 - Coveralls