feat(server): OpenTelemetry integration

[mertalev]

Description

This PR greatly simplifies profiling performance issues by using OpenTelemetry. The current state of the PR is the addition of a /metrics endpoint for Prometheus that includes duration, count and sum metrics for repository methods and HTTP requests as well as host metrics. Additionally, instrumentation for Nest, Postgres and Redis is available if an OTLP exporter is configured through env variables.

To do:

• Convert these spans to metrics so Prometheus can scrape them
• Expose specific configuration settings as env variables
• Disable all of this by default

[cloudflare-pages]

Deploying with    Cloudflare Pages

Latest commit:	90166eb
Status:	✅  Deploy successful!
Preview URL:	https://43e1673d.immich.pages.dev
Branch Preview URL:	https://feat-server-otlp.immich.pages.dev

View logs

[jrasm91]

Would it make sense to have this be something that is turned on or off from the administration settings? Can it be controlled at runtime?

[bo0tzz]

To me this is something that is technical enough (also requiring deployment of other tools in the environment and such) that enabling it through an env var seems more logical.

[danieldietzler]

Awesome stuff! Really looking forward to this

[jrasm91]

To me this is something that is technical enough (also requiring deployment of other tools in the environment and such) that enabling it through an env var seems more logical.

What other tools are required? You basically just need to point a pre-existing prom instance at the /metrics endpoint.

[mertalev]

It looks like the functionality I'm looking for to turn spans to metrics comes with the Span Metrics Connector. But that would mean pushing to an OpenTelemetry Collector and integrating that with Prometheus. Hmm.

[mertalev]

Alright, after some wrestling I finally managed to export the execution time metrics to Prometheus. It uses histograms for metrics rather than spans/traces.

[mertalev]

Also, anything returning a Promise has now been made async because this is the only way for the decorator to know it should await the call.

[mertalev]

Would it make sense to have this be something that is turned on or off from the administration settings? Can it be controlled at runtime?

The SDK can be turned on or off, but it also has a bunch of env variables built into it already so any configuration we add is kinda redundant.

[danieldietzler]

Great stuff!

[alextran1502]

Mert, please feel free to merge this after resolving the conflicts

[jrasm91]

Would it make sense to have this be something that is turned on or off from the administration settings? Can it be controlled at runtime?

The SDK can be turned on or off, but it also has a bunch of env variables built into it already so any configuration we add is kinda redundant.

I just like being able to turn it on without having to restart the instance. I'm thinking of a troubleshooting situation where I turn on the sdk from the Admin Settings, then go to the problematic view, generate the monitoring, then go back and turn it off. It just seems like it could be more convenient than having to set and ENV and restart it, but that's just my opinion at least.

[mertalev]

Would it make sense to have this be something that is turned on or off from the administration settings? Can it be controlled at runtime?

The SDK can be turned on or off, but it also has a bunch of env variables built into it already so any configuration we add is kinda redundant.

I just like being able to turn it on without having to restart the instance. I'm thinking of a troubleshooting situation where I turn on the sdk from the Admin Settings, then go to the problematic view, generate the monitoring, then go back and turn it off. It just seems like it could be more convenient than having to set and ENV and restart it, but that's just my opinion at least.

That makes sense - we can possibly integrate this into the config repo in a later PR. But I think there are some questions to answer around how the SDK actually handles reconfiguration and restarts, whether it's okay to always wrap repo methods with the instrumentation decorator so it can be turned on or off dynamically, etc.