Add Prometheus endpoint and metrics

[andybug]

Discussed a bit in #3294.

• Add a server for serving Prometheus metrics.
• Include a configuration block in the config file.
• Provide HTTP metrics on the API, along with process-level metrics and DB pool metrics.
• The inclusion of the Prometheus components is gated by the prom feature flag.
• Acquired port 10002 for Lemmy in the well-known Prometheus ports.

Follow-up Ideas

• Metrics on federation, capture per-peer
• DB query metrics
• Community metrics (sub count, posts)
• User metrics (user count, bot users, active users)

Example Metrics

The lemmy_api_* metrics will populate as requests come in to different endpoints. This is showing that only one request has come in so far and it was on the "/" endpoint.

# HELP lemmy_api_http_requests_duration_seconds HTTP request duration in seconds for all requests
# TYPE lemmy_api_http_requests_duration_seconds histogram
lemmy_api_http_requests_duration_seconds_bucket{endpoint="/",method="GET",status="200",le="0.005"} 0
lemmy_api_http_requests_duration_seconds_bucket{endpoint="/",method="GET",status="200",le="0.01"} 0
lemmy_api_http_requests_duration_seconds_bucket{endpoint="/",method="GET",status="200",le="0.025"} 1
lemmy_api_http_requests_duration_seconds_bucket{endpoint="/",method="GET",status="200",le="0.05"} 1
lemmy_api_http_requests_duration_seconds_bucket{endpoint="/",method="GET",status="200",le="0.1"} 1
lemmy_api_http_requests_duration_seconds_bucket{endpoint="/",method="GET",status="200",le="0.25"} 1
lemmy_api_http_requests_duration_seconds_bucket{endpoint="/",method="GET",status="200",le="0.5"} 1
lemmy_api_http_requests_duration_seconds_bucket{endpoint="/",method="GET",status="200",le="1"} 1
lemmy_api_http_requests_duration_seconds_bucket{endpoint="/",method="GET",status="200",le="2.5"} 1
lemmy_api_http_requests_duration_seconds_bucket{endpoint="/",method="GET",status="200",le="5"} 1
lemmy_api_http_requests_duration_seconds_bucket{endpoint="/",method="GET",status="200",le="10"} 1
lemmy_api_http_requests_duration_seconds_bucket{endpoint="/",method="GET",status="200",le="+Inf"} 1
lemmy_api_http_requests_duration_seconds_sum{endpoint="/",method="GET",status="200"} 0.01122634
lemmy_api_http_requests_duration_seconds_count{endpoint="/",method="GET",status="200"} 1
# HELP lemmy_api_http_requests_total Total number of HTTP requests
# TYPE lemmy_api_http_requests_total counter
lemmy_api_http_requests_total{endpoint="/",method="GET",status="200"} 1
# HELP lemmy_db_pool_available_connections Number of available connections in the pool
# TYPE lemmy_db_pool_available_connections gauge
lemmy_db_pool_available_connections 1
# HELP lemmy_db_pool_connections Current number of connections in the pool
# TYPE lemmy_db_pool_connections gauge
lemmy_db_pool_connections 1
# HELP lemmy_db_pool_max_connections Maximum number of connections in the pool
# TYPE lemmy_db_pool_max_connections gauge
lemmy_db_pool_max_connections 5
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1024
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 89
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 70168576
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1688323261
# HELP process_threads Number of OS threads in the process.
# TYPE process_threads gauge
process_threads 37
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 2602901504

[dessalines]

Could you provide some instructions with how to run this? Possibly including it in the docker-compose.yml ?

People did the same for otel, and still haven't provided any instructions on how to use them.

[Nutomic]

Generally looks good. Would be best if you can already write instructions for this in lemmy-docs, so we can review it together with the code before merging both.

[andybug]

Added documentation in the above PR.

[Nutomic]

Thanks!

[phiresky]

can this serve any metrics? https://docs.rs/tokio-metrics/latest/tokio_metrics/ would be very interesting. https://github.com/Hanaasagi/tokio-metrics-collector

[andybug]

Yes it can serve any metrics and will do so for metrics added to the default registry. Glancing at those links, I think the task metrics seem easy enough but the runtime ones require tokio unstable so probably not viable for quite some time.

[ubergeek77]

It doesn't look like the official images are compiling with this enabled. At least not as far as I can see in Woodpecker.

Am I mistaken, or is this a "compile only" feature for now?

[andybug]

It doesn't look like the official images are compiling with this enabled. At least not as far as I can see in Woodpecker.

Am I mistaken, or is this a "compile only" feature for now?

This is compile-only for now. I didn't want to introduce any potential bugs and performance problems to existing users since I expect that the number of people who want this feature is rather small.

If it gets some uptake maybe we can talk with the maintainers about making it part of the default build.

If you are interested in building Lemmy with this feature enabled, the details are in the documentation here.