Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Bad Prometheus metric performance due to endpoint label explosion for images and feeds #4431

Open
4 tasks done
SorteKanin opened this issue Feb 6, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@SorteKanin
Copy link
Contributor

Requirements

  • Is this a bug report? For questions or discussions use https://lemmy.ml/c/lemmy_support
  • Did you check to see if this issue already exists?
  • Is this only a single bug? Do not put multiple bugs in one issue.
  • Is this a backend issue? Use the lemmy-ui repo for UI / frontend issues.

Summary

Prometheus metrics currently have performance issues due to label explosion of the endpoint label. This is handled for some endpoints (like endpoint="/comment/{comment_id} instead of using the actual ID) but not for others (like endpoint="/feeds/u/user@instance.xml" and endpoint="/pictrs/image/image_uuid.webp").

Steps to Reproduce

  1. Run pretty much any query involving lemmy_api_http_requests_duration_seconds_bucket. For instance histogram_quantile(0.99, sum by(le) (rate(lemmy_api_http_requests_duration_seconds_bucket[$__rate_interval])))
  2. Observe the very slow performance. The query times out for me if I set it for the last 24 hours.

Technical Details

It's not a bug in the sense that logs will help. It's a performance bug, not a correctness issue. The system is working as designed, just with poor performance.

Suggested solutions

Do the same as the endpoint="/comment/{comment_id} strategy for image and feed endpoints as well.

Version

BE 0.19.3

Lemmy Instance URL

Feddit.dk

@SorteKanin SorteKanin added the bug Something isn't working label Feb 6, 2024
@Nutomic
Copy link
Member

Nutomic commented Feb 7, 2024

I dont see any difference between these endpoint definitions in Lemmy. It seems to be an issue with the rust-prometheus library.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants