Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Modify loki config to address HTTP 429 code (too many reqs) #325

Merged
merged 3 commits into from Dec 1, 2023
Merged

Modify loki config to address HTTP 429 code (too many reqs) #325

merged 3 commits into from Dec 1, 2023

Conversation

sed-i
Copy link
Contributor

@sed-i sed-i commented Nov 22, 2023 •
edited

Issue

Following up on #324, as the problem still persists.

  • Getting HTTP codes 429 and others throughout the load test.
  • Loki logs 2023-11-22T20:28:31.034Z [loki] level=warn ts=2023-11-22T20:28:30.939304741Z caller=cache.go:114 msg="fifocache config is deprecated. use embedded-cache instead"

Solution

  • Enable embedded-cache.
  • Additional tweaks, per refs inline.

With this change, at high load (increase(loki_distributor_lines_received_total[1m]) of up to 320,000), all of the following issues are now gone:

  • 429 / GET / loki_api_v1_query_range
  • 499 / GET / loki_api_v1_query_range
  • 500 / GET / loki_api_v1_query_range
  • 503 / GET / ready
  • 500 / POST / loki_api_v1_push
  • cancel / gRPC / /logproto.Querier/Query
  • cancel / gRPC / /schedulerpb.SchedulerForQuerier/QuerierLoop

Fixes #283.

Context

Testing Instructions

juju refresh --switch loki-k8s --channel=edge/325 loki  # Expires at 2023-12-30T00:00:00Z

Release Notes

@sed-i sed-i marked this pull request as ready for review November 30, 2023 05:33
@simskij
Copy link
Member

simskij commented Nov 30, 2023 •
edited

Didn't we say we also wanted to expose config options, allowing users to override our defaults? I'm not reading too well apparently :D

@simskij simskij merged commit 326dc0a into main Dec 1, 2023
13 checks passed
@simskij simskij deleted the feature/limits_config branch December 1, 2023 14:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@simskij simskij simskij left review comments

@Abuelodelanada Abuelodelanada Awaiting requested review from Abuelodelanada Abuelodelanada is a code owner

@PietroPasotti PietroPasotti Awaiting requested review from PietroPasotti PietroPasotti is a code owner

@dstathis dstathis Awaiting requested review from dstathis dstathis is a code owner

@lucabello lucabello Awaiting requested review from lucabello lucabello is a code owner

Assignees
No one assigned
Labels
Libraries: Out of sync
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Configuration plans for Loki charm
2 participants
@sed-i @simskij