Skip to content
This repository has been archived by the owner on Jan 12, 2022. It is now read-only.

loki|cortex: tweak grpc settings #1362

Merged
merged 2 commits into from
Sep 3, 2021
Merged

loki|cortex: tweak grpc settings #1362

merged 2 commits into from
Sep 3, 2021

Conversation

sreis
Copy link
Contributor

@sreis sreis commented Sep 2, 2021

While triaging for #1226 I saw the following error printed by loki querier:

msg="error processing requests" address=X.X.X.X:9095 err="rpc error: code = Unknown desc = closing transport due to: connection error: desc = \"error reading from server: EOF\", received prior goaway: code: ENHANCE_YOUR_CALM, debug data: too_many_pings"

Some googling later and I found the following PRs to tweak the grpc keepalive config:

grafana/cortex-jsonnet#233
https://github.com/grafana/loki/pull/4182/files

@jgehrcke
Copy link
Contributor

jgehrcke commented Sep 2, 2021

code: ENHANCE_YOUR_CALM

this is beautiful. 🌈

@sreis
Copy link
Contributor Author

sreis commented Sep 2, 2021

PR builds hit #1182 and #1126 but triggering new tests to double-check it's not adding new instability.

Handle the following error:

msg="error processing requests" address=1.2.3.4:9095
err="rpc error: code = Unknown desc = closing transport due to:
connection error: desc = \"error reading from server: EOF\",
received prior goaway: code: ENHANCE_YOUR_CALM, debug data: too_many_pings"

Signed-off-by: Simão Reis <sreis@opstrace.com>
Signed-off-by: Simão Reis <sreis@opstrace.com>
@sreis
Copy link
Contributor Author

sreis commented Sep 3, 2021

All tests are green, and it doesn't seem to increase the error rate of any of the known flakes (for example, #1126).

@sreis sreis merged commit 6bb1f06 into main Sep 3, 2021
@sreis sreis deleted the sreis/tweak-grpc-settings branch September 3, 2021 10:47
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants