Reduce default distributor - ingester push timeout to 2s #2728
Conversation
56quarters
force-pushed
the
56quarters/shorter-ingester-timeout
branch
from
13c35d1
to
ead3c03
Compare
56quarters
force-pushed
the
56quarters/shorter-ingester-timeout
branch
from
ead3c03
to
3fd10e3
Compare
|
I think it would make sense to also update the default of Otherwise requests that are being forwarded to a slow forwarding target could lead to requests being held in memory for a duration longer than the value of |
|
I think we should merge this change only after it has been dogfooded at Grafana Labs (aka: running in all prod envs). We're used to change defaults only after that. |
Sounds good. I'll test in dev/ops and add something to the next release jsonnet so that the change can go out a week after the new metrics for chunk-deduplication #2713. |
|
thanks for doing this in Mimir so we don't have to deal with keeping sync in jsonnet + helm! |
Reduce the default timeout to 2s from 20s since some resources from the request to each ingester are held in memory of the distributor until the ingester responds or the request times out. This reduces distributor memory usage during ingester crashes. Fixes #2727 Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>
Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>
Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>
56quarters
force-pushed
the
56quarters/shorter-ingester-timeout
branch
from
ab8575b
to
4e86cf9
Compare
| @@ -9,6 +9,7 @@ | |||
| * [CHANGE] Ingester: removed deprecated `-blocks-storage.tsdb.isolation-enabled` option. TSDB-level isolation is now always disabled in Mimir. #2782 | |||
| * [CHANGE] Compactor: `-compactor.partial-block-deletion-delay` must either be set to 0 (to disable partial blocks deletion) or a value higher than `4h`. #2787 | |||
| * [CHANGE] Query-frontend: CLI flag `-query-frontend.align-querier-with-step` has been deprecated. Please use `-query-frontend.align-queries-with-step` instead. #2840 | |||
| * [CHANGE] Distributor: change the default value of `-distributor.remote-timeout` to `2s` from `20s` to improve distributor resource usage when ingesters crash. #2728 | |||
There was a problem hiding this comment.
Should also mention -distributor.forwarding.request-timeout
Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>
Signed-off-by: Nick Pillitteri nick.pillitteri@grafana.com
What this PR does
Reduce the default timeout to 2s from 20s since some resources from the
request to each ingester are held in memory of the distributor until the
ingester responds or the request times out. This reduces distributor
memory usage during ingester crashes.
Which issue(s) this PR fixes or relates to
Fixes #2727
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]