Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Helm: fix distributor load balancing not working on scale up #2862

Merged
merged 1 commit into from
Aug 31, 2022

Conversation

krajorama
Copy link
Contributor

@krajorama krajorama commented Aug 30, 2022

What this PR does

Which issue(s) this PR fixes or relates to

[from private repo]

the Deployment was scaled from 2 to 5 instances, but gRPC will only re-resolve when an existing connection is broken, so it carried on using the 2.
This blog post describes the background: https://majidfn.com/blog/grpc-load-balancing/

Relates to: #2206

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@krajorama krajorama marked this pull request as draft August 30, 2022 08:55
@krajorama krajorama added the helm label Aug 30, 2022
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@krajorama krajorama marked this pull request as ready for review August 30, 2022 10:53
Comment on lines -24 to -27
server:
grpc_server_max_connection_age:
grpc_server_max_connection_age_grace:
grpc_server_max_connection_idle:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🥳

operations/helm/charts/mimir-distributed/CHANGELOG.md Outdated Show resolved Hide resolved
The new defaults mean that gRPC will rebalance connections periodically,
letting new servers have connections on scale-out.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Co-authored-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
@krajorama krajorama merged commit 957c33b into main Aug 31, 2022
@krajorama krajorama deleted the krajo/20220830-grpc-lb branch August 31, 2022 17:16
grafanabot pushed a commit that referenced this pull request Sep 7, 2022
)

The new defaults mean that gRPC will rebalance connections periodically,
letting new servers have connections on scale-out.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
(cherry picked from commit 957c33b)
krajorama pushed a commit that referenced this pull request Sep 7, 2022
) (#2901)

The new defaults mean that gRPC will rebalance connections periodically,
letting new servers have connections on scale-out.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
(cherry picked from commit 957c33b)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants