Don't add the memberlist label as a selector to services.

[cstyan]

Adding this selector can result in outages caused by all write traffic being routed to only new distributors during initial migrations to memberlists, causing the few new distributors at the start of a rollout to be OOM killed.

cc @JordanRushing @vlad-diachenko @slim-bean

Signed-off-by: Callum Styan callumstyan@gmail.com

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

[chaudum]

Great find 🎉 !

[kavirajk]

Wow. great find. 🚀 Thanks @cstyan.

Curious how did you endup finding it?

[cstyan]

@kavirajk First I looked at the distributor services via kubectl describe service and noticed that our loki deployments had the selector but mimir deployments did not. Next I started looking for what could be different. I knew that we created the service from a deployment or statefulset via kausals serviceFor to create the service, and I noticed that it by default took all the labels from a deployment/statefulset and added them as selectors. However, the that serviceFor does take the optional ignoreLabels, so then I looked in Mimir's jsonnet for where they were adding those labels.

I missed it earlier when writing our memberlist.libsonnet because the definition of which labels to ignore was in a different file.

[kavirajk]

Nice thinking. thanks for the explanation @cstyan :)