Reduce ingester churn during rolling restart

[csmarchbanks]

Is your feature request related to a problem? Please describe.
Whenever I update ingesters or the configuration for ingesters samples are spread to other ingesters by design. This causes very small and inefficient chunks to be created in an unneeded ingester when running with a reasonable replication factor and a WAL. In addition, if an ingester crashes for some reason (typically an OOM), spreading additional data to other ingesters can amplify the problem.

Describe the solution you'd like
A way to configure ingesters not to remove themselves from the ring when they are being restarted, and for distributors not to extend their replica set in the presence of a leaving or unhealthy ingester.

Also, happy to hear if this is a terrible idea :)

Describe alternatives you've considered
Keep the status quo and try to reduce how often ingesters are updated.

Additional context
This request is very similar to #2823, but for ingesters instead of the store-gateway.

[pracucci]

This is something we recently discussed internally at Grafana Labs and we would like to do too. I haven't looked at the PR you opened yet, but I will do it soon!