Ingester remains in the LEAVING state if starting() terminates

[duricanikolic]

What this PR does

If starting the ingester service (Ingester.starting()) fails for any reason, the corresponding ingester might be in the ring in an inconsistent state. For example, if an ingester receives the SIGTERM while opening TSDBs (replaying WAL), the TSDB opening intentionally doesn't stop, although the context is cancelled (to reduce the likelihood of on-disk corruptions). Nevertheless, the ring lifecycler gets started, and the ingester switches from LEAVING to ACTIVE state in the ring, although the Ingester.starting() function was interrupted due to the context cancellation. The ingester, although terminated, remains in the ACTIVE state because Ingester.stopping() wasn’t called after the failed Ingester.starting().

This PR fixes the problem explained above, by explicitly stopping ingester's lifecycler, in case of an error during Ingester.startint().

Which issue(s) this PR fixes or relates to

Fixes #6753

Checklist

• Tests updated.
• [na] Documentation added.
• [na] CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
• about-versioning.md updated with experimental features.

[pstibrany]

I think the test is still doing too much.

Why do we need to start and stop ingester first? We can setup the ring to initial state that we want to test.

I think we should also add a scenario, where there is no ring entry before ingester starts, and it fails on first start.

[pstibrany]

Why do we need to start and stop ingester first? We can setup the ring to initial state that we want to test.

Thinking about it, setting up the ring may end up being same amount of code :(

[duricanikolic]

I think we should also add a scenario, where there is no ring entry before ingester starts, and it fails on first start.

@pstibrany I can add this scenario, but this will additionally increase the size of this source.

[duricanikolic]

Why do we need to start and stop ingester first? We can setup the ring to initial state that we want to test.

Thinking about it, setting up the ring may end up being same amount of code :(

We can move all the auxiliary structs as well as methods used for tests' intialization in a separate source (e.g., ingester_util_test.go), in order to reduce the size of this source.

[pstibrany]

Fix lgtm.