Move call of stopIncomingRequests before lifecycler shutdown.

[pstibrany]

Lifecycler was recently refactored (by #2166), which unfortunately breaks some expectations in Loki. This PR makes fixing Loki possible.

Before #2166, Lifecycler's Shutdown method called flushTransferer.StopIncomingRequests() and only then stopped lifecycler's loop.

After #2166, this has been modified. Lifecycler is now stopped via StopAsync method (through embedded BasicService), which cancels context used by loop() method, which makes loop() exit. stopping() method is called afterwards. This method still calls flushTransferer.StopIncomingRequests(), but by this time, loop has already finished.

In Cortex, this doesn't make much difference – only Ingester implements StopIncomingRequests method with non-empty body, and it simply sets the readonly flag. In Loki, StopIncomingRequests blocks until data transfer has finished, but it expects loop to be still running.

Solution to the issue is removing StopIncomingRequests from FlushTransferer interface – logically, it doesn't belong there anyway – and let Ingester call it explicitely before stopping the lifecycler. This reverts the method call order back to original.

Why doesn't StopIncomingRequests belong to FlushTransferer? All other FlushTransferer methods are called by lifecycler after main lifecycler loop has exited, during lifecycler's shutdown process. StopIncomingRequests was the only method that was called before triggering lifecycler's shutdown procedure.

(Fix in Loki is the same – call StopIncomingRequests from Ingester's shutdown method, before stopping the lifecycler. I have patch, incl. Cortex update ready.)

cc @rfratto

[pracucci]

I'm joining the party late, but LGTM to me too 😉