osd: add osd_fast_shutdown option (default true)

[liewegas]

If we get a SIGINT or SIGTERM or are deleted from the OSDMap, do a fast
shutdown by exiting immediately. This has a few important benefits:

• We immediately stop responding (binding) to any sockets, which means
  other OSDs will immediately decide we are down (and dead!). This
  minimizes IO interruption.
• We avoid the complex "clean" shutdown process, which is historically a
  source of bugs.

In reality, the only purpose of the "clean" shutdown is to try to tear down
everything in memory so we can do memory leak checking with valgrind. Set
this option to false for valgrind QA runs so we can still do that.

Not that with the new read leases in octopus, we rely on the default
behavior that a ECONNREFUSED is taken to mean that the OSD is fully dead,
so that we don't have to wait for any leases to time out. This works in
sane environments with normal IP networks, but that behavior could
conceivably be a bad idea if there are some weird network shenanigans
going on. If osd_fast_fail_on_connection_refused were disabled, then this
fast shutdown procedure might be worse than the clean shutdown because
we would have to wait for the heartbeat timeout.

Signed-off-by: Sage Weil sage@redhat.com

[liewegas]

In my vstart testing this gives me a sub-second IO interruption when killing an OSD.

[jdurgin]

Seems like this would expose potential consistency bugs in the objectstore/rocksdb more frequently, so it would be good for master, but not backport without plenty of testing

[gregsfortytwo]

Hmm is that fast IO interruption going to scale okay to real clusters, since it's relying on the monitor OSD failure tracking?
The MarkMeDown message was historically important because it got the IO hiccup down to a normal OSDMap distribution, rather than being an event you could track just by looking at client IO latencies.

[liewegas]

Hmm is that fast IO interruption going to scale okay to real clusters, since it's relying on the monitor OSD failure tracking?
The MarkMeDown message was historically important because it got the IO hiccup down to a normal OSDMap distribution, rather than being an event you could track just by looking at client IO latencies.

That used to be true. These days (since luminous or kraken maybe?) when we get ECONNREFUSED we infer that the OSD process is really gone and mark it down immediately.

[leseb]

Are we going to backport this?

[liewegas]

Eventually, probably... but not in any hurry