When using --unsafe-no-fsync still write out the data

[cwedgwood]

There are situations where we don't wish to fsync but we do want to
write the data.

Typically this occurs in clusters where fsync latency (often the
result of firmware) transiently spikes. For Kubernetes clusters this
causes (many) elections which have knock-on effects such that the API
server will transiently fail causing other components fail in turn.

By writing the data (buffered and asynchronously flushed, so in most
situations the write is fast) and avoiding the fsync we no longer
trigger this situation and opportunistically write out the data.

Anecdotally:
Because the fsync is missing there is the argument that certain
types of failure events will cause data corruption or loss, in
testing this wasn't seen. If this was to occur the expectation is
the member can be readded to a cluster or worst-case restored from a
robust persisted snapshot.

The etcd members are deployed across isolated racks with different
power feeds. An instantaneous failure of all of them simultaneously
is unlikely.

Testing was usually of the form:

• create (Kubernetes) etcd write-churn by creating replicasets of
  some 1000s of pods
• break/fail the leader

Failure testing included:

• hard node power-off events
• disk removal
• orderly reboots/shutdown

In all cases when the node recovered it was able to rejoin the
cluster and synchronize.

Please read https://github.com/etcd-io/etcd/blob/master/CONTRIBUTING.md#contribution-flow.

[ptabor]

That is a good catch. Thank you.

Please convert the patch to be against the master branch.
Cherry pick might be considered later - when its in the HEAD.

Otherwise LGTM

[cwedgwood]

@ptabor this can't CP between 3.4 and master because things moved about. For 3.4 this should suffice. For master there is #12752 .

Travis CI shows an error here at present though I don't think it's a real problem. Does that need to be addressed?

[ptabor]

Thank you.