FSync translog outside of the writers global lock

[s1monw]

Today we aquire a write global lock that blocks all modification to the
translog file while we fsync / checkpoint the file. Yet, we don't necessarily
needt to block concurrent operations here. This can lead to a lot of blocked
threads if the machine has high concurrency (lot os CPUs) but uses slow disks
(spinning disks) which is absolutely unnecessary. We just need to protect from
fsyncing / checkpointing concurrently but we can fill buffers and write to the
underlying file in a concurrent fashion.

This change introduces an additional lock that we hold while fsyncing but moves
the checkpointing code outside of the writers global lock.

I had some conversation with @mikemccand who saw congestion on these locks. @mikemccand can you give this patch a go if you see better concurrency? I also think we need to run the powertester on this one again :)

[mikemccand]

Thanks @s1monw I will test this!

[mikemccand]

Maybe add a lock order comment here, e.g. "lock order syncLock -> synchronized(this)"?

[mikemccand]

LGTM

[mikemccand]

I think this change is potentially a biggish performance gain on highly concurrent CPUs with slowish IO.

I tested on 72 core box, on a spinning disk, and without the change (2 runs):

  67.3 M docs in 948.4 seconds
  67.3 M docs in 946.3 seconds

and 2 runs with the change:

  67.3 M docs in 858.1 seconds
  67.3 M docs in 884.7 seconds

I also ran the "random power loss tester" and there were no corruptions after 18 power loss events (at which point I hit disk full!).

[s1monw]

@mikemccand I pushed a new commit

[bleskes]

@mikemccand out of curiosity - how many concurrent threads did you use to index and how many shards are in the index?

[mikemccand]

LGTM, thanks @s1monw

[mikemccand]

how many concurrent threads did you use to index and how many shards are in the index?

6 shards, 12 client side threads, and I told ES to use 36 cores (though I did have a local mod to relax its current 32 core max). I disabled auto-IO merge throttling, and ran with translog durability async.

[mikemccand]

I re-ran perf test with default translog durability (request) fsync:

before.log.0: Indexer: 56952280 docs: 1139.35 sec [49986.8 dps, 16.0 MB/sec]
before.log.1: Indexer: 56952280 docs: 1133.59 sec [50240.5 dps, 16.1 MB/sec]
before.log.2: Indexer: 56952280 docs: 1156.01 sec [49266.4 dps, 15.8 MB/sec]

 after.log.0: Indexer: 56952280 docs: 1058.93 sec [53782.8 dps, 17.2 MB/sec]
 after.log.1: Indexer: 56952280 docs: 1064.17 sec [53518.0 dps, 17.2 MB/sec]
 after.log.2: Indexer: 56952280 docs: 1046.26 sec [54433.9 dps, 17.4 MB/sec]

This is powerful CPU (36 real cores) against slowish IO (single spinning 8 TB disk) ... I think there are real gains here.