XDR: fsync on close (also bucket dir), with option to disable.

[graydon]

Description

Resolves #2202

Checklist

• Reviewed the contributing document
• Rebased on top of master (no merge commits)
• Ran clang-format v5.0.0 (via make format or the Visual Studio extension)
• Compiles
• Ran all tests
• If change impacts performance, include supporting evidence per the performance document

Performance is worse but not hugely. Benchmarks that hammer on the bucket system non-stop run about 10x slower wall time on local NVMe and 100x worse on a spindle. And at the very smallest end of the merge times, slowdown is indeed also 10x-100x: 1ms merge becomes 10ms on NVMe or even 100ms on spindle. At level 0, the 100x case is still only taking 1-3% of allowed time for the merge -- we had a ton of headroom -- but it's worse for sure.

However, it also amortizes-away pretty smoothly as we get into bigger merges. By level 4 we're back down to 0.03% of available time and at level 5 down to 0.009%. That is: the fsync is adding a definitely-noticable 100ms to a merge running on the spindle, but it's not adding proportionally much more as we scale to somewhat larger buckets.

To measure whether this vanishes into the noise completely when scaling up (or if, say, full buffers / actually having some real data to write changes things again), I made a synthetic 1.4gb XDR test and ran it against the spindle in a loop. This gives not super wonderful but also not super deadly results. The large files do definitely show different effects that are not "constant 100ms totally vanishing into the noise":

2019-07-18T14:56:41.781 <test> [Fs INFO] wrote 1410837608 bytes to fsync file in 26094ms
2019-07-18T14:56:52.930 <test> [Fs INFO] wrote 1410837608 bytes to no-fsync file in 11149ms
2019-07-18T14:57:28.145 <test> [Fs INFO] wrote 1410837608 bytes to fsync file in 35214ms
2019-07-18T14:57:40.637 <test> [Fs INFO] wrote 1410837608 bytes to no-fsync file in 12491ms
2019-07-18T14:58:01.238 <test> [Fs INFO] wrote 1410837608 bytes to fsync file in 20600ms
2019-07-18T14:58:13.760 <test> [Fs INFO] wrote 1410837608 bytes to no-fsync file in 12522ms
2019-07-18T14:58:44.089 <test> [Fs INFO] wrote 1410837608 bytes to fsync file in 30328ms
2019-07-18T14:58:56.132 <test> [Fs INFO] wrote 1410837608 bytes to no-fsync file in 12042ms
2019-07-18T14:59:24.591 <test> [Fs INFO] wrote 1410837608 bytes to fsync file in 28458ms
2019-07-18T14:59:36.892 <test> [Fs INFO] wrote 1410837608 bytes to no-fsync file in 12301ms
2019-07-18T14:59:56.876 <test> [Fs INFO] wrote 1410837608 bytes to fsync file in 19983ms
2019-07-18T15:00:09.571 <test> [Fs INFO] wrote 1410837608 bytes to no-fsync file in 12694ms
2019-07-18T15:00:41.030 <test> [Fs INFO] wrote 1410837608 bytes to fsync file in 31459ms
2019-07-18T15:00:52.665 <test> [Fs INFO] wrote 1410837608 bytes to no-fsync file in 11634ms
2019-07-18T15:01:13.054 <test> [Fs INFO] wrote 1410837608 bytes to fsync file in 20388ms
2019-07-18T15:01:24.972 <test> [Fs INFO] wrote 1410837608 bytes to no-fsync file in 11917ms
2019-07-18T15:01:54.835 <test> [Fs INFO] wrote 1410837608 bytes to fsync file in 29863ms
2019-07-18T15:02:06.456 <test> [Fs INFO] wrote 1410837608 bytes to no-fsync file in 11620ms
2019-07-18T15:02:26.358 <test> [Fs INFO] wrote 1410837608 bytes to fsync file in 19901ms
2019-07-18T15:02:39.802 <test> [Fs INFO] wrote 1410837608 bytes to no-fsync file in 13444ms

I.e. varies between about 1.5x and 3x.

Back in the world of NVMe (which we certainly recommend people use for their buckets) this sort of thing is much less severe, though still measurable:

2019-07-18T15:08:01.569 <test> [Fs INFO] wrote 1410803760 bytes to fsync file in 11609ms
2019-07-18T15:08:15.889 <test> [Fs INFO] wrote 1410803760 bytes to no-fsync file in 14319ms
2019-07-18T15:08:28.031 <test> [Fs INFO] wrote 1410803760 bytes to fsync file in 12141ms
2019-07-18T15:08:39.261 <test> [Fs INFO] wrote 1410803760 bytes to no-fsync file in 11230ms
2019-07-18T15:08:51.141 <test> [Fs INFO] wrote 1410803760 bytes to fsync file in 11880ms
2019-07-18T15:09:02.121 <test> [Fs INFO] wrote 1410803760 bytes to no-fsync file in 10979ms
2019-07-18T15:09:14.038 <test> [Fs INFO] wrote 1410803760 bytes to fsync file in 11916ms
2019-07-18T15:09:25.084 <test> [Fs INFO] wrote 1410803760 bytes to no-fsync file in 11045ms
2019-07-18T15:09:37.012 <test> [Fs INFO] wrote 1410803760 bytes to fsync file in 11927ms
2019-07-18T15:09:48.308 <test> [Fs INFO] wrote 1410803760 bytes to no-fsync file in 11296ms
2019-07-18T15:10:00.282 <test> [Fs INFO] wrote 1410803760 bytes to fsync file in 11973ms
2019-07-18T15:10:11.388 <test> [Fs INFO] wrote 1410803760 bytes to no-fsync file in 11105ms
2019-07-18T15:10:23.253 <test> [Fs INFO] wrote 1410803760 bytes to fsync file in 11864ms
2019-07-18T15:10:34.470 <test> [Fs INFO] wrote 1410803760 bytes to no-fsync file in 11217ms
2019-07-18T15:10:46.402 <test> [Fs INFO] wrote 1410803760 bytes to fsync file in 11931ms
2019-07-18T15:10:57.530 <test> [Fs INFO] wrote 1410803760 bytes to no-fsync file in 11127ms
2019-07-18T15:11:09.532 <test> [Fs INFO] wrote 1410803760 bytes to fsync file in 12001ms
2019-07-18T15:11:20.751 <test> [Fs INFO] wrote 1410803760 bytes to no-fsync file in 11219ms
2019-07-18T15:11:32.724 <test> [Fs INFO] wrote 1410803760 bytes to fsync file in 11972ms
2019-07-18T15:11:44.059 <test> [Fs INFO] wrote 1410803760 bytes to no-fsync file in 11335ms

[MonsieurNicolas]

I think that the file and dir flush code have to be moved into fs.cpp as there is non portable things in there: (at a minimum turn those into noop on platforms that don't support those things) on Windows, you have to rename with the MOVEFILE_WRITE_THROUGH flag for https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-movefileexa and for files, it's https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-flushfilebuffers (on a native handle)

[MonsieurNicolas]

just need to adjust the PR to not break Windows (and possibly other OSes)

[graydon]

Pushed update to have windows variants fleshed out. Will do some testing on OSX and BSD tomorrow.

[graydon]

OSX uncovered a bug. pushed fix. moving to BSD.

[graydon]

BSD works (well -- there's a failure but it is also on master, no change with this PR). @MonsieurNicolas I think this is ready to land unless you have any other nits.

[MonsieurNicolas]

Looks good, only one minor miss from what I see (can't really use the new flag)

[MonsieurNicolas]

r+ 7cafbba