Develop 3.0 292

[martinsumner]

The primary change here is in list_dir. There was a hacked version of list_dir implemented previously in bitcask - b972602. This was presumably to handle bad scheduler behaviour with fileops in legacy OTP versions.

This no longer works in OTP 22. However, in OTP 22, file I/O has been overhauled to improve the scheduler situation. So - the OTP list_dir is now used again.

I'm going to run a volume test with bitcask before adding this in - to compare OTP 22 to OTP 20.

[martincox]

👍

[martinsumner]

The volume test didn't give the results I was expecting. I did a mix of GET/PUT/UPDATE/DELETE over 24 hours on an 8-node Riak KV cluster. I had default merge settings - so as merge kicked in this was the dominant factor.

All three effectively had the same overall throughput by the end of the 24 hours (< 2% difference between the runs) - with OTP achieving the most. In OTP22 the merge activity was over much faster, but had a more brutal impact on throughput.

[martincox]

That's quite interesting on the OTP22 run, and something to bear in my mind when we eventually go in that direction. When we've run OTP20 previously, we've generally seen a much higher gain in terms of cluster throughput - ~25% compared to R16.

[martincox]

Any idea what changed between 20 and 22 that might be responsible for that bump in the disk throughput? It's like some bottleneck has been eased around the file subsystem and it smashes IO much harder.

Had a quick peruse through the OTP20 changelog but nothing really sticks out.

[martinsumner]

There are two possible things.

In this PR there is the change to list_dir - which previously used direct access to functions no longer exported in OTP22 to try and handle scheduler issues. We now use the direct exported function (as OTP 22 has various async I/O scheduling improvements).

We've found in other testing that those async I/O and scheduling improvements have allowed us to push things faster when the disk was flat-out (like squeeze and extra 5% throughput).

I wouldn't expect either of these changes to be so dramatic.