Improve mem3_sync event listener performance

[b20n]

There are three fundamental changes here, reflected in the latter three commits. These changes were motivated by observing and profiling mem3_sync's event listener pid during high (10k per second) write throughput load tests on databases with high (300+) q values.

We could conceivably pick and choose any of the three approaches here. The read_concurrency flag seems like a no-brainer and the ets:select/2 switch looks like a win in my limited testing.

I'm ambivalent about the frequency/delay work; it turned out surprisingly subtle (to avoid dropping the "last" update on an otherwise-idle host) and required more LOC than I'd preferred, since the simplest solution wasn't compatible with the way the event listener was being spawned previously.

COUCHDB-2984

[kocolosk]

Agreed, this makes sense.

[kocolosk]

I think the frequency/delay work has merit; executing superfluous mem3_sync:push operations has a measurable cost.

[mikewallace1979]

This looks like a good set of changes to me though I'm dubious whether I am
qualified to give a +1 here.

My only comments concern 5bf9317:

• It might be helpful to have some module-level comments that give a few
  pointers to the reader on the underlying implementation.
• The mechanism for ensuring all bucketed shards are pushed in the event of
  no further updates might not be obvious to all future maintainers. It might
  be worth explicitly noting that the gen_server timeout is being set by the
  return value of the callbacks (via maybe_push_shards/1) and that the timeout
  will trigger the handle_info(timeout, St) clause.
• I can't see how the delay is actually used other than when rebucketing
  the shards in response to a config change.

It also feels like these new config parameters should be documented somewhere
though I'm not exactly sure where that should be.

[b20n]

@mikewallace1979: I've added a couple of comment blocks to address your first two points. Regarding the third, see here. Configuring "bucket count" would be a more direct representation of the implementation, but I believe "delay" should be simpler for operators because it's directing behavior regardless of implementation details.

[mikewallace1979]

@banjiewen Ah that makes sense and I agree that "delay" fits better with operator's understanding of the system than "bucket count".

+1 from me.

[b20n]

@kocolosk: Mind giving a vote on this one?

[kocolosk]

+1 on the code review. I've not tested the changes myself. I understand that at least some of them have been deployed to test clusters and have yielded substantial improvements. Have all the commits been through that testing? If so, let's merge this one.

[b20n]

@kocolosk: This exact changeset hasn't been tested, but a functionally equivalent one has; the Cloudant mem3 isn't quite in sync with the Apache mem3. Performance results were indeed quite positive.

/cc @chewbranca

[b20n]

Squashed.