Cassandra refactor

[rslota]

This PR introduces new implementation of mongoose_cassandra. There are no changes to the API, so the code that uses Cassandra remains unchanged.
The new implementation of Cassandra layer moves all request processing to a worker pool (previously only asynchronous requests were handled this way). This reduces code duplication and improves error handling while also fixes some race conditions that were present in previous implementation.

This PR also adds mongoose_cassandra_SUITE that has some simple "cassandra-only" tests for mongoose_cassandra API. This test suite requires a TCP proxy to be active on top of Cassandra (to enable to connection manipulations like connection drop or connection latency).

[codecov]

Codecov Report

Merging #1847 into master will increase coverage by 0.09%.
The diff coverage is 72.72%.

@@            Coverage Diff            @@
##           master   #1847      +/-   ##
=========================================
+ Coverage    74.6%   74.7%   +0.09%     
=========================================
  Files         298     298              
  Lines       27957   28096     +139     
=========================================
+ Hits        20858   20988     +130     
- Misses       7099    7108       +9

Impacted Files	Coverage Δ
src/cassandra/mongoose_cassandra_sup.erl	96.29% <100%> (ø)
src/cassandra/mongoose_cassandra.erl	86.36% <100%> (ø)
src/cassandra/mongoose_cassandra_worker.erl	70.73% <70.73%> (ø)
..._distrib/mod_global_distrib_outgoing_conns_sup.erl	73.91% <0%> (-8.7%)	⬇️
src/mongoose_subhosts.erl	86.66% <0%> (-6.67%)	⬇️
src/mongoose_client_api_sse.erl	75% <0%> (-6.25%)	⬇️
src/mod_push_service_mongoosepush.erl	82.85% <0%> (-5.72%)	⬇️
src/mod_muc_light_utils.erl	85.26% <0%> (-3.16%)	⬇️
src/mod_muc_log.erl	77.69% <0%> (-0.26%)	⬇️
src/ejabberd_c2s.erl	84.44% <0%> (-0.22%)	⬇️
... and 9 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update af131fd...ed49eb7. Read the comment docs.

[kzemek]

I like the refactor in general, although it brings some questionable indentation changes (not because we care very much, but because some changes consist mostly of whitespace diff from the fine Emacs indentation style).

One point that I'd like to see addressed though - have you considered using https://github.com/inaka/worker_pool instead of extending the custom pooling solution?

[rslota]

@kzemek Actually, no, I haven't considered using worker_pool. This is part of the old implementation that I haven't changed since it was simple irrelevant to the changes in the general worker logic that were the main focus of this refactor.

I'll reformat the code with emacs, I have no idea when and how the code got reformatted. Also, I'll look into the viability of using worker_pool. If it's gonna simplify the code, I'll go for it.

[rslota]

@kzemek I have reformatted the code with emacs, but introducing worker_pool is not the best idea right now. The changes are not as simple as removing the custom supervisor since worker_pool doesn't allow for nested calls (i.e. worker selects another worker) - such calls may end up in deadlock.
I still think that using worker_pool will be cleaner, but it requires non-trivial changes to the worker's logic, therefore I would prefer to do it as a separate PR.