Enable emulated multi-node testing in CI

[ChristopherHogan]

Closes #116.

For local cluster emulation:

# Prerequisites
which docker
which docker-compose

pushd /path/to/hermes/ci/cluster
# Start a cluster
./cluster_up.sh
# Run bin/end_to_end_test
./cluster_test.sh
# Stop the cluster
./cluster_down.sh

Currently this only runs test/end_to_end_test.cc on 2 nodes, but in the future I plan to integrate this into a custom CMake target to make it easy to run any test on the cluster. We can add a whole suite of cluster tests.

• Update CI to run on Ubuntu 20.04 runners.
• Remove third party code from coverage results.
• Stop the system view state update thread before shutting down any rpc clients or servers (because that thread could be in the middle of an RPC).
• Add a global mutex for operations that must be synchronized across ranks on all nodes. So far I'm only using this for GetOrCreateBucketId.

[coveralls]

Pull Request Test Coverage Report for Build 624790611

• 45 of 45 (100.0%) changed or added relevant lines in 4 files are covered.
• 11 unchanged lines in 6 files lost coverage.
• Overall coverage increased (+4.1%) to 85.556%

---

Files with Coverage Reduction	New Missed Lines	%
src/data_placement_engine.cc	1	94.04%
src/hermes_status.h	1	90.0%
src/memory_management.h	1	96.0%
src/rpc_thallium.cc	2	75.71%
src/metadata_management.cc	3	86.4%
src/rpc_thallium.h	3	93.75%

Totals
Change from base Build 611665731:	4.1%
Covered Lines:	4312
Relevant Lines:	5040

---

💛 - Coveralls

[ChristopherHogan]

I'm going to remove the global mutex and implement Hari's single-RPC suggestion. I'll open a new PR.

[ChristopherHogan]

I've changed my mind. I'm reopening this PR because the RPC refactor to avoid global mutexes will be somewhat significant. I'd rather focus on getting the automated cluster testing in now, and have a separate branch for the RPC work.

[jya-kmu]

I am thinking if mutex can be named with the desired operation, for example, global_***_mutex for better understanding. Current local mutex is named as blob_***, bucket_***, vbucket_***. It might be better name the global mutex the same way with global_ prefix.

[ChristopherHogan]

I can change it if you want, but I plan on removing the global mutex altogether in my next PR. This is temporary just to get the multi-node testing code into the repo.

[jya-kmu]

I see. That's sounds good.