Added framework for cluster integration testing

[dgnorton]

This PR adds the framework for cluster integration testing. It also fixes the following bugs:

• CircleCI intermittent raft consensus failures. This was due to having the -parallel argument passed into our test suite. Disabling that has no noticeable affect on how long it takes to run tests, but they no longer fail intermittently.
• Starting the raft store we now set the RemoteAddr. This is only necessary for testing due to spinning up on random system ports like localhost:0
• DatabaseNotFound could return different error text depending on where in the code branch it was determined that there was no database. This caused intermittent test results.
• Added ability to silence the Metastore RPC listener logging. This is only used for testing.

[otoolep]

Keep it up!

[pauldix]

Looks great so far. One thing I'd like is to have every query test represented at the integration level, which can be run on a single server, or different cluster configurations. You mentioned that's still in process, but it would be nice to be able to define the test and output in one place and run in various configs.

[corylanou]

@pauldix I'm not sure if I understand, but to give you an idea of how it currently works:

The suite stores the set of commands (writes, meta commands, queries, etc.) that it will run though. So, if we take a test that did something as simple as Create a database, write some data, and verify the data exists, it would do the following:

• Single server suite - Run as it always did.
• Cluster test:
  • Set up 5 nodes cluster
  • Issue meta commands to first server (currently hardcoded, but could be randomized in the future, not sure it really matters that much)
  • Write to a single server (In the future we will spread the writes across the cluster)
  • Walk through every server and issue the exact same query to make sure every server has the same answer.

Also, we could easily make the cluster tests cycle through and run the tests in single, 2 node, 3 node, 4 node, 5 node, 6 node, and 7 node configuration very easily. Currently, we have coverage in a single node and 5 node cluster.

Based on that, what are you proposing we do add?