[consul] Add support for consul 1.0.0

[byronwolfman]

What does this PR do?

Consul 1.0.0 is the first version that Hashicorp considers "stable" in
terms of API calls. As such, previous versions of consul may return
different payloads, and the API calls that the Datadog agent currently
uses to determine cluster leadership no longer work on 1.0.0 and later.

The current method of determining cluster leadership is to retrieve the
local agent's IP:Port combination and compare it to the IP:Port of the
leader. While consul 0.7.0 provides a more robust way of determining
leadership (literally a true/false key), this method is not available
in consul 0.6.4, so using it would break compatibility. Therefore this
commit makes the smallest possible change to avoid breaking that
compatibility, by instead using a different method of fetching the
local agent's IP:Port combination.

Tests have been updated (and somewhat refactored) and now supports
versions 0.6.4, 0.7.2, and 1.0.0.

Technical details

As noted above, the consul API has not guaranteed stability until version 1.0.0. In previous versions of consul, querying /v1/agent/self would among other things return:

{ 'Config': { 'AdvertiseAddr': '1.2.3.4', 'Ports': { 'Server': 8300 }}}

This works for consul versions 0.6.4 and 0.7.2, but as of version 1.0.0 (and possibly earlier, I haven't checked!) the address and server port keys are gone.

As of newer versions it is now possible to find this information in:

{ 'Member': { 'Addr': '1.2.3.4', 'Tags': { 'port': 8300 }}}

This key seems to function only intermittently in consul 0.6.4 so we can't outright replace the old business logic with the new one. Since the information will be in either one set of keys or another, and "or" is the operative word here, I've dropped in an or operator. agent_addr and agent_port will become whichever key does not return None (unless they both do).

While there is a more robust way of detecting cluster leadership from /v1/agent/self via { 'Stats': { 'consul': { 'leader': true }}} this key is not available in version 0.6.4, and refactoring the check to use both methods would likely introduce more complexity. While comparing IP:Port is a bit roundabout, it works.

Motivation

Wealthsimple has recently deployed consul, which coincided nicely with the version 1.0.0 release. We noticed right away that Datadog was hardly reporting any metrics at all. Digging into the check shows that the bulk of the metrics are retrieved only from the leader node, and the leader node is determined by the above-described IP:Port check, and led us down the road to writing this fix.

Testing Guidelines

Tests are expected to pass for all three flavours.

Some refactoring of consul tests were required to make this work. As a start, the server.json config file has now been divided up into three configs, one each for consul versions 0.6.4, 0.7.2, and 1.0.0. This is necessary because version 1.0.0 requires acl_agent_token to be set, whereas previous versions do not recognize this key and refuse to start. Configs should probably be immutable, so having one per version will support that notion going forward.

consul.rake has also been updated with a few new tricks:

1. All three consul containers load the server config now. This is necessary for consul 1.0.0 where the agent token must be present at the beginning.
2. Because all containers must load the config, it's a bit of a pain to do docker create, docker cp, and docker start three times; instead we use docker run for all three containers with the --volume flag to mount the config file as read-only at runtime.
3. Last but not least, wait_on_docker_logs has been updated to include agent: Synced node info which is how 1.0.0 agents assert readiness.

Versioning

• Bumped the version check in manifest.json
• Updated CHANGELOG.md. Please use Unreleased as the date in the title
  for the new section.

Additional Notes

We have a ticket open with Datadog's excellent support team where more context may be found (ZD ticket 116528).

[byronwolfman]

Hey! Just bumping this to find out what we need to do to move this forward. Thanks!

[masci]

@byronwolfman I'm on it, sorry this took so long!

[masci]

LGTM! Thanks for the thorough explanation of the PR, that made the review process super easy 💯