Fix Consul connection conflicts when using HTTP/1.1

[miklish]

When connecting to Consul using a non-HTTP/2 connection and 2 (or more) threads simultaneously need to get a url for the same service, they can potentially both end up in the ConsulClientImpl.getConnection method.

Refers to PR #716

Method call chain:

LightCluster.serviceToUrl(serviceName)
--> LightCluster.discovery
--> CommandFallbackRegistry.discover (calls super)
--> AbstractRegistry.discover
--> CommandFallbackRegistry.doDiscover
--> ConsulRegistry.discoverService
--> ConsulRegistry.lookupServiceUpdate
--> ConsulRegistry.lookupConsulService
--> ConsulClientImpl.lookupHealthService
--> ConsulClientImpl.getConnection(String cacheKey)

Currently, for each cacheKey, the getConnection method will return the same ConsulConnection regardless of the thread that requests it. This can result in both threads using the same ConsulConnection simultaneously, generating a bad state in the shared ConsulConnection. In practice, this has resulted in NPE's in the serviceToUrl method, and has also resulted in the service being unable to make subsequent requests to Consul.

The solution is to ensure that every thread has its own private ConsulConnection. The downside of this is that it can cause a large cache of ConsulConnection objects (and hence HTTP connection objects) being created. However, the connection cache size cannot grow without bound, as it will be limited by the size of the undertow worker thread pool.

Additionally, with reasonable connection timeouts in Consul and the user service, unused connections contained in cached ConsulConnection objects will eventually be closed, freeing up network resources. If a ConsulConnection object containing a closed connection is requested, a new active HTTP connection will created automatically.

The methods...

• checkPass
• checkFail
• registerService
• unregisterService
• lookupHealthService

...have this issue, as they all call getConnection, and don't use keys that are thread-specific.

NOTE: It is recommended to simply use HTTP/2 to connect to Consul, since a connection cache is not needed with HTTP/2.

[stevehu]

Fix Consul connection conflicts when using HTTP/1.1 1.6.x

[stevehu]

Fix Consul connection conflicts when using HTTP/1.1 master