Reject connection attempts while closing

[DaveCTurner]

Today if there is a constant stream of connection attempts then it's possible for the ClusterConnectionManager to wait forever in close() for connectingRefCounter to be fully released. With this commit we reject connection attempts while closing, avoiding this starvation situation.

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[elasticsearchmachine]

Hi @DaveCTurner, I've created a changelog YAML for you.

[DaveCTurner]

Spotted this while looking at a test failure which apparently failed due to getting stuck exactly here: https://gradle-enterprise.elastic.co/s/rdlucvijy2dby. I'm not sure this is actually the reason for this failure, it is kinda implausible that we open connections at a high enough rate to cause this starvation, but still I don't see any other obvious reasons and this is worth fixing.

[pxsalehi]

LGTM. But a small nit: the test setup seems not very straight forward. Individual pieces/blocks of it could benefit from some extra comments.

[henningandersen]

LGTM.

[DaveCTurner]

LGTM. But a small nit: the test setup seems not very straight forward. Individual pieces/blocks of it could benefit from some extra comments.

Fair comment :) I added some more detail in 2c3b8c6, hope that helps.

[elasticsearchmachine]

💔 Backport failed

Status	Branch	Result
✅	8.6
❌	7.17	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 92465

[DaveCTurner]

On reflection I think this might fix the test failure mentioned above after all. We don't have to constantly open connections until the suite times out, we only have to do it until we terminate the threadpool, because we're using GENERIC threads here which just silently drop work on the floor at shutdown and therefore could well leak a ref.