SOLR-13953: Update eviction behavior of cache in Solr Prometheus exporter to allow for larger clusters

[ajablonski]

Description

Email thread starts here: http://mail-archives.apache.org/mod_mbox/lucene-dev/201911.mbox/%3cCAOz296DSV-tt7rWBirBZ+P4=vT5g29FZrR_2zHrHF084Xq+gyw@mail.gmail.com%3e . Jira issue here: https://issues.apache.org/jira/browse/SOLR-13953

For large (> 100 node) clusters, the exporter fails with "connection pool shut down" for certain nodes. This is caused by only the last 100 HttpSolrClients added to the cache still being open -- earlier additions are evicted and closed, even though the HttpSolrClients are still returned for use in metricsForAllHosts and pingAllCollections .

Solution

Change the cache configuration to evict clients from the cache after twice the scrape interval, instead of using a fixed size cache.

Would love opinions on:

• any other approaches
• whether we should use a different caching strategy
• whether we should use some other timeout besides twice the scrape interval.

Tests

Can add a regression test if it seems valuable, but with past configuration, would need to be with > 100 nodes.

Checklist

Please review the following and check all that apply:

• I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
• I have created a Jira issue and added the issue ID to my pull request title.
• I am authorized to contribute this code to the ASF and have removed any code I do not have a license to distribute.
• I have given Solr maintainers access to contribute to my PR branch. (optional but recommended)
• I have developed this patch against the master branch.
• I have run ant precommit and the appropriate test suite.
• I have added tests for my changes.
• I have added documentation for the Ref Guide (for Solr changes only).

[ErickErickson]

Forgot to close this when I fixed the JIRA.