Speed up ESGF search

[bouweandela]

Description

This pull request speeds up searching for files on ESGF. This requires the latest version of esgf-pyclient (0.3.1), because it uses a feature that was broken in the previous release. See ESGF/esgf-pyclient#75.

This pull request also takes into account that ESGF/esgf-pyclient#71 and conda-forge/esgf-pyclient-feedstock#12 have been solved now.

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🧪 The new functionality is relevant and scientifically sound
• 🛠 This pull request has a descriptive title and labels
• 🛠 Code is written according to the code quality guidelines
• 🧪 and 🛠 Documentation is available
• 🛠 Unit tests have been added
• 🛠 Changes are backward compatible
• 🛠 Any changed dependencies have been added or removed correctly
• 🛠 The list of authors is up to date
• 🛠 All checks below this pull request were successful

---

To help with the number pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[codecov]

Codecov Report

Merging #1512 (82ccb14) into main (06d5de0) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1512   +/-   ##
=======================================
  Coverage   90.74%   90.74%           
=======================================
  Files         197      197           
  Lines       10547    10547           
=======================================
  Hits         9571     9571           
  Misses        976      976           

Impacted Files	Coverage Δ
esmvalcore/esgf/_search.py	100.00% <ø> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 06d5de0...82ccb14. Read the comment docs.

[valeriupredoi]

cheers much @bouweandela - I will trun on one GA action here then turn it off if all goes well, so please don't merge just yet (I'll turn it on in a few minutes, I just merged a PR and that's hogging a fair bit of GA nodes now); @schlunma heads up you guys will have to change the feedstock recipe with the new pins Bouwe put it, I think 👍

[schlunma]

I will do a couple of tests of the download feature with this change 👍

[valeriupredoi]

I don't feel like opening anew issue and since we have @bouweandela 's attention here I'll just ask: we had loads of coonectivity issues to llnl last night: HTTPConnectionPool(host='esgf-node.llnl.gov', port=80): Read timed out. (read timeout=120) - probably everyone was streaming the State of the Union speech, would it be an option to bypass these if it's just a connection hiccup and mark tests xfail or skip? I think there was an issue about this now that I think about it...meh, Bouwe, what you reckon?

[schlunma]

I tested 3 recipes with this and everything worked fine! From these 3 tests I cannot really tell if there is a speedup, but let's use the next round of recipe tests for that.

Thanks @bouweandela!!

@remi-kazeroni Please merge when you are happy with this 👍

[remi-kazeroni]

Thanks a lot @bouweandela! It works fine for me as well. I have run several recipes and the ESGF search seemed rather fast. I can't tell the gain based on these runs. We'll have a better overview after the next round of recipe testing for the v2.5 release.