Geographical 2D plots fail without internet access

[JorgeBornemann]

Describe the bug

Geographical 2D plots try to load the coastlines from the internet. If the host does not have connection to the internet it will crash the whole bake_recipe task in the workflow. Note that this is important as some of the current HPC installations have internet access restrictions from some hosts (i.e. EarthSciences New Zealand (former niwa) has internet access blocked on compute nodes, preventing us from using a large pool of resources and harming scalability).

How to reproduce

Steps to reproduce the behaviour:

1. It is installation dependent, so to reproduce you will need to find a server that doesn't have access to the internet
2. Run cset-workflow with a recipe that runs geographical 2D plots
3. The bake_recipe task should fail with a https socket error visible in any of the bunch.*.err
4. Note that at NCMRWF, running the workflow once in a server with access to the internet, allowed to run it later (same user, same workflow, another run id) from a server without internet access, but still fails for other users using the same environment the first time they run if the server doesn't have internet access. This is a behaviour that I don't fully understand.

Expected behaviour

It should work from any host, probably trying first to run with high-quality coastline that require internet access, and if that fails use lower quality coastlines that are installed as part as the cset environment.

[jfrost-mo]

On first run cartopy downloads the coastline data from an S3 bucket and saves it locally. As coastlines don't really change, it will continue to use this cached data in perpetuity. If you have a shared file system between internet enabled nodes and offline nodes then that would allow you to download the shapefiles. The shapefiles are saved into each user's home directory ($HOME/.local/share/cartopy/shapefiles/ by default) which is why it can work for some users and not others.

There are a few ways we could go about dealing with this. We could handle the case of the download failing, leaving you without coastlines on those plots.

Alternatively (or perhaps in addition to) we could include instructions to pre-cache the shapefiles. This is what is done on the Met Office HPC, with the shapefiles saved to a central location and cartopy configured to search there for them. (IIRC it is via the CARTOPY_DATA_DIR environment variable.)

[jfrost-mo]

The same considerations will apply to borders in #1995.