Retry with certifi cert if curl fails to find any

[sgillies]

Towards resolving #2009

This requires being able to clear the VSI cache, which will have been contaminated by previous failure. And thus requires GDAL 2.2.1, but I'm going to require 2.4 because we might want to scope the cache clear to a single file.

[sgillies]

@snowman2 since OS X and conda-forge builds don't need this, I've built a re-try with certifi into our wrapper around rasterio.open().

[sgillies]

TODO: a test

[snowman2]

Thanks for taking this on. I think this will be very useful for the wheels 👍

since OS X and conda-forge builds don't need this, I've built a re-try with certifi into our wrapper around rasterio.open()

Here are some things I thought might be worth considering/exploring (Disclaimer: I don't have the answers):

• Would setting the certificate path using certifi by default provide more stability or do you foresee it causing issues?
• What errors are raised with GDAL 3 & PROJ 7 in the wheels when the CA bundle path is not set and PROJ_NETWORK=ON?
• certifi has ~374k other packages that depend on it and is very lightweight. Including it as a dependency by default might be worthwhile.

This may or may not be helpful, but here it is for reference all the same:

• In pyproj 3, certifi is the backup if the CA Bundle environment variable is not set code.

[sgillies]

@snowman2 I was hoping to avoid bad surprises on OS X, which has its own SSL lib that the rasterio wheels depend on.

Thanks for pointing out the PROJ situation. Since the PROJ CDN client and caching is all done very low down in C, there's no Python exception to catch and retry, right?

[snowman2]

I was hoping to avoid bad surprises on OS X, which has its own SSL lib that the rasterio wheels depend on.

Have you tested this out to see if there are any issues with this?

Since the PROJ CDN client and caching is all done very low down in C, there's no Python exception to catch and retry, right?

I haven't tested out how GDAL handles this, but the scenario where the values are inf and just prints to a log is possible. I think building wheels with PROJ 7 and GDAL 3 would be the easiest way to test.

[snowman2]

When rasterio is python 3+, this also looks useful: https://docs.python.org/dev/library/ssl.html#ssl.SSLContext.load_verify_locations

[snowman2]

Since the PROJ CDN client and caching is all done very low down in C, there's no Python exception to catch and retry, right?

Side note just for awareness: recently dug in the GDAL code and found GDAL's PROJ Log Handler. From what I can tell, it seems likely that a GDAL exception would be raised in the case of an internal PROJ error.