SSL error connecting to CSW

[PeterParslow]

Describe the bug

SSL error connecting to CSW

I get a certificate error when trying to use the built-in MetaSearch plugin to access CSWs that run over HTTPS. There is no certificate problem on any of the CSWs - I can see that with a browser. I have no problem accessing a WFS, presumably because that does not use CURL.

Message:

Unknown Error: HTTPSConnectionPool(host='geo6catalogue.bgs.ac.uk', port=443): Max retries exceeded with url: /geonetwork/srv/eng/csw?service=CSW&version=2.0.2&request=GetCapabilities (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1045)')))

How to Reproduce

Try to access a CSW that uses HTTPS, for example https://geo6catalogue.bgs.ac.uk/geonetwork/srv/eng/csw?request=GetCapabilities&service=CSW&version=2.0.2

QGIS and OS versions

3.16.17 (& earlier)

Additional context

See closed issue: #27159.

Given that not all users of QGIS have permission to set their Windows environment variables, surely this should at least be a documented installation requirement, so that corporate IT teams set QGIS & all its dependencies up correctly?

Just stating that it's not a QGIS problem (as at the closed issue) prevents some users from using some functionality. QGIS has penetrated quite widely in the public sector, where it's quite rare for "ordinary GIS users" to have administrative rights on their corporate PCs.

[agiudiceandrea]

I don't have any issue connecting to https://geo6catalogue.bgs.ac.uk/geonetwork/srv/eng/csw using QGIS 3.16 on Windows (OSGeo4W old repository).

Do you experience the same issue using other CSW server through SSL, like the predefined services?
Danmark: National CSW (geodata-info) https://geodata-info.dk/srv/dan/csw
New Zealand: LINZ Data Service https://data.linz.govt.nz/services/csw/
UK Location Catalogue Publishing Service https://data.gov.uk/csw

Given that not all users of QGIS have permission to set their Windows environment variables,

What environment variables are you referring to? Anyway, you can set any environment variable in QGIS via Settings->Options->System->Environment https://docs.qgis.org/3.16/en/docs/user_manual/introduction/qgis_configuration.html?highlight=qsettings#system-settings

3.16.17 (& earlier)

Maybe 3.16.1 or 3.16.7?

[gioman]

I tested on Win10, with the ogseo4w v2 installer on 3.16, 3.20 and master. No SSL error, but searching any service returns

Connection error: 'xml.etree.ElementTree.Element' object has no attribute 'getiterator'

works fine on Linux.

[gioman]

---

3.16.17 (& earlier)

@PeterParslow are you on Windows 7?

[agiudiceandrea]

I tested on Win10, with the ogseo4w v2 installer on 3.16, 3.20 and master. No SSL error, but searching any service returns

Connection error: 'xml.etree.ElementTree.Element' object has no attribute 'getiterator'

@gioman I think this is another issue: getiterator for xml.etree.ElementTree.Element object was declared deprecated and then removed in Python 3.9.
But in the MetaSearch plugin code I don't see any occurrence of getiterator. Could you show the whole error message?

[gioman]

Could you show the whole error message?

@agiudiceandrea it is the whole error message, and there is nothing in QGIS logs.

[agiudiceandrea]

@gioman your issue could be in the owslib Python library used by MetaSearch and provided by OSGeo4W.
Maybe better to file another bug report?

[gioman]

Maybe better to file another bug report?

@agiudiceandrea done here #44225

[PeterParslow]

Andrea,

Do you experience the same issue using other CSW server through SSL,

Yes. I should have thought of trying the UK one, I used to own that service until our Cabinet Office team implemented it within CKAN.

What environment variables are you referring to?

CURL_CA_BUNDLE, as described in the issue I linked to.

you can set any environment variable in QGIS via Settings->Options->System->Environment

Thanks - I didn't know that. Sadly, it doesn't help: even with CURL_CA_BUNDLE set to "C:\Program Files\QGIS 3.16\bin\curl-ca-bundle.crt", and QGIS restarted, I still get the same error message from each of those catalogues.

3.16.7 - sorry

On Windows 10

[agiudiceandrea]

Do you have a curl-ca-bundle.crt file in C:\Program Files\QGIS 3.16\bin\?
Which hash has the file curl-ca-bundle.crt?
Have you installed QGIS using the new MSI standalone installer (OSGeo4W v2 repository) or using the old EXE standalone installer (OSGeo4W old repository)?

The fix of the issue #27159 was to install the missing curl-ca-bundle package (OSGeo4W v2 repository), not to set an environmental variable: #27159 (comment), #27159 (comment)

[PeterParslow]

Do you have a curl-ca-bundle.crt file in C:\Program Files\QGIS 3.16\bin?

Yes - that's why I set the variable to point there!

Which hash has the file curl-ca-bundle.crt?

SHA256: c979c6f35714a0fedb17d9e5ba37adecbbc91a8faf4186b4e23d6f9ca44fd6cb

Have you installed QGIS using the new MSI standalone installer (OSGeo4W v2 repository) or using the old EXE standalone installer (OSGeo4W old repository)?

Don't know; I installed it from our corporate "app portal". I have a call open with our in-house team to fix or document whatever needs to be fixed or documented in their install. Hence the interest in finding out what the problem is. I will ask them what QGIS installation package that system provides - I strongly expect they start with the msi, unpack it & repackage it for corporate deployment.

I do see that the QGIS fix was to include this file in various download packages of QGIS, but it was noted that the installation would also need to set the environment variable to the correct value. My comment there is that this is not something that should be left to the end user.

[agiudiceandrea]

SHA256: c979c6f35714a0fedb17d9e5ba37adecbbc91a8faf4186b4e23d6f9ca44fd6cb

The curl-ca-bundle.crt shipped by OSGeo4W old repository (in curl-ca-bundle-20191119-1 package) has SHA256: d455e45744ced843d3e954f1d03c47aa979240c13f4984c8ba8d6f9622d16139
The curl-ca-bundle.crt shipped by OSGeo4W new v2 repository (in curl-ca-bundle-7.75.0-1 package) has SHA256: e010c0c071a2c79a76aa3c289dc7e4ac4ed38492bfda06d766a80b707ebd2f29

My comment there is that this is not something that should be left to the end user.

Yes of course. The package is currently automatically installed by the official OSGeo4W installers and no custom environmental variable is normally needed .

Anyway, the linked issue #27159 is about GDAL, which relies on curl and the curl-ca-bundle.crt file to check the certificates.
Instead, AFAIK the MetaSearch plugin doesn't relies on curl and the curl-ca-bundle.crt, but on the owslib and the certifi Python packages.

It seems to me you are experiencing the issue #35476: to fix (workaround) that issue a new option was added in the MetaSearch Settings window #36932.

[PeterParslow]

Thanks - disabling the SSL check works for me. Which also explains why setting that environment variable didn't make any difference

It would be even more helpful to know why QGIS objects to this certificate when other clients don't, but that's a different question.

[agiudiceandrea]

I would have investigate this issue further, but I didn't find a system (tried QGIS 3.16 e 3.20 from OSGeo4W old and new repository) on which the issue occurs for me...

[PeterParslow]

On reflection - really regarding #35476 - I'm not sure that giving the user the option to disable SSL checking is actual a fix, more of a work around: 'the SSL handling code can't handle something, let's turn the security down'.

[gioman]

On reflection

@PeterParslow I guess that the first thing to understand is why you get the error and others don't.

[PeterParslow]

That would be good - the fact that #35476 exists suggests that others do - or did before QGIS implemented the option of turning it off.