Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GDAL/VSICURL certificate errors loading cloud optimized geotiffs over https via data source manager #27159

Closed
qgib opened this issue Jul 3, 2018 · 24 comments
Closed

GDAL/VSICURL certificate errors loading cloud optimized geotiffs over https via data source manager #27159

qgib opened this issue Jul 3, 2018 · 24 comments
Labels
Bug Either a bug report, or a bug fix. Let's hope for the latter! Data Provider Related to specific vector, raster or mesh data providers

Comments

@qgib
Copy link
Contributor

qgib commented Jul 3, 2018 •
edited
Loading

Author Name: Nick S (@nickrsan)
Original Redmine Issue: 19331
Affected QGIS version: 3.2
Redmine category:data_provider

Hey there

While testing the cloud-optimized geotiff support, I ran into problems with files hosted via HTTPS.

It can successfully load and display over HTTP, but not HTTPS. When loading over HTTPS, it gives errors like "CURL error: SSL certificate problem: unable to get local issuer certificate" and "CURL error: SSL certificate problem: self signed certificate in certificate chain" (two different servers/URLs for those errors). I've tried hosting the files on Box, Github, a server at my office and a server I run. All files could be accessed over HTTPS in chrome and firefox as standard downloads. When possible to load over standard HTTP, those files succeed in QGIS, but fail over HTTPS with the above errors.

The full error message that's most common is:

CRITICAL Invalid Layer : GDAL provider Cannot open GDAL dataset /vsicurl/https://raw.githubusercontent.com/ucd-cws/nitrates-cv/master/1945/Nharvest_actual.tif:
CURL error: SSL certificate problem: unable to get local issuer certificate
Raster layer Provider is not valid (provider: gdal, URI: /vsicurl/https://raw.githubusercontent.com/ucd-cws/nitrates-cv/master/1945/Nharvest_actual.tif

This happens on 3.2.0 on Windows 10 1803. When using Gdal 2.2.3 on Bash on Ubuntu on Windows, I can successfully use gdal_translate on the files over https, so it seems like a QGIS issue or an issue with QGIS specific to my machine - I've not yet been able to get someone else to try on a different machine. One of the files I'm using is here. It works via gdal_translate/VSICURL in Bash on Ubuntu on Windows but not in QGIS: https://github.com/ucd-cws/nitrates-cv/blob/master/1945/Nharvest_actual.tif?raw=true

A URL that allows access via HTTP and HTTPS where the behavior can also be seen to work over HTTP and not HTTPS is http://managedretreat.org/test/NgwDirect.tif

Related issue(s): #28557 (duplicates)
Redmine related issue(s): 20737
@qgib
Copy link
Contributor Author

qgib commented Jul 4, 2018

Author Name: Nick S (@nickrsan)

From some further testing, looks like it's related to a long-ish standing CURL/GDAL bug on Windows: http://osgeo-org.1560.x6.nabble.com/gdal-dev-libcurl-and-the-certificates-and-Windows-td5322919.html has an overview, but there's discussion of the issue in relation to GDAL here (curl/curl#1538), including how the CURL_CA_BUNDLE environment variable will be deprecated and programs should find the bundle and pass it to curl themselves. It looks like there are some GDAL commits in response to find the bundle, maybe on the system PATH, so it's possible QGIS' copy of GDAL needs configuration (I'm not sure, getting out of my knowledge area here).

In the meantime, if I set the CURL_CA_BUNDLE environment variable to point to an existing curl-ca-bundle.crt (in my case, the one that ships with R 3.4.0), everything works without turning off certificate verification. This solves the problem on my machine, but not the problem with the distribution. I'm not sure what QGIS' role here is, if it'd be possible to ship its own curl-ca-bundle.crt file and configure GDAL, but I'll leave this open both so others can find the solution and in case there's a role for QGIS in making VSICURL work within the application on Windows even if the upstream is broken. Thanks!

@qgib
Copy link
Contributor Author

qgib commented Feb 24, 2019 •
edited
Loading

Author Name: Giovanni Manghi (@gioman)

@qgib qgib added Bug Either a bug report, or a bug fix. Let's hope for the latter! Data Provider Related to specific vector, raster or mesh data providers labels May 25, 2019
This was referenced May 25, 2019
Closed
Closed
@sweco-nosihg
Copy link

Had the same issue, not sure how I can create a workaround for this issue. Tried to set the proxy - but doesnt seem to effect the result and can not switch from https to http...

@PedroVenancio
Copy link
Contributor

Same issue here, when accessing data thru HTTPS protocol, using QGIS 3.8 compiled against GDAL 2.4.1 (OSGeo4W 64bits):

Invalid Layer: GDAL provider Cannot open GDAL dataset /vsicurl/https://cld.pt/dl/download/43299142-3a39-40f1-b3d1-3248611c7ec4/S2B_20190907_Mosaico_Portugal_Continental_RGB_12_8_4_3763.tif: CURL error: SSL certificate problem: unable to get local issuer certificate (file: ..\..\..\src\providers\gdal\qgsgdalprovider.cpp row: 2642function QgsGdalProvider::initIfNeeded:) Raster layer Provider is not valid (provider: gdal, URI: /vsicurl/https://cld.pt/dl/download/43299142-3a39-40f1-b3d1-3248611c7ec4/S2B_20190907_Mosaico_Portugal_Continental_RGB_12_8_4_3763.tif (file: ..\..\..\src\core\raster\qgsrasterlayer.cpp row: 627function QgsRasterLayer::setDataProvider:)

C:\>gdalinfo --version
GDAL 2.4.1, released 2019/03/15

C:\>gdalinfo /vsicurl/https://cld.pt/dl/download/43299142-3a39-40f1-b3d1-3248611c7ec4/S2B_20190907_Mosaico_Portugal_Continental_RGB_12_8_4_3763.tif
ERROR 11: CURL error: SSL certificate problem: unable to get local issuer certificate
gdalinfo failed - unable to open '/vsicurl/https://cld.pt/dl/download/43299142-3a39-40f1-b3d1-3248611c7ec4/S2B_20190907_Mosaico_Portugal_Continental_RGB_12_8_4_3763.tif'.

Using ' --config GDAL_HTTP_UNSAFESSL YES', solves the problem in the CLI, but not the access in QGIS.

C:\>gdalinfo --config GDAL_HTTP_UNSAFESSL YES /vsicurl/https://cld.pt/dl/download/43299142-3a39-40f1-b3d1-3248611c7ec4/S2B_20190907_Mosaico_Portugal_Continental_RGB_12_8_4_3763.tif
Driver: GTiff/GeoTIFF
Files: /vsicurl/https://cld.pt/dl/download/43299142-3a39-40f1-b3d1-3248611c7ec4/S2B_20190907_Mosaico_Portugal_Continental_RGB_12_8_4_3763.tif
Size is 31065, 60493
Coordinate System is:
PROJCS["ETRS89 / Portugal TM06",
    GEOGCS["ETRS89",
        DATUM["European_Terrestrial_Reference_System_1989",
            SPHEROID["GRS 1980",6378137,298.257222101,
                AUTHORITY["EPSG","7019"]],
            TOWGS84[0,0,0,0,0,0,0],
            AUTHORITY["EPSG","6258"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4258"]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",39.66825833333333],
    PARAMETER["central_meridian",-8.133108333333334],
    PARAMETER["scale_factor",1],
    PARAMETER["false_easting",0],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AXIS["X",EAST],
    AXIS["Y",NORTH],
    AUTHORITY["EPSG","3763"]]
Origin = (-127786.591583720612107,297279.380753421806730)
Pixel Size = (10.002778289544233,-10.002778289544233)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  COMPRESSION=JPEG
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  ( -127786.592,  297279.381) (  9d41' 1.00"W, 42d20' 4.79"N)
Lower Left  ( -127786.592, -307818.686) (  9d33'59.60"W, 36d53'11.46"N)
Upper Right (  182949.716,  297279.381) (  5d54'49.30"W, 42d19'25.19"N)
Lower Right (  182949.716, -307818.686) (  6d 4'52.23"W, 36d52'38.80"N)
Center      (   27581.562,   -5269.653) (  7d48'42.80"W, 39d37'13.27"N)
Band 1 Block=31065x16 Type=Byte, ColorInterp=Red
  Overviews: 15533x30247, 7767x15124, 3884x7562, 1942x3781, 971x1891, 486x946, 243x473, 122x237
  Mask Flags: PER_DATASET ALPHA
  Overviews of mask band: 15533x30247, 7767x15124, 3884x7562, 1942x3781, 971x1891, 486x946, 243x473, 122x237
Band 2 Block=31065x16 Type=Byte, ColorInterp=Green
  Overviews: 15533x30247, 7767x15124, 3884x7562, 1942x3781, 971x1891, 486x946, 243x473, 122x237
  Mask Flags: PER_DATASET ALPHA
  Overviews of mask band: 15533x30247, 7767x15124, 3884x7562, 1942x3781, 971x1891, 486x946, 243x473, 122x237
Band 3 Block=31065x16 Type=Byte, ColorInterp=Blue
  Overviews: 15533x30247, 7767x15124, 3884x7562, 1942x3781, 971x1891, 486x946, 243x473, 122x237
  Mask Flags: PER_DATASET ALPHA
  Overviews of mask band: 15533x30247, 7767x15124, 3884x7562, 1942x3781, 971x1891, 486x946, 243x473, 122x237
Band 4 Block=31065x16 Type=Byte, ColorInterp=Alpha
  Overviews: 15533x30247, 7767x15124, 3884x7562, 1942x3781, 971x1891, 486x946, 243x473, 122x237

This does not happen in QGIS on a Linux machine, compiled against GDAL 2.2.2, with the same dataset. The same in the CLI.

pedro@omen ~ $ gdalinfo --version
GDAL 2.2.2, released 2017/09/15

pedro@omen ~ $ gdalinfo /vsicurl/https://cld.pt/dl/download/43299142-3a39-40f1-b3d1-3248611c7ec4/S2B_20190907_Mosaico_Portugal_Continental_RGB_12_8_4_3763.tif
Driver: GTiff/GeoTIFF
Files: /vsicurl/https://cld.pt/dl/download/43299142-3a39-40f1-b3d1-3248611c7ec4/S2B_20190907_Mosaico_Portugal_Continental_RGB_12_8_4_3763.tif
Size is 31065, 60493
Coordinate System is:
PROJCS["ETRS89 / Portugal TM06",
    GEOGCS["ETRS89",
        DATUM["European_Terrestrial_Reference_System_1989",
            SPHEROID["GRS 1980",6378137,298.257222101,
                AUTHORITY["EPSG","7019"]],
            TOWGS84[0,0,0,0,0,0,0],
            AUTHORITY["EPSG","6258"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4258"]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",39.66825833333333],
    PARAMETER["central_meridian",-8.133108333333334],
    PARAMETER["scale_factor",1],
    PARAMETER["false_easting",0],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AXIS["X",EAST],
    AXIS["Y",NORTH],
    AUTHORITY["EPSG","3763"]]
Origin = (-127786.591583720612107,297279.380753421806730)
Pixel Size = (10.002778289544233,-10.002778289544233)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  COMPRESSION=JPEG
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  ( -127786.592,  297279.381) (  9d41' 1.00"W, 42d20' 4.79"N)
Lower Left  ( -127786.592, -307818.686) (  9d33'59.60"W, 36d53'11.46"N)
Upper Right (  182949.716,  297279.381) (  5d54'49.30"W, 42d19'25.19"N)
Lower Right (  182949.716, -307818.686) (  6d 4'52.23"W, 36d52'38.80"N)
Center      (   27581.562,   -5269.653) (  7d48'42.80"W, 39d37'13.27"N)
Band 1 Block=31065x16 Type=Byte, ColorInterp=Red
  Overviews: 15533x30247, 7767x15124, 3884x7562, 1942x3781, 971x1891, 486x946, 243x473, 122x237
  Mask Flags: PER_DATASET ALPHA
  Overviews of mask band: 15533x30247, 7767x15124, 3884x7562, 1942x3781, 971x1891, 486x946, 243x473, 122x237
Band 2 Block=31065x16 Type=Byte, ColorInterp=Green
  Overviews: 15533x30247, 7767x15124, 3884x7562, 1942x3781, 971x1891, 486x946, 243x473, 122x237
  Mask Flags: PER_DATASET ALPHA
  Overviews of mask band: 15533x30247, 7767x15124, 3884x7562, 1942x3781, 971x1891, 486x946, 243x473, 122x237
Band 3 Block=31065x16 Type=Byte, ColorInterp=Blue
  Overviews: 15533x30247, 7767x15124, 3884x7562, 1942x3781, 971x1891, 486x946, 243x473, 122x237
  Mask Flags: PER_DATASET ALPHA
  Overviews of mask band: 15533x30247, 7767x15124, 3884x7562, 1942x3781, 971x1891, 486x946, 243x473, 122x237
Band 4 Block=31065x16 Type=Byte, ColorInterp=Alpha
  Overviews: 15533x30247, 7767x15124, 3884x7562, 1942x3781, 971x1891, 486x946, 243x473, 122x237

@sweco-nosihg
Copy link

sweco-nosihg commented Sep 17, 2019 •
edited
Loading

I can access the https://cld.pt/dl/download/43299142-3a39-40f1-b3d1-3248611c7ec4/S2B_20190907_Mosaico_Portugal_Continental_RGB_12_8_4_3763.tif
in QGIS OK (ver 3.8.2-Zanzibar) and Windows 64-bit.

According to this however:
http://cog-validate.radiant.earth/html

It does not seem to be a valid COGEOTiff.

`Errors:

The file is greater than 512xH or Wx512,but is not tiled
The offset of the first block of overview of index 6 should be after the one of the overview of index 7
The offset of the first block of overview of index 5 should be after the one of the overview of index 6
The offset of the first block of overview of index 4 should be after the one of the overview of index 5
The offset of the first block of overview of index 3 should be after the one of the overview of index 4
The offset of the first block of overview of index 2 should be after the one of the overview of index 3
The offset of the first block of overview of index 1 should be after the one of the overview of index 2
The offset of the first block of overview of index 0 should be after the one of the overview of index 1
The offset of the first block of the main resolution image should be after the one of the overview of index 7`

@PedroVenancio
Copy link
Contributor

Hi @sweco-nosihg

This raster was just an example, that works ok in Linux and not in Windows.

From what I read, I believed that this issue only affects Windows. But you could access my raster in Windows. Do you use QGIS from standalone installer or from OSGeo4W?

Thank you very much!

@sweco-nosihg
Copy link

From Standalone installer
After adding the GDAL_HTTP_UNSAFESSL=YES in QGIS settings I also think you need to restart QGIS if I remember correctly (as this is loaded during QGIS-start?)

image

@PedroVenancio
Copy link
Contributor

Hi @sweco-nosihg

Thanks!

I've set the custom environment variable

unsafessl

and QGIS already loads layers from HTTPS servers without problems!

However, disabling the certificate check is not the best option.

Can this be solved in QGIS side?

@glw
Copy link

glw commented Nov 4, 2019

This seems to be fixed.

QGIS version 3.4.13-Madeira
Compiled against GDAL/OGR 2.2.2

@nickrsan
Copy link

nickrsan commented Nov 5, 2019 •
edited
Loading

I'm still seeing it in 3.8.1-Zanzibar on Windows 10 Enterprise 1903. Same VSICURL errors as before with a relatively new (and settings not modified) QGIS install. Installed via the dedicated installer.

@PedroVenancio
Copy link
Contributor

This seems to be fixed.
QGIS version 3.4.13-Madeira
Compiled against GDAL/OGR 2.2.2

@glw
This does not happen in GDAL 2.2.2, only in latest versions.
It is still true in gdal 3.1.0dev.

@nirvn
Copy link
Contributor

nirvn commented Nov 17, 2019

It'll be fixed in the next point release standalone installers. If you use the package based osgeo4w installer, it is fixed now.

@nirvn nirvn closed this as completed Nov 17, 2019
@PedroVenancio
Copy link
Contributor

Hi @nirvn

I've just updated QGIS 3.10 nightly (OSGeo4W64) to

QGIS version 3.10.0-A Coruña
QGIS code revision 8096b0b91f
Compiled against Qt 5.11.2
Running against Qt 5.11.2
Compiled against GDAL/OGR 3.1.0dev
Running against GDAL/OGR 3.1.0dev
Compiled against GEOS 3.8.0-CAPI-1.13.1
Running against GEOS 3.8.0-CAPI-1.13.1 
Compiled against SQLite 3.29.0
Running against SQLite 3.29.0
PostgreSQL Client Version 11.5
SpatiaLite Version 4.3.0
QWT Version 6.1.3
QScintilla2 Version 2.10.8
Compiled against PROJ 7.0.0
Running against PROJ Rel. 7.0.0, March 1st, 2020
OS Version Windows 10 (10.0)

and this issue still persists. In what code revision should this be fixed?

Thank you very much!

@jef-n
Copy link
Member

jef-n commented Nov 19, 2019

QGIS code revision 8096b0b

and this issue still persists. In what code revision should this be fixed?

It's not a problem in QGIS' code.

@PedroVenancio
Copy link
Contributor

Hi @jef-n

It's related with the recent curl update in OSGeo4W?

Should be fixed in some GDAL update?

@jef-n
Copy link
Member

jef-n commented Nov 19, 2019

curl-ca-bundle package added to OSGeo4W.

@PedroVenancio
Copy link
Contributor

Perfect! Thanks @jef-n !!

@nirvn
Copy link
Contributor

nirvn commented Nov 20, 2019

@jef-n , thanks for that follow up package work.

@nirvn
Copy link
Contributor

nirvn commented Nov 22, 2019

@PedroVenancio , can you confirm that the issue is fixed on your system?

@PedroVenancio
Copy link
Contributor

Hi @nirvn ,

Yes, I confirm that it's fixed!

@nirvn
Copy link
Contributor

nirvn commented Nov 22, 2019

@PedroVenancio , woupidou.

@PedroVenancio PedroVenancio mentioned this issue Dec 4, 2019
Closed
@duka97
Copy link

duka97 commented Jan 29, 2020

curl-ca-bundle package added to OSGeo4W.

can someone explain how in detail?

@PedroVenancio
Copy link
Contributor

curl-ca-bundle package added to OSGeo4W.

can someone explain how in detail?

imagem

@PeterParslow
Copy link

Given that not all users of QGIS have permission to set their Windows environment variables, surely this should at least be a documented installation requirement, so that corporate IT teams set QGIS & all its dependencies up correctly?

At present, with 3.16.17, I get a certificate error when trying to use the built-in MetaSearch plugin to access CSWs that run over HTTPS. There is no certificate problem on any of the CSWs - I can see that with a browser. I have no problem accessing a WFS, presumably because that does not use CURL.

Just stating that it's not a QGIS problem prevents some users from using some functionality. QGIS has penetrated quite widely in the public sector, where it's quite rare for "ordinary GIS users" to have administrative rights on their corporate PCs.

@PeterParslow PeterParslow mentioned this issue Jul 16, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug Either a bug report, or a bug fix. Let's hope for the latter! Data Provider Related to specific vector, raster or mesh data providers
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@jef-n @glw @nickrsan @PedroVenancio @nirvn @PeterParslow @duka97 @sweco-nosihg @qgib