The curl-ca-bundle.crt file is ignored on Windows, except if CURL_CA_BUNDLE is set

[joa-quim]

I did this

Put the curl-ca-bundle.crt file in the several places mentioned in here

I expected the following

that it woks, but it doesn't. Only if the CURL_CA_BUNDLE ENV variable is set
See here for more details

curl/libcurl version

7.56.0

operating system

Win10 64 bits

[bagder]

First, there is no 7.56.0 version. Please tell us the real version number.

Then, you haven't told us exactly what you did that doesn't work so there's nothing here for us to explain, fix or otherwise act on. Please include all necessary details in your report here instead of referring to other web pages - lots of us read and work on these issues via mail.

[joa-quim]

Sorry, quick fingers and from memory. It's 7.54.0, and I did tell what I did. I put the curl-ca-bundle.crt in current, dir system's path and application dir but if you want to blame the messenger ...

[bagder]

You seem to refer to the way the curl tool searches for curl-ca-bundle.crt. Are you? If so, can you please provide a more exact recipe for us to reproduce this issue? To be clear: libcurl does not do that searching.

[mkauf]

libcurl does not do that searching.

It does, but only on Windows. See the function FindWin32CACert() in tool_doswin.c.

According to https://curl.haxx.se/docs/sslcerts.html , curl searches in:

• application's directory
• current working directory
• Windows System directory (e.g. C:\windows\system32)
• Windows Directory (e.g. C:\windows)
• all directories along %PATH%

[bagder]

It does, but only on Windows. See the function FindWin32CACert() in tool_doswin.c.

The tool part of tool_doswin.c is a pretty good clue: it is for the curl tool. Not libcurl.

[joa-quim]

I can repeat the steps in the link I provided in first post but, yes, the program uses libcurl and fails because
SSL certificate problem: unable to get local issuer certificate, unless an internal flag to turn off certificate verification is turned on.
So, is this feature on purpose? (the docs don't warn about it)

[bagder]

(Now it seems we've moved away from the topic of this issue?)

That's normally an error message you get when the server doesn't ship its an intermediate certificate in the TLS handshake, and the reason you don't see the error in your favorite browser is because browsers A) cache intermediate certificate and B) some of them implement AIA which goes fetch the missing cert. libcurl does neither A nor B and not providing the intermediate cert is considered a flaw. One way to easily tell if this is the problem is to run the site through ssllabs ssl test.

[joa-quim]

(Now it seems we've moved away from the topic of this issue?)

No, it's exactly the same issue. I will re-post here what's in the link in my first post.

For quite some time I cannot use the 'vsis' because of certificates issue. For example, a GMT test that has a command like this no longer works on Windows


gdalinfo /vsicurl/http://larryfire.files.wordpress.com/2009/07/untooned_jessicarabbit.jpg


because

ERROR 11: HTTP response code: 301 - SSL certificate problem: unable to get local issuer certificate gdalinfo failed - unable to open '/vsicurl/http://larryfire.files.wordpress.com/2009/07/untooned_jessicarabbit.jpg'.


It used to work but probably with an older libcurl dll.

The above is with my own build gdal and dependencies (libcurl included) but the same happens with the gisinternals binaries.


I have re(and re)ad this page about the certificates

https://curl.haxx.se/docs/sslcerts.html

but regarding Windows and the curl-ca-bundle.crt file what is said about it simply does not work. The only thing that works is setting the ENV variable


set CURL_CA_BUNDLE=V:\bin\curl-ca-bundle.crt

Now, we had this in GMT recently and I used the nuke option

curl_easy_setopt (Curl, CURLOPT_SSL_VERIFYPEER, 0L); /* Tell libcurl to not verify the peer */


so tried to do the same thing in the GDAL code (the obvious point seamed to be VSICurlSetOptions in cpl_vsi_curl.cpp) but still does not work.


OSGeo4W works but probably because they are still using a 4 years old libcurl.dll


Am I the only one seeing this?

Thanks
````

[bagder]

So is there any remaining questions now?

[joa-quim]

Well, yes two actually:

1. Do you consider this a bug or not?
2. In case yes, do you intend to fix it?

[bagder]

Do you consider this a bug or not?

The ca cert search thing is not a bug as I already explained. The search for the CA cert file that the curl command line tool does is a feature of the curl command line tool and not of the libcurl library. You can of course easily add that search yourself in your application if you want to mimic what curl does. I suppose we could view it as a documentation bug if the documentation isn't clear enough in specifying the distinction between what the curl tool does for this and what the library does.

We could possibly discuss adding that feature to the library, but that's a slightly different take on the subject. I'm not convinced this is a good thing for the library to do, at least not unconditionally.

The unable to get local issuer certificate thing is a server bug, not something we can fix. We could work on dealing with it better, like more in the style of how browsers do, but that's a pretty big piece of work and neither easy nor quickly done.

[joa-quim]

Ok, good that we agree that it is a documentation fault, which was the base for me to submit this as a bug report.

Regarding future improvements I'm certainly not aware of the details an job difficulties but if the same sort of command work on linux I (and other that will cross with this behavior) was certainly expecting that it worked similarly on Windows, even if under certain conditions.

[bagder]

4. If you're using the curl command line tool, you can specify your own CA
   cert path by setting the environment variable CURL_CA_BUNDLE to the path
   of your choice.

   If you're using the curl command line tool on Windows, curl will search
   for a CA cert file named "curl-ca-bundle.crt" in these directories and in
   this order:

   1. application's directory
   2. current working directory
   3. Windows System directory (e.g. C:\windows\system32)
   4. Windows Directory (e.g. C:\windows)
   5. all directories along %PATH%

It says command line tool twice. What else can we do to emphasize that?

[joa-quim]

4. If you're using the curl command line tool, you can specify your own CA
   cert path by setting the environment variable CURL_CA_BUNDLE to the path
   of your choice.

Which in fact also works when using the libcurl library so not so unreasonable to expect the same applied to the CA certificate.

[bagder]

That's a specific behavior only in some builds (those that defined CURL_WANTS_CA_BUNDLE_ENV), which is why it is not document as a general libcurl behavior. I would probably argue that it should rather be removed completely, for consistency...

[joa-quim]

OK, let me summarize

1. On Linux this command (form the GDAL library) works
   gdalinfo /vsicurl/http://larryfire.files.wordpress.com/2009/07/untooned_jessicarabbit.jpg

2. On Windows the same command
   ERROR 11: HTTP response code: 301 - SSL certificate problem: unable to get local issuer certificate

3. On Windows, this it works if one add
   set CURL_CA_BUNDLE=C:\bin\curl-ca-bundle.crt
gdalinfo /vsicurl/http://larryfire.files.wordpress.com/2009/07/untooned_jessicarabbit.jpg

Conclusion, since 3. is a lack of consistency let's remove it and make the command fail too. Than it will work on Linux and fail on Windows. Consistency above all..

[bagder]

It doesn't "work on windows" in a generic sense. It works only for a small subset of Windows builds. The ones that happen to use one of those makefiles that had this enabled.

If you want your tool to accept the CURL_CA_BUNDLE environment variable to point out the path, then why don't you just getenv() it and send in that path to CURLOPT_CAINFO ? You could then have it work identically over all platforms you support. You'd even use the API as documented!

[joa-quim]

GDAL is not "my tool" it's a library used by really many programs, among them one in which I participate.

[bagder]

How does that make any difference? I would imagine that users of the GDAL library would appreciate a single and documented way on how to override/set which CA cert to use. If said method should be "use an environment variable", then the GDAL library should get that environment variable and pass it to libcurl as explained. If it would use another method, like perhaps an option or config file or similar, then it should get the path that way and pass it to libcurl...

[jay]

Nobody here is responsible for that library, it is up to them to correctly set up the SSL usage. You happened to have found out that CURL_CA_BUNDLE env works for some libcurls when it should work for just the curl tool. That is undocumented behavior and will soon be removed from libcurl, but it will still be possible to do getenv("CURL_CA_BUNDLE") yourself and pass that to CURLOPT_CAINFO as Daniel said.

I installed GDAL using gdal-201-1600-x64-core.msi and confirm the same problem. Using process monitor I can see no attempt is made to access the curl-ca-bundle.crt that is included in the same directory, or any for that matter (ie no unusual 'name not found' or 'no file's).

If they want to search directories including the app directory for curl-ca-bundle.crt they can use Windows API function SearchPath like curl tool function FindWin32CACert does and then set CURLOPT_CAINFO to the found location.

[kalosma]

What is still not clear to me is why the certificate verification works on unix systems (actually I tested on Ubuntu and Mac OS) while it is not working by default on Windows. Is curl using some system CA list on unix while it fails to do the same on Windows? I just want ot understand better to avoid bad assumptions in my code. Thanks

[bagder]

@kalosma this is an old BUG REPORT. Your added questions will not be answered here. If you experience a bug with the latest curl version, file a new bug. If you have a question about the curl tool, ask on the curl-users mailing list. Thanks!