New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The curl-ca-bundle.crt file is ignored on Windows, except if CURL_CA_BUNDLE is set #1538

Closed
joa-quim opened this Issue Jun 3, 2017 · 22 comments

Comments

Projects
None yet
5 participants
@joa-quim

joa-quim commented Jun 3, 2017

I did this

Put the curl-ca-bundle.crt file in the several places mentioned in here

I expected the following

that it woks, but it doesn't. Only if the CURL_CA_BUNDLE ENV variable is set
See here for more details

curl/libcurl version

7.56.0

operating system

Win10 64 bits

@bagder bagder added the SSL/TLS label Jun 3, 2017

@bagder

This comment has been minimized.

Member

bagder commented Jun 3, 2017

First, there is no 7.56.0 version. Please tell us the real version number.

Then, you haven't told us exactly what you did that doesn't work so there's nothing here for us to explain, fix or otherwise act on. Please include all necessary details in your report here instead of referring to other web pages - lots of us read and work on these issues via mail.

@joa-quim

This comment has been minimized.

joa-quim commented Jun 3, 2017

Sorry, quick fingers and from memory. It's 7.54.0, and I did tell what I did. I put the curl-ca-bundle.crt in current, dir system's path and application dir but if you want to blame the messenger ...

@bagder

This comment has been minimized.

Member

bagder commented Jun 3, 2017

You seem to refer to the way the curl tool searches for curl-ca-bundle.crt. Are you? If so, can you please provide a more exact recipe for us to reproduce this issue? To be clear: libcurl does not do that searching.

@mkauf

This comment has been minimized.

Contributor

mkauf commented Jun 4, 2017

libcurl does not do that searching.

It does, but only on Windows. See the function FindWin32CACert() in tool_doswin.c.

According to https://curl.haxx.se/docs/sslcerts.html , curl searches in:

  • application's directory
  • current working directory
  • Windows System directory (e.g. C:\windows\system32)
  • Windows Directory (e.g. C:\windows)
  • all directories along %PATH%
@bagder

This comment has been minimized.

Member

bagder commented Jun 4, 2017

It does, but only on Windows. See the function FindWin32CACert() in tool_doswin.c.

The tool part of tool_doswin.c is a pretty good clue: it is for the curl tool. Not libcurl.

@joa-quim

This comment has been minimized.

joa-quim commented Jun 4, 2017

I can repeat the steps in the link I provided in first post but, yes, the program uses libcurl and fails because
SSL certificate problem: unable to get local issuer certificate, unless an internal flag to turn off certificate verification is turned on.
So, is this feature on purpose? (the docs don't warn about it)

@bagder

This comment has been minimized.

Member

bagder commented Jun 4, 2017

(Now it seems we've moved away from the topic of this issue?)

That's normally an error message you get when the server doesn't ship its an intermediate certificate in the TLS handshake, and the reason you don't see the error in your favorite browser is because browsers A) cache intermediate certificate and B) some of them implement AIA which goes fetch the missing cert. libcurl does neither A nor B and not providing the intermediate cert is considered a flaw. One way to easily tell if this is the problem is to run the site through ssllabs ssl test.

@joa-quim

This comment has been minimized.

joa-quim commented Jun 4, 2017

(Now it seems we've moved away from the topic of this issue?)

No, it's exactly the same issue. I will re-post here what's in the link in my first post.

For quite some time I cannot use the 'vsis' because of certificates issue. For example, a GMT test that has a command like this no longer works on Windows


gdalinfo /vsicurl/http://larryfire.files.wordpress.com/2009/07/untooned_jessicarabbit.jpg


because

ERROR 11: HTTP response code: 301 - SSL certificate problem: unable to get local issuer certificate gdalinfo failed - unable to open '/vsicurl/http://larryfire.files.wordpress.com/2009/07/untooned_jessicarabbit.jpg'.


It used to work but probably with an older libcurl dll.

The above is with my own build gdal and dependencies (libcurl included) but the same happens with the gisinternals binaries.


I have re(and re)ad this page about the certificates

https://curl.haxx.se/docs/sslcerts.html

but regarding Windows and the curl-ca-bundle.crt file what is said about it simply does not work. The only thing that works is setting the ENV variable


set CURL_CA_BUNDLE=V:\bin\curl-ca-bundle.crt

Now, we had this in GMT recently and I used the nuke option

curl_easy_setopt (Curl, CURLOPT_SSL_VERIFYPEER, 0L); /* Tell libcurl to not verify the peer */


so tried to do the same thing in the GDAL code (the obvious point seamed to be VSICurlSetOptions in cpl_vsi_curl.cpp) but still does not work.


OSGeo4W works but probably because they are still using a 4 years old libcurl.dll


Am I the only one seeing this?

Thanks
````
@bagder

This comment has been minimized.

Member

bagder commented Jun 4, 2017

So is there any remaining questions now?

@joa-quim

This comment has been minimized.

joa-quim commented Jun 4, 2017

Well, yes two actually:

  1. Do you consider this a bug or not?
  2. In case yes, do you intend to fix it?
@bagder

This comment has been minimized.

Member

bagder commented Jun 4, 2017

Do you consider this a bug or not?

The ca cert search thing is not a bug as I already explained. The search for the CA cert file that the curl command line tool does is a feature of the curl command line tool and not of the libcurl library. You can of course easily add that search yourself in your application if you want to mimic what curl does. I suppose we could view it as a documentation bug if the documentation isn't clear enough in specifying the distinction between what the curl tool does for this and what the library does.

We could possibly discuss adding that feature to the library, but that's a slightly different take on the subject. I'm not convinced this is a good thing for the library to do, at least not unconditionally.

The unable to get local issuer certificate thing is a server bug, not something we can fix. We could work on dealing with it better, like more in the style of how browsers do, but that's a pretty big piece of work and neither easy nor quickly done.

@joa-quim

This comment has been minimized.

joa-quim commented Jun 4, 2017

Ok, good that we agree that it is a documentation fault, which was the base for me to submit this as a bug report.

Regarding future improvements I'm certainly not aware of the details an job difficulties but if the same sort of command work on linux I (and other that will cross with this behavior) was certainly expecting that it worked similarly on Windows, even if under certain conditions.

@bagder

This comment has been minimized.

Member

bagder commented Jun 4, 2017

  1. If you're using the curl command line tool, you can specify your own CA
    cert path by setting the environment variable CURL_CA_BUNDLE to the path
    of your choice.

    If you're using the curl command line tool on Windows, curl will search
    for a CA cert file named "curl-ca-bundle.crt" in these directories and in
    this order:

    1. application's directory
    2. current working directory
    3. Windows System directory (e.g. C:\windows\system32)
    4. Windows Directory (e.g. C:\windows)
    5. all directories along %PATH%

It says command line tool twice. What else can we do to emphasize that?

@joa-quim

This comment has been minimized.

joa-quim commented Jun 4, 2017

  1. If you're using the curl command line tool, you can specify your own CA
    cert path by setting the environment variable CURL_CA_BUNDLE to the path
    of your choice.

Which in fact also works when using the libcurl library so not so unreasonable to expect the same applied to the CA certificate.

@bagder

This comment has been minimized.

Member

bagder commented Jun 5, 2017

That's a specific behavior only in some builds (those that defined CURL_WANTS_CA_BUNDLE_ENV), which is why it is not document as a general libcurl behavior. I would probably argue that it should rather be removed completely, for consistency...

bagder added a commit that referenced this issue Jun 5, 2017

lib/curl_setup.h: remove CURL_WANTS_CA_BUNDLE_ENV
When this define was set, libcurl would check the environment variable
named CURL_CA_BUNDLE at run-time and use that CA cert bundle. This
feature was only defined by the watcom and m32 makefiles and caused
inconsistent behaviours among libcurls built on different platforms.

The curl tool does already feature its own similar logic and the library
does not really need it, and it isn't documented libcurl behavior. So
this change removes it.

Ref: #1538
@joa-quim

This comment has been minimized.

joa-quim commented Jun 5, 2017

OK, let me summarize

  1. On Linux this command (form the GDAL library) works
    gdalinfo /vsicurl/http://larryfire.files.wordpress.com/2009/07/untooned_jessicarabbit.jpg

  2. On Windows the same command
    ERROR 11: HTTP response code: 301 - SSL certificate problem: unable to get local issuer certificate

  3. On Windows, this it works if one add
    set CURL_CA_BUNDLE=C:\bin\curl-ca-bundle.crt
    gdalinfo /vsicurl/http://larryfire.files.wordpress.com/2009/07/untooned_jessicarabbit.jpg

Conclusion, since 3. is a lack of consistency let's remove it and make the command fail too. Than it will work on Linux and fail on Windows. Consistency above all..

@bagder

This comment has been minimized.

Member

bagder commented Jun 5, 2017

It doesn't "work on windows" in a generic sense. It works only for a small subset of Windows builds. The ones that happen to use one of those makefiles that had this enabled.

If you want your tool to accept the CURL_CA_BUNDLE environment variable to point out the path, then why don't you just getenv() it and send in that path to CURLOPT_CAINFO ? You could then have it work identically over all platforms you support. You'd even use the API as documented!

@joa-quim

This comment has been minimized.

joa-quim commented Jun 5, 2017

GDAL is not "my tool" it's a library used by really many programs, among them one in which I participate.

@bagder

This comment has been minimized.

Member

bagder commented Jun 5, 2017

How does that make any difference? I would imagine that users of the GDAL library would appreciate a single and documented way on how to override/set which CA cert to use. If said method should be "use an environment variable", then the GDAL library should get that environment variable and pass it to libcurl as explained. If it would use another method, like perhaps an option or config file or similar, then it should get the path that way and pass it to libcurl...

@jay

This comment has been minimized.

Member

jay commented Jun 5, 2017

Nobody here is responsible for that library, it is up to them to correctly set up the SSL usage. You happened to have found out that CURL_CA_BUNDLE env works for some libcurls when it should work for just the curl tool. That is undocumented behavior and will soon be removed from libcurl, but it will still be possible to do getenv("CURL_CA_BUNDLE") yourself and pass that to CURLOPT_CAINFO as Daniel said.

I installed GDAL using gdal-201-1600-x64-core.msi and confirm the same problem. Using process monitor I can see no attempt is made to access the curl-ca-bundle.crt that is included in the same directory, or any for that matter (ie no unusual 'name not found' or 'no file's).

If they want to search directories including the app directory for curl-ca-bundle.crt they can use Windows API function SearchPath like curl tool function FindWin32CACert does and then set CURLOPT_CAINFO to the found location.

@jay jay closed this Jun 5, 2017

kwrobot pushed a commit to aashish24/gdal-svn that referenced this issue Jun 5, 2017

rouault added a commit to OSGeo/gdal that referenced this issue Jun 5, 2017

bagder added a commit that referenced this issue Jun 15, 2017

lib/curl_setup.h: remove CURL_WANTS_CA_BUNDLE_ENV
When this define was set, libcurl would check the environment variable
named CURL_CA_BUNDLE at run-time and use that CA cert bundle. This
feature was only defined by the watcom and m32 makefiles and caused
inconsistent behaviours among libcurls built on different platforms.

The curl tool does already feature its own similar logic and the library
does not really need it, and it isn't documented libcurl behavior. So
this change removes it.

Ref: #1538
@kalosma

This comment has been minimized.

kalosma commented Apr 3, 2018

What is still not clear to me is why the certificate verification works on unix systems (actually I tested on Ubuntu and Mac OS) while it is not working by default on Windows. Is curl using some system CA list on unix while it fails to do the same on Windows? I just want ot understand better to avoid bad assumptions in my code. Thanks

@bagder

This comment has been minimized.

Member

bagder commented Apr 3, 2018

@kalosma this is an old BUG REPORT. Your added questions will not be answered here. If you experience a bug with the latest curl version, file a new bug. If you have a question about the curl tool, ask on the curl-users mailing list. Thanks!

@curl curl locked and limited conversation to collaborators Apr 3, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.