Feature request: Use environment variable CURL_CA_BUNDLE if set

[schlichtanders]

I have problems using r curl via Conda environments. Concretely, the r-base seems to be compiled without the proper standard cert place /etc/ssl/certs/ca-certificates.crt, and hence all url request fail by default with an error similar to SSL peer certificate or SSH remote key was not OK: [] Cert verify failed: BADCERT_NOT_TRUSTED.

Setting the environment CURL_CA_BUNDLE fixes base::url so that readLines(base::url("https://httpbin.org/get")) indeed works, but the change is not picked up by r curl. curl_downlod still fails with the same error.

[jeroen]

The curl package by default uses windows Schannel, with certificates from the Windows native cert store, so no custom bundle should be needed.

You can switch to the openssl backend by setting the CURL_SSL_BACKEND variable, see: https://cran.r-project.org/web/packages/curl/vignettes/windows.html. In that case it should automatially pick up on the CURL_CA_BUNDLE variable as well.

[schlichtanders]

I am running on debian bookworm, there is no windows and is still not picking up CURL_CA_BUNDLE

[schlichtanders]

I am using R via julia, hence my reproducible example uses Julia to install the respective Conda pkgs. If you are nevertheless interested, I can provide the minimal example.

[schlichtanders]

I was curious myself - here the replication

1. start a docker by running

docker run -it --rm --env CURL_CA_BUNDLE="/etc/ssl/certs/ca-certificates.crt" julia:1.9-bookworm julia

2. copy and paste the following code into the docker julia REPL

# paste into `docker run -it --rm --env CURL_CA_BUNDLE="/etc/ssl/certs/ca-certificates.crt" julia:1.9-bookworm julia` 
import Pkg
Pkg.add("CondaPkg")
using CondaPkg
ENV["R_HOME"] = CondaPkg.envdir() * "/lib/R"
CondaPkg.add("r-curl")


Pkg.add("RCall")
using RCall
reval("library(curl)")
reval("readLines(base::url('https://httpbin.org/get'))")
reval("curl_download('https://httpbin.org/get', 'tmp.html')")

3. the output should be like follows

...

julia> reval("readLines(base::url('https://httpbin.org/get'))")
RObject{StrSxp}
 [1] "{"                                                                    
 [2] "  \"args\": {}, "                                                     
 [3] "  \"headers\": {"                                                     
 [4] "    \"Accept\": \"*/*\", "                                            
 [5] "    \"Host\": \"httpbin.org\", "                                      
 [6] "    \"User-Agent\": \"libcurl/7.84.0\", "                             
 [7] "    \"X-Amzn-Trace-Id\": \"Root=1-65491364-5c77566a08285f666ea55b0f\""
 [8] "  }, "                                                                
 [9] "  \"origin\": \"109.125.100.179\", "                                  
[10] "  \"url\": \"https://httpbin.org/get\""                               
[11] "}"                                                                    


julia> reval("curl_download('https://httpbin.org/get', 'tmp.html')")
ERROR: REvalError: Error in curl_download("https://httpbin.org/get", "tmp.html") : 
  SSL peer certificate or SSH remote key was not OK: [] Cert verify failed: BADCERT_NOT_TRUSTED

Note that you can also activate R mode in the julia REPL by pressing a single dollar sign $. (then you can use standard R to investigate it further. Alternatively, CondaPkg.envdir() * "/bin/R" is the respective R executable which is run inside the julia session.

[schlichtanders]

Just tested: Starting the docker with docker run -it --rm --env CURL_CA_BUNDLE="/etc/ssl/certs/ca-certificates.crt" --env CURL_SSL_BACKEND="openssl" julia:1.9-bookworm julia does not help, i.e. CURL_SSL_BACKEND is also not picked up.

[jeroen]

Ah sorry I thought you were on Windows. Yeah on Linux we assume that your libcurl has been built with the proper curl-ca-bundle configured. This should be the case if you use libcurl from debian/ubuntu, but I guess you have your own version?

BTW you can really save yourself a lof of problems by installing r-base from cran like everyone else, instead of anaconda: https://cran.r-project.org/bin/linux/ubuntu/

[schlichtanders]

Anaconda is much better at dependency tracking than installing r libraries from source (or probably binaries as well - because of all the non-R dependencies).

Most crucially this is the case on Linux arm64 which don't have binaries on any CRAN. Anaconda is the only system which has binaries for arm64 architectures for R (as far as I know of).
I think it makes a lot of sense for a standard packages as curl to also support conda distribution. EDIT: or at least match the behaviour of base R url (which works in this case)

[jeroen]

So which libcurl are you using then? Why is it not properly configured with a default ca? Nobody else on Linux has had this problem.

[schlichtanders]

I am sorry, I am really only using anaconda. (micromamba concretely). It has packaged r-curl, and hence should also provide a respective libcurl. The ca default is anyway taken from r-base which has the same problem (if you don't set --env CURL_CA_BUNDLE above you will see that both r base and curl will fail). conda also packaged r-base of course.

[schlichtanders]

Is there a current workaround? (I know one could use handles, but this does not work when curl is used inside read_csv for instance)

[schlichtanders]

I found a workaround...

trace(curl::new_handle, exit=quote(handle_setopt(h, cainfo=Sys.getenv("CURL_CA_BUNDLE"))))

This indeed will change the default new_handle behaviour to pick up the right ca cert store

[schlichtanders]

I think the strongest argument for this feature is to match the behaviour of base R url (which picks up CURL_CA_BUNDLE dynamically)

[jeroen]

Can you test with the new version:

install.packages('curl', repos = 'https://jeroen.r-universe.dev')

[schlichtanders]

Thank you for your work. Unfortunately, I am not using install.packages at all - I would wait until it is available via conda-forge channel (EDIT: also the issue only appeared when using conda)