Global curl settings not used in handle

[CajH]

Description

Settings for curl can be set using for example:

set_config(config(ssl_verifypeer = 0L))

The ssl_verifypeer = 0L setting is common when overriding certificate validity checks.

Expected behavior

The curl setting should be applied to the handle.

Minimal, reproducible example

In an environment where a proxy has an intercept certificate which cannot be authenticated:

set_config(config(ssl_verifypeer = 0L))
getSymbols("MSFT")

Might generate an error:

Error in curl::curl_download("https://finance.yahoo.com", tmp, handle = h) : 
  Peer certificate cannot be authenticated with given CA certificates

Session Info

R version 3.4.0 (2017-04-21)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 7 x64 (build 7601) Service Pack 1

Matrix products: default

locale:
[1] LC_COLLATE=Swedish_Sweden.1252  LC_CTYPE=Swedish_Sweden.1252    LC_MONETARY=Swedish_Sweden.1252
[4] LC_NUMERIC=C                    LC_TIME=Swedish_Sweden.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] quantmod_0.4-8 RCurl_1.95-4.8 bitops_1.0-6   httr_1.2.1     TTR_0.23-1     xts_0.9-7      zoo_1.8-0     

loaded via a namespace (and not attached):
 [1] lattice_0.20-35 digest_0.6.12   withr_1.0.2     grid_3.4.0      R6_2.2.2        git2r_0.18.0    curl_2.7       
 [8] devtools_1.13.2 tools_3.4.0     compiler_3.4.0  memoise_1.1.0  

[joshuaulrich]

Thanks for the report! I'll look into how to ensure global curl settings are honored.

[CajH]

I got the GetSymbols.R script to work by temporarily adding

handle_setopt(h, ssl_verifypeer = 0)

when the handle is created.

[PsaksuMeRap]

I write :

library(quantmod)  
getSymbols("AAPL")

I get :

Error in curl::curl_download(cu, tmp, handle = h) :
Couldn't connect to server

sessionInfo()
R version 3.4.1 (2017-06-30)
attached base packages:

 [1] stats     graphics  grDevices utils     datasets  methods   base 
other attached packages:

 [1] quantmod_0.4-10 TTR_0.23-1      xts_0.9-7       zoo_1.8-0  
loaded via a namespace (and not attached):

[1] compiler_3.4.1  tools_3.4.1     curl_2.7        grid_3.4.1      lattice_0.20-35

Any ideas?

[CajH]

@PsaksuMeRap: I think you are experiencing a different issue. Maybe you can create a separate issue?

[joshuaulrich]

@PsaksuMeRap Are you saying that you get an error when calling getSymbols, but curl requests work outside of getSymbols? Does this command work?

w <- curl::curl_fetch_memory("https://finance.yahoo.com",curl::new_handle(verbose=TRUE))

[PsaksuMeRap]

Hi Joshua, I tried it and still get no connection The output is the following: I get : Rebuilt URL to: finance.yahoo.com * timeout on name lookup is not supported * Trying 87.248.118.23... * TCP_NODELAY set * After 4976ms connect time, move on! * connect to 87.248.118.23 port 443 failed: Timed out * Trying 87.248.118.22... * TCP_NODELAY set * After 2359ms connect time, move on! * connect to 87.248.118.22 port 443 failed: Timed out * Failed to connect to finance.yahoo.com port 443: Timed out * Closing connection 0 Error in curl::curl_fetch_memory("finance.yahoo.com";, curl::new_handle(verbose = TRUE)) : Couldn't connect to server Do you have any further suggestions? Thanks again for the help!

…

Sent from my iPhone

On 6 Jul 2017, at 15:06, Joshua Ulrich ***@***.***> wrote: @PsaksuMeRap Are you saying that you get an error when calling getSymbols, but curl requests work outside of getSymbols? Does this command work? w <- curl::curl_fetch_memory("https://finance.yahoo.com",curl::new_handle(verbose=TRUE)) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[joshuaulrich]

@PsaksuMeRap that seems to be an issue with your local setup being able to connect to the internet. I don't think it's related to this GitHub issue. Please email me if you would like to continue the discussion.

@CajH A possible solution to this problem is to pass curl options via .... We just have to be careful to only send curl options to the curl::new_handle() function, else it will error...

R> curl::new_handle(foo=1)
Error in handle_setopt(h, ...) : Unknown option: foo

Or, maybe we put them in a special argument (e.g. curl.options). That would be easier all around.

[joshuaulrich]

@CajH config() and set_config() are not functions in the curl package. What package exports them?

[CajH]

I think the set_config() function is from the httr package. I am not sure how options are supposed to be propagated with curl. RCurl seems to use global settings.

[arthur1616]

Same problem here, but the error code is:

Error in curl::curl_download(cu, tmp, handle = h) :
SSL certificate problem: unable to get local issuer certificate

@CajH I followed your step to put handle_setopt(h, ssl_verifypeer = 0) under h <- get0("_handle_", .quantmodEnv). Am I doing right? Now when I call getSymbols it returns Error: inherits(handle, "curl_handle") is not TRUE Do you mind to share the revised getSymbols.R with me? Thank you very much! My email is arthur1616@uchicago.edu

[arthur1616]

Solution from curl:

jeroen/curl#122

[joshuaulrich]

@CajH Can you please test the patch I just pushed to the 177_curl_options branch?

[CajH]

@joshuaulrich

I get this:

Error in curl::curl_download(cu, tmp, handle = h) :
Peer certificate cannot be authenticated with given CA certificates

When running this:

gs=getSymbols("GOOG")

After doing this:

devtools::install_github("joshuaulrich/quantmod", ref="177_curl_options")
library(quantmod)

[joshuaulrich]

@CajH Sorry for not being more explicit. You need to pass the ssl_verifypeer = 0 via the new curl.options argument:

goog <- getSymbols("GOOG", curl.options = list(ssl_verifypeer = 0), auto.assign = FALSE)

[CajH]

@joshuaulrich Thanks! We got this to work!

[joshuaulrich]

@CajH Awesome! Thanks again for the report and testing!

[jberninger]

Adding the curl.options = list(ssl_verifypeer = 0) parameter fixes issues i was having with the tq_get() command for tidyquant