Quantmode Yahoo getSymbols broken ? #202

shane0078 · 2017-11-30T03:37:47Z

Yahoo getSymbols is failing since this morning. was working until last night

> getSymbols("CSCO", from="2017-01-01")
Warning: CSCO download failed; trying again.
Error: CSCO download failed after two attempts. Error message:
HTTP error 400.

Using R 3.3.3 version so thought it would resolve HTTP / HTTPS issue.
Please advise.

joshuaulrich · 2017-11-30T03:50:50Z

Cannot reproduce using the latest version on CRAN.

Please advise.

As requested in the issue template, please provide the output from sessionInfo().

shane0078 · 2017-11-30T03:55:32Z

Attached is the output

> sessionInfo()
R version 3.3.3 (2017-03-06)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X El Capitan 10.11.1

locale:
[1] C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] quantmod_0.4-11 TTR_0.23-2      xts_0.10-0      zoo_1.8-0      

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.14      magrittr_1.5      forecast_8.2      munsell_0.4.3     stsm_1.9         
 [6] colorspace_1.3-2  lattice_0.20-34   rlang_0.1.4       quadprog_1.5-5    plyr_1.8.4       
[11] tools_3.3.3       tsoutliers_0.6-6  nnet_7.3-12       parallel_3.3.3    grid_3.3.3       
[16] timeDate_3042.101 gtable_0.2.0      KFKSDS_1.6        tseries_0.10-42   lazyeval_0.2.1   
[21] lmtest_0.9-35     tibble_1.3.4      ggplot2_2.2.1     curl_3.0          fracdiff_1.4-2   
[26] scales_0.5.0

shane0078 · 2017-11-30T05:11:04Z

Getting the same error when I upgraded to 3.4.2 version.

> getSymbols("CSCO",from="2017-01-25")
Warning: CSCO download failed; trying again.
Error: CSCO download failed after two attempts. Error message:
HTTP error 400.
> sessionInfo()
R version 3.4.2 (2017-09-28)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: OS X El Capitan 10.11.1

Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib

locale:
[1] C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] boot_1.3-20     quantmod_0.4-11 TTR_0.23-2      xts_0.10-0      zoo_1.8-0      

loaded via a namespace (and not attached):
[1] compiler_3.4.2  tools_3.4.2     curl_2.8.1      grid_3.4.2      lattice_0.20-35

goodcheer · 2017-11-30T08:06:13Z

Has the same issue when calling getSymbols.yahoo()

> google.xts <- getSymbols('GOOGL',auto.assign = FALSE,verbose = TRUE )
downloading  GOOGL .....

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   125  100   125    0     0    125      0  0:00:01  0:00:01 --:--:--   112
GOOGL download failed; trying again. Error: GOOGL download failed after two attempts. Error message:
HTTP error 400.
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   125  100   125    0     0    125      0  0:00:01 --:--:--  0:00:01     0

My session info

R version 3.4.2 (2017-09-28)
Platform: i386-w64-mingw32/i386 (32-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=Korean_Korea.949  LC_CTYPE=Korean_Korea.949    LC_MONETARY=Korean_Korea.949
[4] LC_NUMERIC=C                 LC_TIME=Korean_Korea.949    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] quantmod_0.4-11 TTR_0.23-2      xts_0.10-0      zoo_1.8-0      

loaded via a namespace (and not attached):
[1] compiler_3.4.2  tools_3.4.2     curl_3.0        yaml_2.1.14     grid_3.4.2      knitr_1.17     
[7] lattice_0.20-35`

goodcheer · 2017-11-30T08:40:36Z

It seems like yahoo intentionally blocked the request.
I got a response as below
{ "finance": { "error": { "code": "Unauthorized", "description": "Invalid cookie" } } }

and also when using getQuote(),
the server responds as

It has come to our attention that this service is being used in violation of the Yahoo Terms of Service. As such, the service is being discontinued. For all future markets and equities data research, please refer to finance.yahoo.com.

slopesluff · 2017-11-30T08:44:34Z

I also get the same error as of this morning, 2017-11-30. You can access historical data on their web without any problems. Yahoo Finance

options("getSymbols.yahoo.warning"=FALSE).
Warning: ^VIX download failed; trying again.
Error: ^VIX download failed after two attempts. Error message:
HTTP error 400

sessionInfo()
R version 3.4.2 (2017-09-28)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Arch Linux

Matrix products: default
BLAS: /usr/lib/libblas.so.3.8.0
LAPACK: /usr/lib/liblapack.so.3.8.0

locale:
 [1] LC_CTYPE=sv_SE.ISO-8859-1       LC_NUMERIC=C                   
 [3] LC_TIME=sv_SE.ISO-8859-1        LC_COLLATE=sv_SE.ISO-8859-1    
 [5] LC_MONETARY=sv_SE.ISO-8859-1    LC_MESSAGES=sv_SE.ISO-8859-1   
 [7] LC_PAPER=sv_SE.ISO-8859-1       LC_NAME=C                      
 [9] LC_ADDRESS=C                    LC_TELEPHONE=C                 
[11] LC_MEASUREMENT=sv_SE.ISO-8859-1 LC_IDENTIFICATION=C            

attached base packages:
[1] datasets  grDevices utils     graphics  stats     methods   base     

other attached packages:
 [1] fBasics_3042.89            timeSeries_3042.102       
 [3] timeDate_3042.101          AlphaVantageClient_0.0.1  
 [5] Quandl_2.8.0               PerformanceAnalytics_1.5.1
 [7] quantmod_0.4-11            TTR_0.23-3                
 [9] xts_0.10-1                 zoo_1.8-0                 
[11] devtools_1.13.4           

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.14    lattice_0.20-35 digest_0.6.12   withr_2.1.0    
 [5] grid_3.4.2      R6_2.2.2        jsonlite_1.5    httr_1.3.1     
 [9] spatial_7.3-11  curl_3.0        compiler_3.4.2  memoise_1.1.0

braverock · 2017-11-30T10:24:39Z

It works for me here. To @goodcheer 's point, I almost never use Yahoo data, so I wouldn't get flagged as an offender worthy of blocking.

joshuaulrich · 2017-11-30T12:12:39Z

Thanks for the report. I could replicate this morning. The problem is that the "to" date was being formatted incorrectly in the request URL because it was being converting to character in scientific notation.

R> as.character(unclass(as.POSIXct("2017-11-30","UTC")))
[1] "1.512e+09"

This should be fixed in the 202_getSymbols_scipen branch. Please test and provide feedback.

joshuaulrich · 2017-11-30T14:47:48Z

Note that you can work around the issue by setting to = Sys.Date() + 1 for today. It won't be an issue tomorrow, even if you don't update any code.

RajaPaul · 2017-12-01T04:48:40Z

When will the next release with this bug fix come out ???

this needs to be fixed, as the following fails :(Date is in IST)

symbol="^NSEI"
date=as.Date("2017-11-30")
getSymbols(Symbols = symbol,warnings = FALSE,auto.assign = FALSE,from=date,to=date)

it works fine for other dates , these special dates needs to be handled in the new release

rholeros · 2018-04-28T13:41:34Z

Hi! getSymbols is failing when pulling multiple quotes, but not when doing it individually. I tried using the to = Sys.Date() + 1 but did not change anything.

Not sure if this belongs to the same issue.

>getSymbols(etfs,` env = data_env, src = "yahoo")
pausing 1 second between requests for more than 5 symbols
pausing 1 second between requests for more than 5 symbols
pausing 1 second between requests for more than 5 symbols
pausing 1 second between requests for more than 5 symbols
Warning: AOA  download failed; trying again.
Error: AOA  download failed after two attempts. Error message:
HTTP error 400.

>getSymbols("AOA", env = data_env, src = "yahoo")
[1] "AOA"

> sessionInfo()
R version 3.4.2 (2017-09-28)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS High Sierra 10.13.4

Matrix products: default
BLAS: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRblas.0.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] Quandl_2.8.0               quantmod_0.4-13            TTR_0.23-2                
[4] tseries_0.10-44            PerformanceAnalytics_1.5.2 xts_0.10-0                
[7] zoo_1.8-0                 

loaded via a namespace (and not attached):
 [1] quadprog_1.5-5  lattice_0.20-35 digest_0.6.12   withr_2.1.2     mime_0.5       
 [6] grid_3.4.2      R6_2.2.2        jsonlite_1.5    git2r_0.21.0    httr_1.3.1     
[11] curl_3.0        devtools_1.13.5 tools_3.4.2     compiler_3.4.2  memoise_1.1.0  
>

Roadrunner-one · 2019-05-27T05:00:57Z

Hi! getSymbols is failing when pulling multiple quotes, but not when doing it individually. I tried using the to = Sys.Date() + 1 but did not change anything.

Not sure if this belongs to the same issue.

>getSymbols(etfs,` env = data_env, src = "yahoo")
pausing 1 second between requests for more than 5 symbols
pausing 1 second between requests for more than 5 symbols
pausing 1 second between requests for more than 5 symbols
pausing 1 second between requests for more than 5 symbols
Warning: AOA  download failed; trying again.
Error: AOA  download failed after two attempts. Error message:
HTTP error 400.

>getSymbols("AOA", env = data_env, src = "yahoo")
[1] "AOA"

> sessionInfo()
R version 3.4.2 (2017-09-28)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS High Sierra 10.13.4

Matrix products: default
BLAS: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRblas.0.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] Quandl_2.8.0               quantmod_0.4-13            TTR_0.23-2                
[4] tseries_0.10-44            PerformanceAnalytics_1.5.2 xts_0.10-0                
[7] zoo_1.8-0                 

loaded via a namespace (and not attached):
 [1] quadprog_1.5-5  lattice_0.20-35 digest_0.6.12   withr_2.1.2     mime_0.5       
 [6] grid_3.4.2      R6_2.2.2        jsonlite_1.5    git2r_0.21.0    httr_1.3.1     
[11] curl_3.0        devtools_1.13.5 tools_3.4.2     compiler_3.4.2  memoise_1.1.0  
>

I have the exact same issue when I trying to pull multiple data, it always fails at the same position, but it works 1 out of 3 times

sessionInfo()
R version 3.6.0 (2019-04-26)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Mojave 10.14.5

Matrix products: default
BLAS: >>/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versi>>ons/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] tis_1.37.1 quantmod_0.4-14 TTR_0.23-4 xts_0.11-2 zoo_1.8-5

loaded via a namespace (and not attached):
[1] Rcpp_1.0.1 lattice_0.20-38 digest_0.6.19 grid_3.6.0 evaluate_0.13
[6] curl_3.3 rmarkdown_1.13 tools_3.6.0 xfun_0.7 yaml_2.2.0
[11] compiler_3.6.0 htmltools_0.3.6 knitr_1.23

shane0078 mentioned this issue Nov 30, 2017

getSymbols.yahoo does not follow redirect to HTTPS #149

Closed

sunnyc7 mentioned this issue Nov 30, 2017

I am getting a HTTP error 400 on tq_get() from Yahoo business-science/tidyquant#81

Closed

joshuaulrich self-assigned this Nov 30, 2017

joshuaulrich added the bug label Nov 30, 2017

shane0078 mentioned this issue Nov 30, 2017

Fix timestamps formatted in scientific notation #203

Closed

joshuaulrich mentioned this issue Dec 1, 2017

getSymbols() doesn't work #204

Closed

joshuaulrich closed this in 75968ee Dec 2, 2017

joshuaulrich added this to the Release 0.4-12 milestone Nov 24, 2018

joshuaulrich / quantmod

Quantmode Yahoo getSymbols broken ? #202

Quantmode Yahoo getSymbols broken ? #202

shane0078 commented Nov 30, 2017 •

edited by joshuaulrich

joshuaulrich commented Nov 30, 2017

shane0078 commented Nov 30, 2017 •

edited by joshuaulrich

shane0078 commented Nov 30, 2017 •

edited by joshuaulrich

goodcheer commented Nov 30, 2017 •

edited

goodcheer commented Nov 30, 2017 •

edited

slopesluff commented Nov 30, 2017 •

edited

braverock commented Nov 30, 2017

joshuaulrich commented Nov 30, 2017

joshuaulrich commented Nov 30, 2017 •

edited

RajaPaul commented Dec 1, 2017

rholeros commented Apr 28, 2018

Roadrunner-one commented May 27, 2019 •

edited

joshuaulrich / quantmod

Sponsor joshuaulrich/quantmod

Join GitHub today

Quantmode Yahoo getSymbols broken ? #202

Quantmode Yahoo getSymbols broken ? #202

Comments

shane0078 commented Nov 30, 2017 • edited by joshuaulrich

joshuaulrich commented Nov 30, 2017

shane0078 commented Nov 30, 2017 • edited by joshuaulrich

shane0078 commented Nov 30, 2017 • edited by joshuaulrich

goodcheer commented Nov 30, 2017 • edited

goodcheer commented Nov 30, 2017 • edited

slopesluff commented Nov 30, 2017 • edited

braverock commented Nov 30, 2017

joshuaulrich commented Nov 30, 2017

joshuaulrich commented Nov 30, 2017 • edited

RajaPaul commented Dec 1, 2017

rholeros commented Apr 28, 2018

Roadrunner-one commented May 27, 2019 • edited

Essential cookies

Always active

Analytics cookies

shane0078 commented Nov 30, 2017 •

edited by joshuaulrich

shane0078 commented Nov 30, 2017 •

edited by joshuaulrich

shane0078 commented Nov 30, 2017 •

edited by joshuaulrich

goodcheer commented Nov 30, 2017 •

edited

goodcheer commented Nov 30, 2017 •

edited

slopesluff commented Nov 30, 2017 •

edited

joshuaulrich commented Nov 30, 2017 •

edited

Roadrunner-one commented May 27, 2019 •

edited