getSymbols.yahoo() always throws "Error in new.session()"

[abernal30]

Description

When applying the function getSymbols(), it errors and does not return a result.

Minimal, reproducible example

getSymbols("AAPL")
## Error in new.session() : Could not establish session after 5 attempts.

[pverspeelt]

@joshuaulrich,

The issue is located in the function .getHandle and specifically in the local function new.session. All the 5 calls to the yahoo website return with no cookies, hence the function fails.

Looking over the code. I don't see much point in the .gethandle function as the yahoo query is also being built in the .yahooURL function. The only thing that .gethandle gives to .yahooURL is the crumb request to the query.srv. and cb has the value: "" for either query1 or query2. So this part is not needed to get the data.

Removing the .gethandle and removing the crumb part from building the yahoo url in .yahooURL returns the data without any issues.

[mpadge]

seeing same issue here

[alihru]

one also needs to get rid of the if statement inside the .getHandle function which stops the process if the number of cookies is zero

if (NROW(curl::handle_cookies(h)) ==0
stop(„Could not establish session after 5 attempts.“)

p.s. thank you for the quick fix! Really appreciate it

[joshuaulrich]

@pverspeelt and @alihru thanks for taking a look! I have a patch and am running CRAN and reverse-dependency checks now.

[msperlin]

Thanks for helping out @joshuaulrich . Im the author of BatchGetSymbols, which uses quantmod::getSymbols(), and got lots of emails this morning about the issue!

[ThomasTranGithub]

Seeing the same issue here.

[datawookie]

@joshuaulrich just confirming that I have tested the fix on the 358-getsymbols-new.session branch and it works for me. 👍 Thank you for responding so quickly! 🥇

[asaeb1480]

what's the solution? These comments are not working!

[datawookie]

@asaeb1480 please try installing from the 358-getsymbols-new.session branch. It's working for me again.

[asaeb1480]

What's the exact code for installation of 358-getsymbols-new.session plz?

[datawookie]

@asaeb1480 please look at https://cran.r-project.org/web/packages/githubinstall/vignettes/githubinstall.html.

[joshuaulrich]

remotes::install_github("joshuaulrich/quantmod@358-getsymbols-new.session")

[m1270]

Thank you @joshuaulrich

I can't install it due to this error:

Error: package ‘xts’ 0.12.1 was found, but > 0.12.1 is required by ‘quantmod’

The latest version of xts package is 0.12.1. I am running ubuntu 20.04 and R version of 4.1.2

[joshuaulrich]

@m1270 ah, that's because I have a change in development to move the S4 xts/zoo class to xts... I need to revert that. Thanks for the catch!

[fededemo]

Thanks joshua. getSymbols working now.
@m1270 working for me with an remotes::install_github("joshuaulrich/xts", force = TRUE)

[mik3hall]

That installs but doesn't appear to fix for me.

[TommyTorstP]

I have the same issue as @mik3hall here! I tried to find it in the given directory on the computer, but the file still seems to be deleted once install is complete

[asaeb1480]

Thanks, Joshuaulrich. It's working for me.

[mik3hall]

@TommyTorstP it is in the user library path that it indicated in the install but I must not default search that. Looking at how I can include that now. I assume it will eventually cran install to a normal system path.

[pawatkins]

Hello,

I'm very sorry but I'm an idiot when it comes to this stuff. I'm currently enrolled in a master's class but I've only been coding since January 2022.

I have a project due tonight at midnight and it uses quantmod.

How can I fix this error? I'm in Rstudio

Thanks in advance. I know this has to be work.

[joshuaulrich]

@pawatkins remotes::install_github("joshuaulrich/quantmod@358-getsymbols-new.session")

[Baroni77]

after a bit of experimenting, it is working for me too. Thanks @joshuaulrich!

[m1270]

Thank you @joshuaulrich
I confirm it is working for me as well.

[MiraiLi00]

Hello, @joshuaulrich, thank you for helping with this issue, but I was trying to run the code you provided, but it is still not working for me and it gives me this error:

ERROR: package installation failed
Error: Failed to install 'quantmod' from GitHub:
System command 'Rcmd.exe' failed, exit status: 1, stdout & stderr were printed

(sorry I am new to this)

Do you know what could be a problem?

[mik3hall]

@TommyTorstP Still not working for me.

Installed to...
installing to /Users/mjh/Library/R/x86_64/4.1/library/00LOCK-xts/00new/xts/libs

portfolio <- c("SPY","ORCL","FCX","X","MRO","ZIM","USO")
from.dat <- as.Date("01/01/22", format="%m/%d/%y")
getSymbols(portfolio,from=from.dat,auto.assign=T)
‘getSymbols’ currently uses auto.assign=TRUE by default, but will
use auto.assign=FALSE in 0.5-0. You will still be able to use
‘loadSymbols’ to automatically load data. getOption("getSymbols.env")
and getOption("getSymbols.auto.assign") will still be checked for
alternate defaults.

This message is shown once per session and may be disabled by setting
options("getSymbols.warning4.0"=FALSE). See ?getSymbols for details.

Error in new.session() : Could not establish session after 5 attempts.

.libPaths()
[1] "/Users/mjh/Library/R/x86_64/4.1/library" "/Library/Frameworks/R.framework/Versions/4.1/Resources/library"

[joshuaulrich]

Hi @zlfang00. That's very odd and it's not clear why remotes::install_github() thinks the packages it just installed are not available. I could be missing something, or it may be a bug in the function. Either way, I'm afraid I can't help.

[joshuaulrich]

'package ‘quantmod’ is in use and will not be installed

@yw6010, that means you have an R session that already has quantmod loaded, so you can't change anything in the directory the package is installed in. You need to close all your R sessions on that computer, install quantmod, and then you can load and use quantmod.

[ethanbsmith]

rstudio can load and keep links to packages active for intellisense parsing and such, even after a Session -> Restart R. best to shut down everything and do installs from a clean vanilla r session.

[yw6010]

'package ‘quantmod’ is in use and will not be installed

@yw6010, that means you have an R session that already has quantmod loaded, so you can't change anything in the directory the package is installed in. You need to close all your R sessions on that computer, install quantmod, and then you can load and use quantmod.

it works now!!! very appreciate. time to resume my project haha

[rhamo]

Thank you for the quick fix @joshuaulrich
Once installing "remotes::install_github("joshuaulrich/quantmod@358-getsymbols-new.session")"
this worked for me once when requesting a small group of tickers, however all new attempts since I now constantly receive "Not Cached - Error in download.." errors.

Example:

EMN | yahoo (161|506) | Found cache file - Got 100% of valid prices | Feels good!
EMR | yahoo (162|506) | Found cache file - Got 100% of valid prices | Got it!
ENPH | yahoo (163|506) | Found cache file - Got 100% of valid prices | Boa!
EOG | yahoo (164|506) | Not Cached - Error in download..
EPAM | yahoo (165|506) | Not Cached - Error in download..
EQIX | yahoo (166|506) | Not Cached - Error in download..
EQR | yahoo (167|506) | Not Cached - Error in download..
ES | yahoo (168|506) | Not Cached - Error in download..

I have tried restarting R but now I cannot even receive one successful return. Each ticker states "Error in download" once again. Any ideas?

Thank you

[edwinhung]

Hi, Thank you very much for the fix and comments here. getSymols.yahoo() now works for me, but I have a different problem. When I run getSymbols.yahoo() successfully in a loop for more than about 300-400 tickers, I started to get "HTTP error 401" for all following downloads. There are some failed downloads of invalid tickers in between though. Does anyone know what the issue is? Maybe I have the same problem with @rhamo.

Here is an example of the subsequent download:
> getSymbols.yahoo("AAPL",from="2022-04-27",to="2022-04-29",auto.assign=FALSE)
Warning: AAPL download failed; trying again. Warning: Unable to import “AAPL”. AAPL download failed after two attempts. Error message: HTTP error 401. [1] "Error in open.connection(file, \"rt\") : HTTP error 401.\n" attr(,"class") [1] "try-error" attr(,"condition") <simpleError in open.connection(file, "rt"): HTTP error 401.>

Thank you!

[zpio]

> install.packages("quantmod")
> library(quantmod)
> remotes::install_github("joshuaulrich/xts", force = TRUE)

* installing *source* package 'xts' ...
ERROR: cannot remove earlier installation, is it in use?
* removing 'C:/Users/Tama/Documents/R/win-library/3.6/xts'
* restoring previous 'C:/Users/Tama/Documents/R/win-library/3.6/xts'
Error in file.copy(lp, dirname(pkgdir), recursive = TRUE, copy.date = TRUE) : 
  (converted from warning) problem copying C:\Users\Tama\Documents\R\win-library\3.6\00LOCK-xts\xts\libs\x64\xts.dll to C:\Users\Tama\Documents\R\win-library\3.6\xts\libs\x64\xts.dll: Permission denied
* removing 'C:/Users/Tama/Documents/R/win-library/3.6/xts'
* restoring previous 'C:/Users/Tama/Documents/R/win-library/3.6/xts'
Error in file.copy(lp, dirname(pkgdir), recursive = TRUE, copy.date = TRUE) : 
  (converted from warning) problem copying C:\Users\Tama\Documents\R\win-library\3.6\00LOCK-xts\xts\libs\x64\xts.dll to C:\Users\Tama\Documents\R\win-library\3.6\xts\libs\x64\xts.dll: Permission denied
Execution halted
Error: Failed to install 'xts' from GitHub:
  (converted from warning) installation of package ‘C:/Users/Tama/AppData/Local/Temp/RtmpAnQpkC/file449050346c2/xts_0.12.1.2.tar.gz’ had non-zero exit status

> remotes::install_github("joshuaulrich/quantmod@358-getsymbols-new.session")

 installing *source* package 'quantmod' ...
   ** using staged installation
   ** R
   ** demo
   ** byte-compile and prepare package for lazy loading
   Error: package 'xts' required by 'quantmod' could not be found
   Execution halted
   ERROR: lazy loading failed for package 'quantmod'
-  removing 'C:/Users/Tama/AppData/Local/Temp/RtmpqubKOV/Rinst31ac26a1569f/quantmod'
         -----------------------------------
   ERROR: package installation failed
Error: Failed to install 'quantmod' from GitHub:
  System command 'Rcmd.exe' failed, exit status: 1, stdout + stderr (last 10 lines):
E> ** using staged installation
E> ** R
E> ** demo
E> ** byte-compile and prepare package for lazy loading
E> Error: package 'xts' required by 'quantmod' could not be found
E> Execution halted
E> ERROR: lazy loading failed for package 'quantmod'
E> * removing 'C:/Users/Tama/AppData/Local/Temp/RtmpqubKOV/Rinst31ac26a1569f/quantmod'
E>       -----------------------------------
E> ERROR: package installation failed

I don't know what else to do. Help

[BathosEconTrade]

get this as well when apply patch:
"Warning: package ‘quantmod’ is in use and will not be installed"
this after I removed quantmod and Quandl and then new session then new install.
get this:
Error in new.session() : Could not establish session after 5 attempts.Error in new.session() : Could not establish session after 5 attempts.
Error in new.session() : Could not establish session after 5 attempts.
I am so grateful for quantmod and really dependent upon it

[rpfreitasxyz]

Hi, Thank you very much for the fix and comments here. getSymols.yahoo() now works for me, but I have a different problem. When I run getSymbols.yahoo() successfully in a loop for more than about 300-400 tickers, I started to get "HTTP error 401" for all following downloads. There are some failed downloads of invalid tickers in between though. Does anyone know what the issue is? Maybe I have the same problem with @rhamo.

Here is an example of the subsequent download: > getSymbols.yahoo("AAPL",from="2022-04-27",to="2022-04-29",auto.assign=FALSE) Warning: AAPL download failed; trying again. Warning: Unable to import “AAPL”. AAPL download failed after two attempts. Error message: HTTP error 401. [1] "Error in open.connection(file, \"rt\") : HTTP error 401.\n" attr(,"class") [1] "try-error" attr(,"condition") <simpleError in open.connection(file, "rt"): HTTP error 401.>

Thank you!

I've been having the same issue!

[zlfang00]

Hi, @joshuaulrich, One thing I noticed is zoo is already installed in the public R library, while xts and TTR are not. My guess is this version quantmod only searches for its dependencies in the public library path?
Could you please confirm of that? Thanks very much!

Hi @zlfang00. That's very odd and it's not clear why remotes::install_github() thinks the packages it just installed are not available. I could be missing something, or it may be a bug in the function. Either way, I'm afraid I can't help.

[BathosEconTrade]

ok remote is working
then use getSymbols.yahoo("AAPL)
and still get:
Error in new.session() : Could not establish session after 5 attempts.
rpfreitasxyz you are doing better than I at least working but this this other problem.
any insights would be great.
all my models are based on getSymbols and yahoo, and am sure everyone notices is a very interesting time now.
many thanks

[joshuaulrich]

Patched quantmod on CRAN now

The source package for quantmod 0.4.20 has been released to CRAN! It may take another day or two for CRAN to build the Windows and macOS binaries, so users on those systems can install from source or wait for the binaries to be built.

---

Sponsors

I'd like to take this opportunity to thank those who started sponsoring me on GitHub today!

Check out my sponsors profile if you didn't know about GitHub sponsors and you love my work on quantmod, xts, TTR, IBrokers, microbenchmark, etc. Tiers start at $1, and higher tiers give you access to an exclusive Slack community of my sponsors, private messages with me, and more.

---

Yahoo Finance alternative

I also want to mention an alternative to Yahoo Finance data. I've found the end of day data from tiingo to be higher quality and the endpoint is more stable than Yahoo Finance. You need to sign up to get a free API key, and that gives individuals free access with 500 requests/hour and 20,000 requests/day, up to 5GB of data/month.

Disclaimer: While I don't receive any referral compensation for pointing individuals to the Free Plan or Power Plan, I may receive compensation (I honestly don't remember how much) from them for new Commercial Plan customers who learned of tiingo through me. But you have to mention me when you join, I don't have an affiliate link or anything like that.

[joshuaulrich]

@rhamo, @edwinhung, and @rpfreitasxyz A HTTP 401 response means the request is "unauthorized". That can happen for various reasons, but my guess is that you're hitting rate limits imposed by the Yahoo Finance servers. getSymbols() adds pauses when you request a lot of symbols at once to try and avoid hitting these rate limits. @msperlin (author of BatchGetSymbols) may have a better idea.

Let's move this conversation to #360.

---

@zpio you shouldn't need to install xts from GitHub. You only need the version on CRAN. Can you try to install quantmod via remotes::install_github("joshuaulrich/quantmod") and see if that works?

---

@zlfang00 The new version of quantmod uses zoo, xts, and TTR just like prior versions. Nothing has changed in that regard, so I'm stumped about what's causing your issue.

---

Error in new.session() : Could not establish session after 5 attempts.

@BathosEconTrade That looks like version 0.4.18 is being loaded instead of 0.4.19 or 0.4.20. Check packageVersion("quantmod") to see which version you're loading. Double check your library paths in case you are using more than one library...

[BathosEconTrade]

Many thanks all! Great help. Must have version 4.2.

[rhamo]

Thank you for the quick fix @joshuaulrich Once installing "remotes::install_github("joshuaulrich/quantmod@358-getsymbols-new.session")" this worked for me once when requesting a small group of tickers, however all new attempts since I now constantly receive "Not Cached - Error in download.." errors.

Example:

EMN | yahoo (161|506) | Found cache file - Got 100% of valid prices | Feels good! EMR | yahoo (162|506) | Found cache file - Got 100% of valid prices | Got it! ENPH | yahoo (163|506) | Found cache file - Got 100% of valid prices | Boa! EOG | yahoo (164|506) | Not Cached - Error in download.. EPAM | yahoo (165|506) | Not Cached - Error in download.. EQIX | yahoo (166|506) | Not Cached - Error in download.. EQR | yahoo (167|506) | Not Cached - Error in download.. ES | yahoo (168|506) | Not Cached - Error in download..

I have tried restarting R but now I cannot even receive one successful return. Each ticker states "Error in download" once again. Any ideas?

Thank you

Have tried with new quantmod 0.4.20 update but still no success. Every ticker states "Not Cached - Error in download" error as above.

Any ideas on how to solve this issue?
Thanks again

[samiranvekar]

Thank you@joshuaulrich. The issue is resolved

[msperlin]

It seems that yahoo finance is now restricting access to the data. Once you download too much data, you get the "HTTP 401" error. This is controled by Yahoo and there is nothing myself or @joshuaulrich can do about it.

As with any webscrapping script, you can try to overcome the restriction with:

• changing your ip one you reach the limit (use VPN)
• use a caching system to save data locally and download data over the course of days. Usually the restricted ips are released in 24 hours. A caching system is already available in yfR/BatchGetSymbols).

Overall, if the limits are too restrictive, it might be best to find other source of price data.

[NilssonHedge]

Is there any indication of what the Yahoo Rate limit is?

I can also confirm that the VPN fix is working, thanks for putting some serious effort into this. :)

[msperlin]

@NilssonHedge Im looking into it right now.. It doenst seem too restrictive. After a 401 error, I can download again in a few minutes.

[zlfang00]

are you sure it is the yahoo restriction? I tried Python package yfinance to grab the same size of date, no issues.

[joshuaulrich]

are you sure it is the yahoo restriction? I tried Python package yfinance to grab the same size of date, no issues.

The yfinance python library uses a different endpoint that returns JSON instead of a CSV. I'll look into adding something to getSymbols() that tries the other endpoint if one of them fails.

[rhamo]

Changing ip has worked wonders. Once again BatchGetSymbols now retrieves data without errors.

Thanks so much for all your help @msperlin and @joshuaulrich

[mif2000]

Hi, Really appreciate your dedication. I'm a new user to R (v 3.5.3) through Bert for Excel. I finally had everything working to call R functionality and getSymbols (with a single ticker) through an Excel VBA macro and then it abruptly stopped as noted in prior comments. It appears that the problem with getSymbols has been resolved and that I can now download quantmod v 4.20 zip file for Windows (10), which I did from the quantmod.zip link on the CRAN page dtd 2022-04-29, copying the unzipped quantmod directory to the BERT2/lib folder. The command "library (quantmod)" ouputs an error: "package or namespace load failed for 'quantmod' in rbind(info, getNamespaceInfo(env,"S3methods")). packageVersion("quantmod") outputs '0.4.20'. I must be doing something wrong! What is the proper way to install the new version of quantmod?

Thanks for your help,
mif2000