Intermittent timeout issues with new getSymbols.yahoo as well as getSplits and getDividends

[the-tourist-]

Description

I am intermittently getting what appear to be timeout errors with getSymbols, getSplits and getDividends using yahoo as the source. Most of the time these work but moderately often I get

Error in new.session(curl::new_handle()) :
Could not establish session after 5 attempts.

Expected behavior

The results of the calls

Minimal, reproducible example

getSymbols("MSFT", src = "yahoo")
getSplits("MSFT")
getDividends("MSFT")

Session Info

R version 3.4.0 (2017-04-21)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows Server >= 2012 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252 LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C LC_TIME=English_United States.1252

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] quantmod_0.4-9 TTR_0.23-1 xts_0.9-7 zoo_1.8-0

loaded via a namespace (and not attached):
[1] compiler_3.4.0 tools_3.4.0 curl_2.6 grid_3.4.0 lattice_0.20-35

[bennysiu]

I also have this problem, i need to insert Sys.sleep(60) between each getXXXXX.

[joshuaulrich]

I've noticed similar intermittent behavior at the beginning of a clean R session, but calling getSymbols a second time usually works. And subsequent calls to getSymbols in the same session also work.

I've run @the-tourist-'s example a few times in a new R session, but it works for me...

Can someone please provide a reproducible example that produces the behavior somewhat frequently? I know this is an intermittent issue, but I need code I can run to reliably reproduce the issue in order to investigate.

Please provide as much detail as possible. Does it happen at the start of a new/clean R session? Or after the R session has been running for awhile? Does it stop working after prior calls to getSymbols worked? Et cetera.

[bennysiu]

Hi Joshua,

I run below script in command prompt, it will no problem for first few run, but after it, it will show
"Could not establish session after 5 attempts."

####test.r
.libPaths("C:/Users/goI/Documents/R/win-library/3.4")
library(quantmod)
library(TTR)
args <- commandArgs(TRUE)
stockNumber <- args[1]
fromDate <- args[2]
toDate <- args[3]
stock = getSymbols(stockNumber, auto.assign=FALSE ,from=fromDate , to=toDate,src="yahoo")
print(Cl(stock))

###Command Prompt
c:\>.\Rscript.exe" test.r MSFT 2010-01-01 2017-06-01

Loading required package: xts
Loading required package: zoo

Attaching package: 'zoo'

The following objects are masked from 'package:base':

    as.Date, as.Date.numeric

Loading required package: TTR
Loading required package: methods
Version 0.4-0 included new data defaults. See ?getSymbols.
'getSymbols' currently uses auto.assign=TRUE by default, but will
use auto.assign=FALSE in 0.5-0. You will still be able to use
'loadSymbols' to automatically load data. getOption("getSymbols.env")
and getOption("getSymbols.auto.assign") will still be checked for
alternate defaults.

This message is shown once per session and may be disabled by setting
options("getSymbols.warning4.0"=FALSE). See ?getSymbols for details.

WARNING: There have been significant changes to Yahoo Finance data.
Please see the Warning section of '?getSymbols.yahoo' for details.

This message is shown once per session and may be disabled by setting
options("getSymbols.yahoo.warning"=FALSE).
Error in new.session(curl::new_handle()) :
  Could not establish session after 5 attempts.
Calls: getSymbols ... do.call -> getSymbols.yahoo -> .getHandle -> new.session
Execution halted

[bennysiu]

i run below code using rscript.exe in command prompt with 5 times, it will have "Could not establish session after 5 attempts." for 3 times.

####Scripts
test.r
h <- (curl::new_handle())
tmp <- tempfile()

for (i in 1:5) {
  curl::curl_download("https://finance.yahoo.com", tmp, handle = h)
	   
  if (NROW(curl::handle_cookies(h)) > 0)
	curl::handle_cookies(h)
    break;
  }

  if (NROW(curl::handle_cookies(h)) == 0)
      stop("Could not establish session after 5 attempts.")
####Scripts	  

####Result
Rscript.exe test.r
C:\Users\goI>"c:\Program Files\R\R-3.4.0\bin\x64\Rscript.exe" test.r

C:\Users\goI>"c:\Program Files\R\R-3.4.0\bin\x64\Rscript.exe" test.r

C:\Users\goI>"c:\Program Files\R\R-3.4.0\bin\x64\Rscript.exe" test.r
Error: Could not establish session after 5 attempts.
Execution halted

C:\Users\goI>"c:\Program Files\R\R-3.4.0\bin\x64\Rscript.exe" test.r
Error: Could not establish session after 5 attempts.
Execution halted

C:\Users\goI>"c:\Program Files\R\R-3.4.0\bin\x64\Rscript.exe" test.r
Error: Could not establish session after 5 attempts.
Execution halted
####Result

[joshuaulrich]

@bennysiu Thanks for the examples! I see some failed attempts (i.e. the value of i printed to the console) if I run the following command:

while ./curl_cookies.R; do :; done

Where curl_cookies.R contains:

#!/usr/bin/Rscript
.libPaths("~/R/library")
tmp <- tempfile()
on.exit(unlink(tmp))

h <- curl::new_handle()

for (i in 1:5) {
  curl::curl_download("https://finance.yahoo.com", tmp, handle = h)
  n <- NROW(curl::handle_cookies(h))
  if (n > 0)
    break
  cat(i, "\n")
}

if (n == 0)
  quit("no", status = 1)

That said, I've let that command run for ~5 minutes and didn't encounter an instance where it failed 5 times and therefore exited with status = 1. Maybe it is related to your physical location? Also, what version of the curl package are you using?

---

Here's a version that actually calls getSymbols, in an attempt to replicate the error. As with the curl example, I wasn't able to trigger the "Could not establish session after 5 attempts." error.

#!/usr/bin/Rscript
.libPaths("~/R/library")
suppressMessages(require(quantmod))
options("getSymbols.yahoo.warning"=FALSE)
options("getSymbols.warning4.0"=FALSE)

x <- try(getSymbols("SPY;EEM;AGG;IWM"))
if (inherits(x, "try-error")) {
  quit("no", status = 1)
} else {
  quit("no", status = 0)
}

[the-tourist-]

Joshua,

The scripts I am using are running on a server in France with 10mbs up and down stream bandwidths. Doing a ping of finance.yahoo.com takes about 30ms when I tried it just now.

I am working around the issue by using tryCatch and repeated calls with a small delay until it works, but maybe this could be moved into the quantmod code instead?

[joshuaulrich]

@the-tourist- getSymbols.yahoo already re-attempts the connection... but I just added some code that waits 100ms between attempts. It would be great if you could test it. The changes are on the 166_getSymbols_timeout branch.

[the-tourist-]

Hi, I just retried the alternate version, but still received the same message.

[joshuaulrich]

Thanks for the feedback!

[joshuaulrich]

@the-tourist-, @bennysiu I just pushed a commit that seems to fix this issue. Please test and let me know if it works for you too.

[the-tourist-]

Is it committed to the master or to the 166_getSymbols_timeout branch?

[joshuaulrich]

It's only on the 166_getSymbols_timeout branch right now.

[the-tourist-]

@joshuaulrich it would appear to be working. I tried calling a simple getSymbols script from the command line and after calling it 10 times I have not encountered the curl session error.

Thanks.

[joshuaulrich]

Awesome, thanks for the feedback!

[bennysiu]

it works great for me too. Thanks joshuaulrich.

[fededemo]

Hi!
today I'm having a similar issue with quantmod::getSymbols('EEM', from = (Sys.Date()-10)).
I'm getting: Error in new.session() : Could not establish session after 5 attempts

more related problems like this in the last 2 days:

• https://stackoverflow.com/questions/72057342/error-in-new-session-could-not-establish-session-after-5-attempts
• https://stackoverflow.com/questions/72051854/quantmodgetsymbols-cannot-retrieve-data-from-yahoo-finance

Thanks for your help!

[joshuaulrich]

@fededemo see #358.