HTTP Error 401 when called on large amount of tickers

[rpfreitasxyz]

Hi, Thank you very much for the fix and comments here. getSymbols.yahoo() now works for me, but I have a different problem. When I run getSymbols.yahoo() successfully in a loop for more than about 300-400 tickers, I started to get "HTTP error 401" for all following downloads. There are some failed downloads of invalid tickers in between though. Does anyone know what the issue is? Maybe I have the same problem with @rhamo.

Here is an example of the subsequent download:

> getSymbols.yahoo("AAPL",from="2022-04-27",to="2022-04-29",auto.assign=FALSE)
Warning: AAPL download failed; trying again.
Warning: Unable to import "AAPL".
AAPL download failed after two attempts. Error message:
HTTP error 401.
[1] "Error in open.connection(file, \"rt\") : HTTP error 401.\n"
attr(,"class")
[1] "try-error"
attr(,"condition")
<simpleError in open.connection(file, "rt"): HTTP error 401.>

Thank you!

Originally posted by @edwinhung in #358 (comment)

Thank you all for the quick fix in regards to tq_get()! However, I believe that this new issue has arisen.

[Knxd3]

Is a fix planned for tq_get? getSymbols works with many tickers because of the 1 second pause. Thanks.

[msperlin]

The limits do not seem to be too restrictive. After reaching the 404 error, I was able to get successfull api calls in a few minutes.

After that, I downloaded all sp500 stocks (2010-today) in a single call:
df_sp500 <- yfR::yf_collection_get("SP500", first_date = '2010-01-01')

If anyone can confirm this in your own R session, please do.

It looks as the restrictions are based on time between api calls for the same ip. This could invalidate any parallel computation, which is what I'm testing now.

[msperlin]

As expected, any parallel use of quantmod::getSymbol() reaches the limit very easilly. As such, I'm removing the parallel option from yfR and BatchGetSymbols.

When using a single session (non-parallel), yfR runs fine for any large sample of stocks.

[msperlin]

If you can, please confirm if the code below runs fine:

remotes::install_github("msperlin/yfR")

df_sp500 <- yfR::yf_collection_get("SP500", first_date = '2010-01-01')

[rpfreitasxyz]

If you can, please confirm if the code below runs fine:

remotes::install_github("msperlin/yfR")

df_sp500 <- yfR::yf_collection_get("SP500", first_date = '2010-01-01')

The code runs fine, with a caveat:

The issue really seems to be the amount of API calls. Even though they're not in parallel, they are still (500+) sequential calls to the Yahoo! Finance API, and as such, it is quite inconsistent whether or not the whole dataframe will be downloaded. I believe this has already been worked around with your implementation of a cache system.

Thus, if it hasn't been downloaded completely at once, I suggest users wait for a few minutes, then run the code again, until completion.

Either way, thank you, @msperlin!

[msperlin]

Yes, lack of consistency in equivalent calls to BatchGetSymbols/yf_get can be troublesome. I'll see how I can control for this, at least letting the user know about the 404 error.

[joshuaulrich]

I just added an option to use the JSON endpoint instead of the CSV endpoint. Can you try that and see if you still get the 401 responses? You can install the patch via: remotes::install_github("joshuaulrich/quantmod@362-yahoo-json-endpoint"). Then call quantmod::getSymbols("SPY", use.json.api = TRUE)

[msperlin]

Sure, let me try..

[msperlin]

I just added an option to use the JSON endpoint instead of the CSV endpoint. Can you try that and see if you still get the 401 responses? You can install the patch via: remotes::install_github("joshuaulrich/quantmod@362-yahoo-json-endpoint"). Then call quantmod::getSymbols("SPY", use.json.api = TRUE)

Changed the call to getSymbols and I tried my best to reach the 401 response, with not success. I ranned yfR with parallel execution (14 cores) and it worked as expected.

[msperlin]

looks good..