Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

getSymbols: catch errors and continue processing #135

Closed
joshuaulrich opened this issue Jan 21, 2017 · 6 comments
Closed

getSymbols: catch errors and continue processing #135

joshuaulrich opened this issue Jan 21, 2017 · 6 comments
Assignees

Comments

@joshuaulrich
Copy link
Owner

Wrap download.file() in a try() or tryCatch() in order to catch errors and provide more useful messages to the user.

For example, Google Finance does not provide historical data for the S&P 500 index, even though it's displayed on their website. Note there's no "Export/Download to spreadsheet" link under the historical chart like there are for symbols where getSymbols.google() works (e.g. General Electric).

R> getSymbols(".INX", src = "google")
Error in download.file(paste(google.URL, "q=", Symbols.name, "&startdate=",  : 
  cannot open URL 'http://finance.google.com/finance/historical?q=.INX&startdate=Jan+01,+2007&enddate=Jan+21,+2017&output=csv'
In addition: Warning message:
In download.file(paste(google.URL, "q=", Symbols.name, "&startdate=",  :
  cannot open URL 'http://finance.google.com/finance/historical?q=.INX&startdate=Jan+01,+2007&enddate=Jan+21,+2017&output=csv': HTTP status was '404 Not Found'

It would also be nice if a download.file() error did not completely stop getSymbols() when getSymbols() is called with multiple symbols.

@joshuaulrich
Copy link
Owner Author

joshuaulrich commented Mar 7, 2017

Although my initial thought was to put the try/catch in each getSymbols "method", it might be better to put into getSymbols proper, where errors from any future or custom "methods" would be caught. That might be difficult however, because getSymbols currently groups symbols by source and calls getSymbols.[source] one time for each group. So none of the symbols for a given source would have data if any one of the tickers in the getSymbols.[source] call fails.

It might be simple to just loop over the symbols in whatever order they're passed in, and call each method on the symbol. Though I'm not sure the reason they're pulled in groups by source, and therefore a bit reluctant to change that.

@helgasoft
Copy link

Was going to open a new entry, but saw this one which is (maybe) the enhancement I was thinking about.
When multiple symbols are requested, getSymbols stops on the first encountered problem, like this:
> getSymbols(c('F','delisted','GE'), from='2017-02-22')
Warning: delisted download failed; trying again.
Error: delisted download failed after two attempts. Error message:
HTTP error 404.

Wouldn't it be better if it continues with all symbols and spits out the error(s) at the end ? Companies are delisted often and we do not always know exactly when that happened. Finishing the list and issuing a message like 'X, Y, Z failed' will help us users update our records and not ask for a delisted symbol again.

@joshuaulrich
Copy link
Owner Author

@helgasoft Yes, this issue is to implement what you describe. I started working on it, then ran into the issue I described in my first comment.

@helgasoft
Copy link

helgasoft commented Jul 29, 2017

FYI - this is working well, skipping delisted/misspelled symbols:

library(tidyquant)
mylist <- c("FB", "AMZN", "dummy", "GOOG") %>%
	tq_get(get="stock.prices", from="2017-01-01", to="2017-05-01")

Warning message:
x = 'dummy', get = 'stock.prices': Error: dummy download failed after two attempts. Error message:
HTTP error 404.
Removing dummy.

@khanh-ld
Copy link

@helgasoft you may use this to work around this in the mean time:
library(quantmod)
e <- new.env()

lapply(ticker_list, function(x){
try(
getSymbols(
x,
env=WoW),
silent=TRUE)
})

@joshuaulrich joshuaulrich changed the title getSymbols: catch download.file errors getSymbols: catch errors and continue processing Apr 16, 2018
joshuaulrich added a commit that referenced this issue Apr 18, 2018
A long-term annoyance with getSymbols() is that an error downloading
or reading data for any symbol causes no data to be loaded.

Wrap the for loop body inside a try() call, and convert any errors
into warnings. Also keep track of symbols that failed to load, so we
only return ticker symbols that actually have data.

See #135.
joshuaulrich added a commit that referenced this issue Apr 18, 2018
Use the same logic as getSymbols.yahoo().

See #135.
joshuaulrich added a commit that referenced this issue Apr 18, 2018
This is similar logic as getSymbols.yahoo(), for most of the remaining
getSymbols() 'methods'. This does not include Alpha Vantage or Tiingo,
because the logic in those functions are fairly different from the
other 'methods'.

See #135.
joshuaulrich added a commit that referenced this issue Apr 18, 2018
Replace the lapply(Symbols, ...) with a for loop, so we can reuse the
same error-catching logic as the other getSymbols() 'methods'.

Also add tests to ensure all getSymbols() 'methods' catch errors
correctly. Only those that do not require a DB are tested.

Fixes #135.
@joshuaulrich
Copy link
Owner Author

One consequence of this change is that getSymbols() calls for a single ticker that used to error will now only warn and possibly error somewhere after the download (e.g. when trying to read the response, return a non-existent object, etc). Those things need to be tested and addressed before this is merged.

joshuaulrich added a commit that referenced this issue Apr 28, 2018
A long-term annoyance with getSymbols() is that an error downloading
or reading data for any symbol causes no data to be loaded.

Wrap the for loop body inside a try() call, and convert any errors
into warnings. Also keep track of symbols that failed to load, so we
only return ticker symbols that actually have data.

See #135.
joshuaulrich added a commit that referenced this issue Apr 28, 2018
Use the same logic as getSymbols.yahoo().

See #135.
joshuaulrich added a commit that referenced this issue Apr 28, 2018
This is similar logic as getSymbols.yahoo(), for most of the remaining
getSymbols() 'methods'. This does not include Alpha Vantage or Tiingo,
because the logic in those functions are fairly different from the
other 'methods'.

See #135.
joshuaulrich added a commit that referenced this issue Nov 22, 2018
It makes sense to throw an error when getSymbols() is called with only
one single ticker. The new try-catch logic only warns.

Check whether the main getSymbols() function is called with only a
single ticker symbol and pass that logical value through to the
"methods". We shouldn't check in each individual "method" because the
main function may be called with tickers for multiple sources, but
only one ticker for any single source.

See #135.
joshuaulrich added a commit that referenced this issue Nov 24, 2018
@joshuaulrich joshuaulrich added this to the Release 0.4-14 milestone Nov 24, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants