getSymbols.av 503 errors

[alecthekulak]

Description

All calls to getSymbols.av() return errors when they are maintaining their website (every once in a while). When that happens, the code runs for around 16.9 seconds (varies) before returning the error. I would be interested in seeing code that can be run to quickly check if Alpha Vantage data is currently retrievable. That way, you could run something like this:

if( check.av() ){
  getSymbols(Symbols = x, src="av",  api.key = "api.key")
}else{
  getSymbols(Symbols = x, src="google")
}

Minimal, reproducible example

> system.time({
       getSymbols(Symbols = "MU", src="av", warnings = F,
                           periodicity = "daily", #data.type="csv",
                           adjusted = TRUE, api.key = "api.key",
                           auto.assign=T, env = .stock.data)
    })
   user  system elapsed 
   0.06    0.00   16.89 

^ Such an example is probably only reproducible during AV server maintenance,

[joshuaulrich]

This would be nice, but I don't know of an easy way to quickly check if their server is up. What do you propose?

[alecthekulak]

I figured a ping-test would do the trick, this is the format that I've been using for other such checks:

ip <- "23.23.241.244"
suppressWarnings(!as.logical(system(command = paste("ping -n 1 -w 1000", ip),
                                    show.output.on.console = FALSE)))

That IP is what I found today from an IP lookup on their site. Previously the IP address "216.239.32.21" has come up for them. Not sure why their IP would be changing, or if their public IP would be indicative of their server status, but I think it's a start.

[joshuaulrich]

Why do you need the IP instead of just the URL? If you do need the IP, then you need an automatic and cross-platform way to determine their IP. Does your command work on Windows, Mac, Linux, and BSD?

[alecthekulak]

URL could work. I was hoping the IP could be specific to the API server, rather than just to their website in general. It wouldn't be hard to loop through this function for a number of IP's if all are possible. I don't have access to other OS's to verify the code there, just on Windows.

[braverock]

This sounds more like a routing problem either on the user's end, or on Alpha Vantage's hosting. I'm not sure that a workaround for network failure belongs in getSymbols

[alecthekulak]

Well I think ultimately this sort of problem would always arise from the data provider's end. I wasn't suggesting necessarily that a workaround be implemented into getSymbols, but if such a "server check" was implemented before running getSymbols calls, you could error out rapidly, rather than waiting 7 or 8 seconds for getSymbols to give up trying to connect to a down server. This could also be implemented for other sources, such as FRED or Google Finance.

[ethanbsmith]

Seems like this should be able to be resolved by configuring timeouts.
download.file says it honors the timeout option: "The timeout for many parts of the transfer can be set by the option timeout which defaults to 60 seconds."

based on the description above saying this takes 17 seconds, there may be something else going on, but in general timeouts is how this type of networking problem is handled

[joshuaulrich]

Closing this as not planned because it would be a lot of work to figure out the best way to do this in a comprehensive, cross-platform way. And it would be weird to only have it for one data source.