getOptionChain() working discretely "Error in open.connection(con, "rb") : HTTP error 500."

[jrburl]

Description

thanks for your work
Unable to pull option chains for a few high profile companies. With the same code, am able to pull for other companies. Does work for 'F', doesn't work for 'MSFT', 'AAPL'.

Expected behavior

Expecting a response.

Minimal, reproducible example

library(ggplot2)
library(reshape2)
library(lubridate)
library(xlsx)
library(quantmod)
library(readxl)
library(purrr)
library(dplyr)
library(plotly)
library(BatchGetSymbols)
library(derivmkts)

test_ticker <- 'AAPL'
test_total = data.frame()
spot_check <- 
  tryCatch(
  test <- as.matrix(quantmod::getOptionChain(test_ticker,Exp = NULL)),
      error=function(e) e)
  mylist = do.call(rbind,do.call(rbind,test)) %>%
    tibble::rownames_to_column(., "Name")
  
  test_total <- rbind(test_total,mylist)

  View(test_total)

[joshuaulrich]

Thanks for the report. I cannot replicate with a simple, minimal example:

x <- quantmod::getOptionChain("AAPL", Exp = NULL)
length(x)
# [1] 18
str(x[1:2])
# List of 2
#  $ Feb.21.2020:List of 2
#   ..$ calls:'data.frame':	78 obs. of  7 variables:
#   .. ..$ Strike: num [1:78] 100 105 110 120 130 135 140 150 155 160 ...
#   .. ..$ Last  : num [1:78] 224 219 214 205 194 ...
#   .. ..$ Chg   : int [1:78] 0 0 0 0 0 0 0 0 0 0 ...
#   .. ..$ Bid   : num [1:78] 0 0 0 0 0 0 0 0 0 0 ...
#   .. ..$ Ask   : int [1:78] 0 0 0 0 0 0 0 0 0 0 ...
#   .. ..$ Vol   : int [1:78] 2 NA 5 NA NA 3 7 180 NA 31 ...
#   .. ..$ OI    : int [1:78] 0 0 0 NA 0 0 0 0 0 0 ...
#   ..$ puts :'data.frame':	74 obs. of  7 variables:
#   .. ..$ Strike: num [1:74] 100 105 110 115 120 125 130 135 140 145 ...
#   .. ..$ Last  : num [1:74] 0.01 0.01 0.01 0.01 0.01 0.02 0.03 0.03 0.03 0.04 ...
#   .. ..$ Chg   : int [1:74] 0 0 0 0 0 0 0 0 0 0 ...
#   .. ..$ Bid   : num [1:74] 0 0 0 0 0 0 0 0 0 0 ...
#   .. ..$ Ask   : num [1:74] 0.01 0.04 0.01 0 0 0.08 0.08 0.01 0 0 ...
#   .. ..$ Vol   : int [1:74] 8 96 124 270 32 25 15 45 35 1 ...
#   .. ..$ OI    : int [1:74] 1724 0 193 0 0 85 89 110 0 0 ...
#  $ Feb.28.2020:List of 2
#   ..$ calls:'data.frame':	60 obs. of  7 variables:
#   .. ..$ Strike: num [1:60] 200 205 210 215 220 225 230 235 240 245 ...
#   .. ..$ Last  : num [1:60] 127 112 114 103 104 ...
#   .. ..$ Chg   : int [1:60] 0 0 0 0 0 0 0 0 0 0 ...
#   .. ..$ Bid   : num [1:60] 0 0 0 0 0 0 0 0 0 0 ...
#   .. ..$ Ask   : int [1:60] 0 0 0 0 0 0 0 0 0 0 ...
#   .. ..$ Vol   : int [1:60] 15 24 45 10 67 28 1 1 2 15 ...
#   .. ..$ OI    : int [1:60] 0 0 0 0 0 0 0 0 0 0 ...
#   ..$ puts :'data.frame':	58 obs. of  7 variables:
#   .. ..$ Strike: num [1:58] 200 205 210 215 220 225 230 235 240 245 ...
#   .. ..$ Last  : num [1:58] 0.01 0.01 0.01 0.01 0.02 0.01 0.01 0.02 0.01 0.02 ...
#   .. ..$ Chg   : int [1:58] 0 0 0 0 0 0 0 0 0 0 ...
#   .. ..$ Bid   : num [1:58] 0 0 0 0 0.01 0 0 0.03 0 0 ...
#   .. ..$ Ask   : int [1:58] 0 0 0 0 0 0 0 0 0 0 ...
#   .. ..$ Vol   : int [1:58] 9 168 602 402 402 49 29 49 7 113 ...
#   .. ..$ OI    : int [1:58] 0 0 0 0 0 0 0 0 0 0 ...

A HTTP 500 error is a generic error that suggests something went wrong with the Yahoo Finance server(s). I suspect your problem will be resolved once your request goes to a functioning Yahoo Finance server.

So I'm going to close this because I don't think there's anything I can do to help resolve your issue. Please feel free to re-open it if you think I've missed something.

Also, I'd appreciate if your minimal reproducible example were more minimal. I hope you would agree that it's not minimal to attach 11 packages (and however many their Depends also attach) that require installation of nearly 100 packages. It probably didn't matter in this case, but interactions between attached packages can potentially cause issues that are very hard to track down.

[jrburl]

Sincerely appreciate the second look. Thank you.

…

Sent from my iPhone

> On Feb 20, 2020, at 7:04 AM, Joshua Ulrich ***@***.***> wrote:  Thanks for the report. I cannot replicate with a simple, minimal example: x <- quantmod::getOptionChain("AAPL", Exp = NULL) length(x) # [1] 18 str(x[1:2]) # List of 2 # $ Feb.21.2020:List of 2 # ..$ calls:'data.frame': 78 obs. of 7 variables: # .. ..$ Strike: num [1:78] 100 105 110 120 130 135 140 150 155 160 ... # .. ..$ Last : num [1:78] 224 219 214 205 194 ... # .. ..$ Chg : int [1:78] 0 0 0 0 0 0 0 0 0 0 ... # .. ..$ Bid : num [1:78] 0 0 0 0 0 0 0 0 0 0 ... # .. ..$ Ask : int [1:78] 0 0 0 0 0 0 0 0 0 0 ... # .. ..$ Vol : int [1:78] 2 NA 5 NA NA 3 7 180 NA 31 ... # .. ..$ OI : int [1:78] 0 0 0 NA 0 0 0 0 0 0 ... # ..$ puts :'data.frame': 74 obs. of 7 variables: # .. ..$ Strike: num [1:74] 100 105 110 115 120 125 130 135 140 145 ... # .. ..$ Last : num [1:74] 0.01 0.01 0.01 0.01 0.01 0.02 0.03 0.03 0.03 0.04 ... # .. ..$ Chg : int [1:74] 0 0 0 0 0 0 0 0 0 0 ... # .. ..$ Bid : num [1:74] 0 0 0 0 0 0 0 0 0 0 ... # .. ..$ Ask : num [1:74] 0.01 0.04 0.01 0 0 0.08 0.08 0.01 0 0 ... # .. ..$ Vol : int [1:74] 8 96 124 270 32 25 15 45 35 1 ... # .. ..$ OI : int [1:74] 1724 0 193 0 0 85 89 110 0 0 ... # $ Feb.28.2020:List of 2 # ..$ calls:'data.frame': 60 obs. of 7 variables: # .. ..$ Strike: num [1:60] 200 205 210 215 220 225 230 235 240 245 ... # .. ..$ Last : num [1:60] 127 112 114 103 104 ... # .. ..$ Chg : int [1:60] 0 0 0 0 0 0 0 0 0 0 ... # .. ..$ Bid : num [1:60] 0 0 0 0 0 0 0 0 0 0 ... # .. ..$ Ask : int [1:60] 0 0 0 0 0 0 0 0 0 0 ... # .. ..$ Vol : int [1:60] 15 24 45 10 67 28 1 1 2 15 ... # .. ..$ OI : int [1:60] 0 0 0 0 0 0 0 0 0 0 ... # ..$ puts :'data.frame': 58 obs. of 7 variables: # .. ..$ Strike: num [1:58] 200 205 210 215 220 225 230 235 240 245 ... # .. ..$ Last : num [1:58] 0.01 0.01 0.01 0.01 0.02 0.01 0.01 0.02 0.01 0.02 ... # .. ..$ Chg : int [1:58] 0 0 0 0 0 0 0 0 0 0 ... # .. ..$ Bid : num [1:58] 0 0 0 0 0.01 0 0 0.03 0 0 ... # .. ..$ Ask : int [1:58] 0 0 0 0 0 0 0 0 0 0 ... # .. ..$ Vol : int [1:58] 9 168 602 402 402 49 29 49 7 113 ... # .. ..$ OI : int [1:58] 0 0 0 0 0 0 0 0 0 0 ... A HTTP 500 error is a generic error that suggests something went wrong with the Yahoo Finance server(s). I suspect your problem will be resolved once your request goes to a functioning Yahoo Finance server. So I'm going to close this because I don't think there's anything I can do to help resolve your issue. Please feel free to re-open it if you think I've missed something. Also, I'd appreciate if your minimal reproducible example were more minimal. I hope you would agree that it's not minimal to attach 11 packages (and however many their Depends also attach) that require installation of nearly 100 packages. It probably didn't matter in this case, but interactions between attached packages can potentially cause issues that are very hard to track down. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[joshuaulrich]

Here's another example of an error caused by the Yahoo Finance server response.

I ran this code 2 days ago and got an error:

R> x <- quantmod::getOptionChain("MSFT", Exp = NULL)
Error in data.frame(Strike = strike, Last = lastprice, Chg = change, Bid = bid,  : 
  object 'volume' not found

I ran it again just now and it worked. It might have something to do with the weekly options. We might be able to address that in quantmod, if we can find the root cause of the error. I'd appreciate it if you're able to investigate. If not, no worries.

[jrburl]

@joshuaulrich
quantmod::getOptionChain('PG', NULL) is failing for the same reasons. I went and checked out the option chain on Yahoo Finance and if you check out the 4/9/2020 expiry it's got near zilch for volume. No volume on calls, 1 on puts. I believe this is the culprit.

quantmod::getOptionChain('PG', Exp = "2020-04-09") fails w/ the same error & adds to this idea.

[jrburl]

Have submitted a fork request. I think this is what you're looking for after your NewToOld function:

                        Vol= if("volume" %in% names(x))
                        {
                          volume
                        }
                        else
                        {
                          NA
                        }

Also. The same needs to be done for open interest. I've seen that fail for the same reason as well.

[joshuaulrich]

I see your patch, thanks! It looks ready to merge to me, but you need to click the [Create Pull Request] button for me to be able to merge. Thank you very much for taking the initiative to fix that issue!

[joshuaulrich]

As @jrburl mentioned in the commit comment, this error occurred again. Thankfully, I can replicate, so I'm reopening.

Here's some relevant information from stepping through the debug call that fails.

debugonce(quantmod:::getOptionChain.yahoo)
oc <- quantmod::getOptionChain("FB", Exp = "2020-04-09")
...
# Browse[2]> all.expiries.posix
 [1] "2020-03-13 UTC" "2020-03-20 UTC" "2020-03-27 UTC" "2020-04-03 UTC"
 [5] "2020-04-09 UTC" "2020-04-17 UTC" "2020-04-24 UTC" "2020-05-01 UTC"
 [9] "2020-05-15 UTC" "2020-06-19 UTC" "2020-09-18 UTC" "2020-11-20 UTC"
[13] "2021-01-15 UTC" "2021-03-19 UTC" "2021-06-18 UTC" "2022-01-21 UTC"
[17] "2026-02-21 UTC"
...
Browse[2]> 
debug: return(getOptionChain.yahoo(Symbols, expiry.subset, .expiry.known = TRUE))
Browse[2]> expiry.subset
[1] 1586390400
Browse[2]> .POSIXct(expiry.subset, tz = "UTC")
[1] "2020-04-09 UTC"
Browse[2]> 
Error in open.connection(con, "rb") : HTTP error 500.

So here's some relevant information from stepping through a debugging call to getOptionChain.yahoo()

quantmod:::getOptionChain.yahoo("FB", 1586390400, .expiry.known = TRUE)
# Error in open.connection(con, "rb") : HTTP error 500.
debugonce(quantmod:::getOptionChain.yahoo)
...
Browse[2]> 
# debug: tbl <- jsonlite::fromJSON(urlExp)
Browse[2]> urlExp
# [1] "https://query2.finance.yahoo.com/v7/finance/options/FB?&date=1586390400"
Browse[2]> 
# Error in open.connection(con, "rb") : HTTP error 500.

[joshuaulrich]

Something is odd with the site. I'm able to select the 2020-04-09 expiry from the drop-down, but all the contracts listed in the output do not change. The values are for whatever expiry was last selected. In the screenshot below, that's 2020-04-03.

[jrburl]

Good evening. The flavor of the day is GOOG200501. Appears to be on yahoo's end.

[joshuaulrich]

Hey @jrburl, what do you think of the patch I just pushed?

Here's an example of the output when Exp = NULL and there are expiry dates that do not have data. In this case, oc will not have list elements named "Jun.15.2020", "Jun.17.2020", "Jun.26.2020", or "Mar.18.2022".

oc <- quantmod::getOptionChain("SPY", NULL)
Warning: no data for 'SPY' expiry 2020-06-15, omitting
	(server response: HTTP error 500.)
Warning: no data for 'SPY' expiry 2020-06-17, omitting
	(server response: HTTP error 500.)
Warning: no data for 'SPY' expiry 2020-06-26, omitting
	(server response: HTTP error 500.)
Warning: no data for 'SPY' expiry 2022-03-18, omitting
	(server response: HTTP error 500.)

[ekuutan]

It stopped working altogether for me "Error in open.connection(con, "rb") : HTTP error 500"