Yahoo Finance returns 502 Bad Gateway #157

Closed
joshuaulrich opened this Issue May 16, 2017 · 70 comments

Comments

Projects
None yet
@joshuaulrich
Owner

joshuaulrich commented May 16, 2017

After switching from HTTP to HTTPS (#149), it looks like Yahoo has now changed the URL.

R> getSymbols("AAPL", src="yahoo")
Error in download.file(paste(yahoo.URL, "s=", Symbols.name, "&a=", from.m,  : 
  cannot open URL 'https://ichart.finance.yahoo.com/table.csv?s=AAPL&a=0&b=01&c=2007&d=4&e=16&f=2017&g=d&q=q&y=0&z=AAPL&x=.csv'
In addition: Warning message:
In download.file(paste(yahoo.URL, "s=", Symbols.name, "&a=", from.m,  :
  cannot open URL 'https://ichart.finance.yahoo.com/table.csv?s=AAPL&a=0&b=01&c=2007&d=4&e=16&f=2017&g=d&q=q&y=0&z=AAPL&x=.csv': HTTP status was '502 Bad Gateway'

The "Download Data" link on the historical data page for AAPL is now:
https://query1.finance.yahoo.com/v7/finance/download/AAPL?period1=1492379145&period2=1494971145&interval=1d&events=history&crumb=sgdBTt48gg.

@joshuaulrich joshuaulrich added the bug label May 16, 2017

@arbitamrazian

This comment has been minimized.

Show comment
Hide comment
@arbitamrazian

arbitamrazian May 17, 2017

I also noticed this bug today, but how do we deal with the crumb parameter? It's required since after https://query1.finance.yahoo.com/v7/finance/download/AAPL?period1=1492379145&period2=1494971145&interval=1d&events=history (i.e. without the crumb param) I get
{ "finance": { "error": { "code": "Unauthorized", "description": "Invalid cookie" } } }

Any ideas here?

I also noticed this bug today, but how do we deal with the crumb parameter? It's required since after https://query1.finance.yahoo.com/v7/finance/download/AAPL?period1=1492379145&period2=1494971145&interval=1d&events=history (i.e. without the crumb param) I get
{ "finance": { "error": { "code": "Unauthorized", "description": "Invalid cookie" } } }

Any ideas here?

@joshuaulrich

This comment has been minimized.

Show comment
Hide comment
@joshuaulrich

joshuaulrich May 17, 2017

Owner

You need to establish a session with the Yahoo Finance server in order to get the crumb. That will probably require using curl (via either the RCurl or curl R package).

Owner

joshuaulrich commented May 17, 2017

You need to establish a session with the Yahoo Finance server in order to get the crumb. That will probably require using curl (via either the RCurl or curl R package).

@machuck

This comment has been minimized.

Show comment
Hide comment
@machuck

machuck May 17, 2017

Can someone post an example of how to use RCurl + getSymbols etc?

machuck commented May 17, 2017

Can someone post an example of how to use RCurl + getSymbols etc?

joshuaulrich added a commit that referenced this issue May 17, 2017

Use new Yahoo Finance URL
Yahoo changed their URL scheme to require an established session with
cookies. The 'curl' package handles all the details. First establish a
connection to the main site to get a cookie, then use that to get a
crumb, and use the crumb in the download URL. Note that times also
changed to POSIX/UNIX timestamps.

No additional changes were needed for getDividends(). getSplits() was
updated to read the new CSV format and parse the new split format.

getSymbols.yahoo() needed to re-order the columns, because Volume and
Adj.Close were swapped. It also appears the Open, High, and Low columns
are already adjusted for splits, but not dividends. Un-adjust them for
splits so the result is consistent with prior behavior.

See #157.

joshuaulrich added a commit that referenced this issue May 17, 2017

Store curl handle in .quantmodEnv
Keep a reference to the curl handle in quantmod's private environment.
This avoids having to re-establish a connection for every query.

The .getHandle() function will establish a session the first time it's
called. Then it will simply return the handle for subsequent calls.

Note that the curl package author recommends to not re-use curl
handles. I think it's okay in this case because we're always referencing
the same site. If we need to use curl for multiple sites, we should
establish handles for each, individually.

See #157.

joshuaulrich added a commit that referenced this issue May 17, 2017

Shorten 'from' in adjustOHLC and getDividends
Setting from="1900-01-01" resulted in a bad request. Pre-1950 should be
adequate for the majority of companies.

See #157.
@lompoc42

This comment has been minimized.

Show comment
Hide comment
@lompoc42

lompoc42 May 17, 2017

I'm sorry, but I'm confused as to how to use the updates. How do I update the functions with the changes you just made?

I'm sorry, but I'm confused as to how to use the updates. How do I update the functions with the changes you just made?

@joshuaulrich

This comment has been minimized.

Show comment
Hide comment
@joshuaulrich

joshuaulrich May 17, 2017

Owner

@lompoc42 one of these:

remotes::install_github("joshuaulrich/quantmod", ref="157_yahoo_502")
# or
devtools::install_github("joshuaulrich/quantmod", ref="157_yahoo_502")

Or wait until it's merged to master and use:

remotes::install_github("joshuaulrich/quantmod")
# or
devtools::install_github("joshuaulrich/quantmod")
Owner

joshuaulrich commented May 17, 2017

@lompoc42 one of these:

remotes::install_github("joshuaulrich/quantmod", ref="157_yahoo_502")
# or
devtools::install_github("joshuaulrich/quantmod", ref="157_yahoo_502")

Or wait until it's merged to master and use:

remotes::install_github("joshuaulrich/quantmod")
# or
devtools::install_github("joshuaulrich/quantmod")
@lompoc42

This comment has been minimized.

Show comment
Hide comment
@lompoc42

lompoc42 May 17, 2017

THANK YOU SO MUCH!!!!!!!!!!!

THANK YOU SO MUCH!!!!!!!!!!!

@lompoc42

This comment has been minimized.

Show comment
Hide comment
@lompoc42

lompoc42 May 17, 2017

Getting an error with the update:

> getSymbols('^GSPC',env = stock)
Error in `/.default`(fr[, 4], fr[, 6]) : 
  non-numeric argument to binary operator

> getSymbols.yahoo('^GSPC',env=stock)
Error in `/.default`(fr[, 4], fr[, 6]) : 
  non-numeric argument to binary operator

lompoc42 commented May 17, 2017

Getting an error with the update:

> getSymbols('^GSPC',env = stock)
Error in `/.default`(fr[, 4], fr[, 6]) : 
  non-numeric argument to binary operator

> getSymbols.yahoo('^GSPC',env=stock)
Error in `/.default`(fr[, 4], fr[, 6]) : 
  non-numeric argument to binary operator
@joshuaulrich

This comment has been minimized.

Show comment
Hide comment
@joshuaulrich

joshuaulrich May 17, 2017

Owner

@lompoc42 Please provide the command that causes that error. It's nearly impossible to help without it.

Owner

joshuaulrich commented May 17, 2017

@lompoc42 Please provide the command that causes that error. It's nearly impossible to help without it.

@lompoc42

This comment has been minimized.

Show comment
Hide comment
@lompoc42

lompoc42 May 17, 2017

In RStudio

>stock = new.env()
>getSymbols.yahoo(Symbols = '^GSPC',env = stock)

In RStudio

>stock = new.env()
>getSymbols.yahoo(Symbols = '^GSPC',env = stock)
@yankodev

This comment has been minimized.

Show comment
Hide comment
@yankodev

yankodev May 17, 2017

Hey Joshua, thanks for giving us quantmod, and thanks for the quick fix.

After installing the update I can download FB and APPL historical prices with, e.g., loadSymbols("FB").

I get the same error as lompoc42 when trying ^GSPC, ^GDAXI, SIE.DE, or LQD. It seems the query string varies with the ticker (just a shot in the dark, no time to check).

Cheers,
Yanko

Hey Joshua, thanks for giving us quantmod, and thanks for the quick fix.

After installing the update I can download FB and APPL historical prices with, e.g., loadSymbols("FB").

I get the same error as lompoc42 when trying ^GSPC, ^GDAXI, SIE.DE, or LQD. It seems the query string varies with the ticker (just a shot in the dark, no time to check).

Cheers,
Yanko

@lompoc42

This comment has been minimized.

Show comment
Hide comment
@lompoc42

lompoc42 May 17, 2017

Yeah, it works for some but not for others.

Got the same error also with tickers: PRWCX, VBTLX

lompoc42 commented May 17, 2017

Yeah, it works for some but not for others.

Got the same error also with tickers: PRWCX, VBTLX

joshuaulrich added a commit that referenced this issue May 17, 2017

Handle null values in file
The downloaded file may have "null" for some observations. Use the
na.strings argument to convert them to NA.

See #157.
@joshuaulrich

This comment has been minimized.

Show comment
Hide comment
@joshuaulrich

joshuaulrich May 17, 2017

Owner

The issue with the symbols listed should now be resolved. Thanks for the testing and feedback!

Owner

joshuaulrich commented May 17, 2017

The issue with the symbols listed should now be resolved. Thanks for the testing and feedback!

@markmurphynyc markmurphynyc referenced this issue in pydata/pandas-datareader May 17, 2017

Closed

Recent failure in 0.4 #330

@lompoc42

This comment has been minimized.

Show comment
Hide comment
@lompoc42

lompoc42 May 17, 2017

The '502 Bad Gateway' error is back. I didn't change anything which was working before and now no longer working.

I tried re-installing the package you referenced earlier

>devtools::install_github("joshuaulrich/quantmod", ref="157_yahoo_502")

and am getting the following error at the end

Error in dev_meta(pkg$package) : Namespace not found for quantmod. Is it loaded?

lompoc42 commented May 17, 2017

The '502 Bad Gateway' error is back. I didn't change anything which was working before and now no longer working.

I tried re-installing the package you referenced earlier

>devtools::install_github("joshuaulrich/quantmod", ref="157_yahoo_502")

and am getting the following error at the end

Error in dev_meta(pkg$package) : Namespace not found for quantmod. Is it loaded?

@lompoc42

This comment has been minimized.

Show comment
Hide comment
@lompoc42

lompoc42 May 17, 2017

Not sure what's happened, but it's not working anymore

Not sure what's happened, but it's not working anymore

@bdfden

This comment has been minimized.

Show comment
Hide comment
@bdfden

bdfden May 17, 2017

It is now working for me. Thank you for your help. However, it looks like it is only returning data going back to 2014. Also, the finance.yahoo.com site seems to be having issues returning older data.

bdfden commented May 17, 2017

It is now working for me. Thank you for your help. However, it looks like it is only returning data going back to 2014. Also, the finance.yahoo.com site seems to be having issues returning older data.

@joshuaulrich

This comment has been minimized.

Show comment
Hide comment
@joshuaulrich

joshuaulrich May 17, 2017

Owner

Not sure what's happened, but it's not working anymore

@lompoc42 it works fine for me from a fresh install on another machine. I assume you're running from RStudio; have you tried restarting RStudio? Unload/reload doesn't always work well. If that doesn't work, see if it works if you run from the GUI that comes with R and/or the terminal.

...it looks like it is only returning data going back to 2014. Also, the finance.yahoo.com site seems to be having issues returning older data.

@bdfden I just pulled data from 2007. If you're only getting data back to 2014 for a specific symbol, and you get the same behavior for the same instrument on the website, then there's no issue with quantmod.

Owner

joshuaulrich commented May 17, 2017

Not sure what's happened, but it's not working anymore

@lompoc42 it works fine for me from a fresh install on another machine. I assume you're running from RStudio; have you tried restarting RStudio? Unload/reload doesn't always work well. If that doesn't work, see if it works if you run from the GUI that comes with R and/or the terminal.

...it looks like it is only returning data going back to 2014. Also, the finance.yahoo.com site seems to be having issues returning older data.

@bdfden I just pulled data from 2007. If you're only getting data back to 2014 for a specific symbol, and you get the same behavior for the same instrument on the website, then there's no issue with quantmod.

@bdfden

This comment has been minimized.

Show comment
Hide comment
@bdfden

bdfden May 17, 2017

Agreed - this issue seems to be with Yahoo. Thanks again.

bdfden commented May 17, 2017

Agreed - this issue seems to be with Yahoo. Thanks again.

@wave-electron

This comment has been minimized.

Show comment
Hide comment
@wave-electron

wave-electron May 17, 2017

@joshuaulrich

Just want to thank you for the great work on the quantmod package. I spent a few weeks developing a shiny app that could mimic yahoo charts in April 2017...my goal was to showcase what is now possible using the shiny framework & R's immense library of packages. I'm using the highcharts R package which I also highly recommend for charting.

Here is a screen shot.

screen shot 2017-05-18 at 9 37 03 am

When I noticed the 502 Bad Gateway yesterday I feared the worse. But I've implemented your update and the app is working as per usual.

So thanks very much for all your great work!

@joshuaulrich

Just want to thank you for the great work on the quantmod package. I spent a few weeks developing a shiny app that could mimic yahoo charts in April 2017...my goal was to showcase what is now possible using the shiny framework & R's immense library of packages. I'm using the highcharts R package which I also highly recommend for charting.

Here is a screen shot.

screen shot 2017-05-18 at 9 37 03 am

When I noticed the 502 Bad Gateway yesterday I feared the worse. But I've implemented your update and the app is working as per usual.

So thanks very much for all your great work!

@ellisvalentiner

This comment has been minimized.

Show comment
Hide comment
@ellisvalentiner

ellisvalentiner May 17, 2017

@mcohen01

This comment has been minimized.

Show comment
Hide comment
@mcohen01

mcohen01 May 18, 2017

if you happen to encounter the following error while installing under OS X....

> devtools::install_github("joshuaulrich/quantmod", ref="157_yahoo_502")
Downloading GitHub repo joshuaulrich/quantmod@157_yahoo_502
from URL https://api.github.com/repos/joshuaulrich/quantmod/zipball/157_yahoo_502
Installing quantmod
trying URL 'https://cran.rstudio.com/bin/macosx/mavericks/contrib/3.3/zoo_1.8-0.tgz'
Content type 'application/x-gzip' length 887574 bytes (866 KB)
==================================================
downloaded 866 KB

Installing zoo
'/Library/Frameworks/R.framework/Resources/bin/R' --no-site-file  \
  --no-environ --no-save --no-restore --quiet CMD INSTALL  \
  '/private/var/folders/64/7v2m4_bd57z9kt_dttsc79zhrhzc_h/T/RtmpRadWAr/devtools5bc1187d120b/zoo'  \
  --library='/Library/Frameworks/R.framework/Versions/3.3/Resources/library'  \
  --install-tests 

* installing *binary* package ‘zoo’ ...
* DONE (zoo)
'/Library/Frameworks/R.framework/Resources/bin/R' --no-site-file  \
  --no-environ --no-save --no-restore --quiet CMD INSTALL  \
  '/private/var/folders/64/7v2m4_bd57z9kt_dttsc79zhrhzc_h/T/RtmpRadWAr/devtools5bc14faece28/joshuaulrich-quantmod-7e7744f'  \
  --library='/Library/Frameworks/R.framework/Versions/3.3/Resources/library'  \
  --install-tests 

* installing *source* package ‘quantmod’ ...
** libs
gfortran-4.8   -fPIC  -g -O2  -c ohlcq.f -o ohlcq.o
make: gfortran-4.8: No such file or directory
make: *** [ohlcq.o] Error 1
ERROR: compilation failed for package ‘quantmod’
* removing ‘/Library/Frameworks/R.framework/Versions/3.3/Resources/library/quantmod’
* restoring previous ‘/Library/Frameworks/R.framework/Versions/3.3/Resources/library/quantmod’
Error: Command failed (1)

the fix is to brew reinstall gcc48 --with-fortran and then add

FLIBS=""
F77="gfortran-4.8"
FC="gfortran-4.8"

to ~/.R/Makevars as described in http://stackoverflow.com/questions/23916219/os-x-package-installation-depends-on-gfortran-4-8

mcohen01 commented May 18, 2017

if you happen to encounter the following error while installing under OS X....

> devtools::install_github("joshuaulrich/quantmod", ref="157_yahoo_502")
Downloading GitHub repo joshuaulrich/quantmod@157_yahoo_502
from URL https://api.github.com/repos/joshuaulrich/quantmod/zipball/157_yahoo_502
Installing quantmod
trying URL 'https://cran.rstudio.com/bin/macosx/mavericks/contrib/3.3/zoo_1.8-0.tgz'
Content type 'application/x-gzip' length 887574 bytes (866 KB)
==================================================
downloaded 866 KB

Installing zoo
'/Library/Frameworks/R.framework/Resources/bin/R' --no-site-file  \
  --no-environ --no-save --no-restore --quiet CMD INSTALL  \
  '/private/var/folders/64/7v2m4_bd57z9kt_dttsc79zhrhzc_h/T/RtmpRadWAr/devtools5bc1187d120b/zoo'  \
  --library='/Library/Frameworks/R.framework/Versions/3.3/Resources/library'  \
  --install-tests 

* installing *binary* package ‘zoo’ ...
* DONE (zoo)
'/Library/Frameworks/R.framework/Resources/bin/R' --no-site-file  \
  --no-environ --no-save --no-restore --quiet CMD INSTALL  \
  '/private/var/folders/64/7v2m4_bd57z9kt_dttsc79zhrhzc_h/T/RtmpRadWAr/devtools5bc14faece28/joshuaulrich-quantmod-7e7744f'  \
  --library='/Library/Frameworks/R.framework/Versions/3.3/Resources/library'  \
  --install-tests 

* installing *source* package ‘quantmod’ ...
** libs
gfortran-4.8   -fPIC  -g -O2  -c ohlcq.f -o ohlcq.o
make: gfortran-4.8: No such file or directory
make: *** [ohlcq.o] Error 1
ERROR: compilation failed for package ‘quantmod’
* removing ‘/Library/Frameworks/R.framework/Versions/3.3/Resources/library/quantmod’
* restoring previous ‘/Library/Frameworks/R.framework/Versions/3.3/Resources/library/quantmod’
Error: Command failed (1)

the fix is to brew reinstall gcc48 --with-fortran and then add

FLIBS=""
F77="gfortran-4.8"
FC="gfortran-4.8"

to ~/.R/Makevars as described in http://stackoverflow.com/questions/23916219/os-x-package-installation-depends-on-gfortran-4-8

@joshuaulrich

This comment has been minimized.

Show comment
Hide comment
@joshuaulrich

joshuaulrich May 18, 2017

Owner

@mcohen01 See the second-to-last bullet in the Installation section of quantmod's README:
https://github.com/joshuaulrich/quantmod#installation

Owner

joshuaulrich commented May 18, 2017

@mcohen01 See the second-to-last bullet in the Installation section of quantmod's README:
https://github.com/joshuaulrich/quantmod#installation

@jzerp

This comment has been minimized.

Show comment
Hide comment
@jzerp

jzerp May 18, 2017

Thanks for Quantmod Joshua, I'm very new to R and I have been using Quantmod for a couple of months and its been very useful. However, I ran into the same problem other users have talked about. Just wanted to ask you if you are planning to release a new version or should I stick to the fixes you listed before. I tried to run them but I'm getting errors. Thanks again!

jzerp commented May 18, 2017

Thanks for Quantmod Joshua, I'm very new to R and I have been using Quantmod for a couple of months and its been very useful. However, I ran into the same problem other users have talked about. Just wanted to ask you if you are planning to release a new version or should I stick to the fixes you listed before. I tried to run them but I'm getting errors. Thanks again!

@richjung

This comment has been minimized.

Show comment
Hide comment
@richjung

richjung May 18, 2017

Thank you for your help. Should I make any change in Ad(get(symbol[i]]) to get prices from multiple tickers?

Thank you for your help. Should I make any change in Ad(get(symbol[i]]) to get prices from multiple tickers?

@Gabylin

This comment has been minimized.

Show comment
Hide comment
@Gabylin

Gabylin May 18, 2017

Thank you so much, it works now after installed devtools provided by Joshua. I can check stock data as usual now. But it looks like yahoo finance server miss a lot of daily information in some TW STOCKs.

Gabylin commented May 18, 2017

Thank you so much, it works now after installed devtools provided by Joshua. I can check stock data as usual now. But it looks like yahoo finance server miss a lot of daily information in some TW STOCKs.

@bdfden

This comment has been minimized.

Show comment
Hide comment
@bdfden

bdfden May 18, 2017

Seems like this might be a bigger issue with Yahoo. I have the update working, but my historical numbers have totally changed. I think this might be due to Yahoo dropping info on splits/dividends. See https://yahoo.uservoice.com/forums/382977-finance/suggestions/19310356-https-ichart-finance-yahoo-com-table-csv-s-spy-a#comments for further discussion.

bdfden commented May 18, 2017

Seems like this might be a bigger issue with Yahoo. I have the update working, but my historical numbers have totally changed. I think this might be due to Yahoo dropping info on splits/dividends. See https://yahoo.uservoice.com/forums/382977-finance/suggestions/19310356-https-ichart-finance-yahoo-com-table-csv-s-spy-a#comments for further discussion.

@datavigilante

This comment has been minimized.

Show comment
Hide comment
@datavigilante

datavigilante May 18, 2017

Sorry if this is a dumb question, but how to we deploy this update to shinyapps.io? I have it working locally, but even though I republished, I'm still having this problem on shinyapps.io... thanks!

Sorry if this is a dumb question, but how to we deploy this update to shinyapps.io? I have it working locally, but even though I republished, I'm still having this problem on shinyapps.io... thanks!

@ianarvin

This comment has been minimized.

Show comment
Hide comment
@ianarvin

ianarvin May 18, 2017

@joshuaulrich Thank you so much for all you do for the R community. I tried to add your quantmod fix for getting Yahoo stock prices and received this error. How do I fix? Thanks!

remotes::install_github("joshuaulrich/quantmod")
Downloading GitHub repo joshuaulrich/quantmod@master
Error in if (file.exists(dest) && file.mtime(dest) > file.mtime(lib) && :
missing value where TRUE/FALSE needed
In addition: Warning message:
In missing_devel_warning(pkgdir) :
Package quantmod has compiled code, but no suitable compiler(s) were found. Installation will likely fail.
Install Rtools and make sure it is in the PATH.
Error in if (file.exists(dest) && file.mtime(dest) > file.mtime(lib) && :
missing value where TRUE/FALSE needed

@joshuaulrich Thank you so much for all you do for the R community. I tried to add your quantmod fix for getting Yahoo stock prices and received this error. How do I fix? Thanks!

remotes::install_github("joshuaulrich/quantmod")
Downloading GitHub repo joshuaulrich/quantmod@master
Error in if (file.exists(dest) && file.mtime(dest) > file.mtime(lib) && :
missing value where TRUE/FALSE needed
In addition: Warning message:
In missing_devel_warning(pkgdir) :
Package quantmod has compiled code, but no suitable compiler(s) were found. Installation will likely fail.
Install Rtools and make sure it is in the PATH.
Error in if (file.exists(dest) && file.mtime(dest) > file.mtime(lib) && :
missing value where TRUE/FALSE needed

@Javdat

This comment has been minimized.

Show comment
Hide comment
@Javdat

Javdat May 18, 2017

getQuote has probably a related issue:

getQuote("AAPL")

 Error in open.connection(con, "rb") : HTTP error 404. 

The link has changed, but I am not quite sure, what is the new one, yet.

Javdat commented May 18, 2017

getQuote has probably a related issue:

getQuote("AAPL")

 Error in open.connection(con, "rb") : HTTP error 404. 

The link has changed, but I am not quite sure, what is the new one, yet.

@joshuaulrich

This comment has been minimized.

Show comment
Hide comment
@joshuaulrich

joshuaulrich May 18, 2017

Owner

@ianarvin See the second-to-last bullet in the Installation section of quantmod's README.

@Javdat Yeah, I expected something like that to happen... but didn't want to change anything until it did. That command still works for me, but Yahoo may be in the process of migrating servers and I'm still hitting one of the old ones.

@aguadavid This package's issue tracker isn't the forum for troubleshooting shinyapp issues. I don't mean to be rude, but off-topic comments make it harder for me to track comments for this issue.

@bdfden Thanks for the report! I noted that the OHL columns were now adjusted, and that made me wonder about the correctness of the new data. I would really appreciate if you (or anyone else) could do some cross-checking and/or validation!

@jzerp You should keep using the fixes on this branch. I will push an update to CRAN shortly, but I would first like to ensure the fixes are reasonably correct. I do not want to burden CRAN with multiple releases in a very short period of time.

Owner

joshuaulrich commented May 18, 2017

@ianarvin See the second-to-last bullet in the Installation section of quantmod's README.

@Javdat Yeah, I expected something like that to happen... but didn't want to change anything until it did. That command still works for me, but Yahoo may be in the process of migrating servers and I'm still hitting one of the old ones.

@aguadavid This package's issue tracker isn't the forum for troubleshooting shinyapp issues. I don't mean to be rude, but off-topic comments make it harder for me to track comments for this issue.

@bdfden Thanks for the report! I noted that the OHL columns were now adjusted, and that made me wonder about the correctness of the new data. I would really appreciate if you (or anyone else) could do some cross-checking and/or validation!

@jzerp You should keep using the fixes on this branch. I will push an update to CRAN shortly, but I would first like to ensure the fixes are reasonably correct. I do not want to burden CRAN with multiple releases in a very short period of time.

@lompoc42

This comment has been minimized.

Show comment
Hide comment
@lompoc42

lompoc42 May 18, 2017

@bdfden Which tickers are you looking at? I just checked multiple tickers from Yahoo's adjusted close to WIKI and Quotemedia data; all checked out fine.

Sorry, meant to caveat this by saying I only checked Adjusted Close prices.

lompoc42 commented May 18, 2017

@bdfden Which tickers are you looking at? I just checked multiple tickers from Yahoo's adjusted close to WIKI and Quotemedia data; all checked out fine.

Sorry, meant to caveat this by saying I only checked Adjusted Close prices.

@Javdat

This comment has been minimized.

Show comment
Hide comment
@Javdat

Javdat May 18, 2017

Here is a quick test (I am not sure if it should go here or get its own issue).

appl.yahoo <- getSymbols("AAPL", from = "2001-01-01", auto.assign = FALSE)
appl.google <- getSymbols("AAPL", from = "2001-01-01", auto.assign = FALSE, src = "google")
appl.quandl <- Quandl("EOD/AAPL", api_key="xxxxxxxxxxxxxxxxx", start_date="2001-01-01", type = "xts")

# Before split 09-06-2014
appl.yahoo["2014-06-02"]
appl.google["2014-06-02"]
appl.quandl["2014-06-02"]

Out:

> appl.yahoo["2014-06-02"]
           AAPL.Open AAPL.High AAPL.Low AAPL.Close AAPL.Volume AAPL.Adjusted
2014-06-02    633.96    634.83    622.5     628.65    92337700      89.80714
> appl.google["2014-06-02"]
           AAPL.Open AAPL.High AAPL.Low AAPL.Close AAPL.Volume
2014-06-02     90.57     90.69    88.93      89.81    92337903
> appl.quandl["2014-06-02"]
             Open   High   Low  Close   Volume Dividend Split Adj_Open Adj_High  Adj_Low Adj_Close Adj_Volume
2014-06-02 633.96 634.83 622.5 628.65 13191100        0     1 85.58922 85.70667 84.04203  84.87233   92337700

Splits are reflected in yahoo Adjusted, but not in OHLC.

# After split, before divdend at 2017-05-11 (ex-div)
appl.yahoo["2017-05-10::2017-05-12"]
appl.google["2017-05-10::2017-05-12"]
appl.quandl["2017-05-10::2017-05-12"]

Out:

> appl.yahoo["2017-05-10::2017-05-12"]
           AAPL.Open AAPL.High AAPL.Low AAPL.Close AAPL.Volume AAPL.Adjusted
2017-05-10    153.63    153.94   152.11     153.26    25805700        153.26
2017-05-11    152.45    154.07   152.31     153.95    27255100        153.95
2017-05-12    154.70    156.42   154.67     156.10    32527000        156.10
> appl.google["2017-05-10::2017-05-12"]
           AAPL.Open AAPL.High AAPL.Low AAPL.Close AAPL.Volume
2017-05-10    153.63    153.94   152.11     153.26    25805692
2017-05-11    152.45    154.07   152.31     153.95    27255058
2017-05-12    154.70    156.42   154.67     156.10    32527017
> appl.quandl["2017-05-10::2017-05-12"]
             Open   High    Low  Close   Volume Dividend Split Adj_Open Adj_High  Adj_Low Adj_Close Adj_Volume
2017-05-10 153.63 153.94 152.11 153.26 25805692     0.00     1 153.0039 153.3126 151.4901  152.6354   25805692
2017-05-11 152.45 154.07 152.31 153.95 27255058     0.63     1 152.4500 154.0700 152.3100  153.9500   27255058
2017-05-12 154.70 156.42 154.67 156.10 32527017     0.00     1 154.7000 156.4200 154.6700  156.1000   32527017
> 

Dividends are not reflected on OHLC or Adjusted.

Javdat commented May 18, 2017

Here is a quick test (I am not sure if it should go here or get its own issue).

appl.yahoo <- getSymbols("AAPL", from = "2001-01-01", auto.assign = FALSE)
appl.google <- getSymbols("AAPL", from = "2001-01-01", auto.assign = FALSE, src = "google")
appl.quandl <- Quandl("EOD/AAPL", api_key="xxxxxxxxxxxxxxxxx", start_date="2001-01-01", type = "xts")

# Before split 09-06-2014
appl.yahoo["2014-06-02"]
appl.google["2014-06-02"]
appl.quandl["2014-06-02"]

Out:

> appl.yahoo["2014-06-02"]
           AAPL.Open AAPL.High AAPL.Low AAPL.Close AAPL.Volume AAPL.Adjusted
2014-06-02    633.96    634.83    622.5     628.65    92337700      89.80714
> appl.google["2014-06-02"]
           AAPL.Open AAPL.High AAPL.Low AAPL.Close AAPL.Volume
2014-06-02     90.57     90.69    88.93      89.81    92337903
> appl.quandl["2014-06-02"]
             Open   High   Low  Close   Volume Dividend Split Adj_Open Adj_High  Adj_Low Adj_Close Adj_Volume
2014-06-02 633.96 634.83 622.5 628.65 13191100        0     1 85.58922 85.70667 84.04203  84.87233   92337700

Splits are reflected in yahoo Adjusted, but not in OHLC.

# After split, before divdend at 2017-05-11 (ex-div)
appl.yahoo["2017-05-10::2017-05-12"]
appl.google["2017-05-10::2017-05-12"]
appl.quandl["2017-05-10::2017-05-12"]

Out:

> appl.yahoo["2017-05-10::2017-05-12"]
           AAPL.Open AAPL.High AAPL.Low AAPL.Close AAPL.Volume AAPL.Adjusted
2017-05-10    153.63    153.94   152.11     153.26    25805700        153.26
2017-05-11    152.45    154.07   152.31     153.95    27255100        153.95
2017-05-12    154.70    156.42   154.67     156.10    32527000        156.10
> appl.google["2017-05-10::2017-05-12"]
           AAPL.Open AAPL.High AAPL.Low AAPL.Close AAPL.Volume
2017-05-10    153.63    153.94   152.11     153.26    25805692
2017-05-11    152.45    154.07   152.31     153.95    27255058
2017-05-12    154.70    156.42   154.67     156.10    32527017
> appl.quandl["2017-05-10::2017-05-12"]
             Open   High    Low  Close   Volume Dividend Split Adj_Open Adj_High  Adj_Low Adj_Close Adj_Volume
2017-05-10 153.63 153.94 152.11 153.26 25805692     0.00     1 153.0039 153.3126 151.4901  152.6354   25805692
2017-05-11 152.45 154.07 152.31 153.95 27255058     0.63     1 152.4500 154.0700 152.3100  153.9500   27255058
2017-05-12 154.70 156.42 154.67 156.10 32527017     0.00     1 154.7000 156.4200 154.6700  156.1000   32527017
> 

Dividends are not reflected on OHLC or Adjusted.

@bdfden

This comment has been minimized.

Show comment
Hide comment
@bdfden

bdfden May 18, 2017

Check out the historical data for SPXL. They did a 4:1 split on May 1, but the adjusted closing prices seem to be out of whack.

Also, I've been using the library for years to track a system that trades the SPY. I can't quite figure it out, but those results have totally changed, and they only thing I could come up with was a potential problem with the dividend.

bdfden commented May 18, 2017

Check out the historical data for SPXL. They did a 4:1 split on May 1, but the adjusted closing prices seem to be out of whack.

Also, I've been using the library for years to track a system that trades the SPY. I can't quite figure it out, but those results have totally changed, and they only thing I could come up with was a potential problem with the dividend.

@flemingcra

This comment has been minimized.

Show comment
Hide comment

flemingcra commented May 19, 2017

@joshuaulrich

This comment has been minimized.

Show comment
Hide comment
@joshuaulrich

joshuaulrich May 22, 2017

Owner

@bojanbis Yes, that's the broken version. See this comment for how to install the patched version.

@bdfden @lompoc42 See #152 for a discussion of the issue you encountered.

@herrold Thanks for the information. I'm planning on including a warning to let people know that they should begin to look for alternate data providers.

@rogerjbos Thanks for the note. I noticed that as well, and will add a warning to prevent sending an erroneous date in the request.

@offerm Thanks for the report! I will investigate.

@selgamal Thanks for trying to help! And, yes, I already attempted your suggested fix. ;)

@Hagakure0c Thanks for the note about issues with runCov(). The running/rolling functions in TTR cannot handle missing values. Those are called inhomogeneous indicators, and they aren't simple to implement. While one single missing value may be easy to handle, missing values are hard to handle in these functions in a robust and general way).

Owner

joshuaulrich commented May 22, 2017

@bojanbis Yes, that's the broken version. See this comment for how to install the patched version.

@bdfden @lompoc42 See #152 for a discussion of the issue you encountered.

@herrold Thanks for the information. I'm planning on including a warning to let people know that they should begin to look for alternate data providers.

@rogerjbos Thanks for the note. I noticed that as well, and will add a warning to prevent sending an erroneous date in the request.

@offerm Thanks for the report! I will investigate.

@selgamal Thanks for trying to help! And, yes, I already attempted your suggested fix. ;)

@Hagakure0c Thanks for the note about issues with runCov(). The running/rolling functions in TTR cannot handle missing values. Those are called inhomogeneous indicators, and they aren't simple to implement. While one single missing value may be easy to handle, missing values are hard to handle in these functions in a robust and general way).

joshuaulrich added a commit that referenced this issue May 22, 2017

Capture download error and try again
Users experience intermittent errors that appear quasi-random. The
download for a symbol may fail, but then work fine in a new R session.

Try to download the file a second time, using a new handle, if the
first attempt fails. Throw a warning if the first attempt fails, and
throw an error if the second attempt fails.

See #157.

joshuaulrich added a commit that referenced this issue May 22, 2017

Alternate server request is sent to
Send request to one of two servers, instead of always sending the
request to the same server.

See #157.

joshuaulrich added a commit that referenced this issue May 22, 2017

Remove and warn about missing values in data
Some symbols have missing values for observations that are displayed as
"0" on the site. Neither appear to be correct. Remove missing values
and warn the user about which observations were removed.

See #157.

joshuaulrich added a commit that referenced this issue May 22, 2017

Fix date-to-POSIX conversion
Using as.integer() on a POSIXct object would return NA if the date was
before/after the min/max 32-bit integer value. That would cause the
request to fail.

Use as.numeric() and trunc() instead of as.integer(), and put all the
logic in a new function. This fixes the error thrown when users would
set from="1900-01-01" (which is just slightly less than the min int).

See #157.
@leekaishing

This comment has been minimized.

Show comment
Hide comment
@leekaishing

leekaishing May 23, 2017

Hi after installing the patch from devtools::install_github("joshuaulrich/quantmod", ref="157_yahoo_502")
I found the following error message:
Error in .getHandle() : could not find function "get0"
Could you advice what could be the reason behind ? Thanks for the help.

Hi after installing the patch from devtools::install_github("joshuaulrich/quantmod", ref="157_yahoo_502")
I found the following error message:
Error in .getHandle() : could not find function "get0"
Could you advice what could be the reason behind ? Thanks for the help.

@lowcsharp

This comment has been minimized.

Show comment
Hide comment
@lowcsharp

lowcsharp May 23, 2017

Thanks to Joshua Ulrich for the timely response to Yahoo's sudden disconnect. I was able to implement his getSymbols fix and obtain price data. However, there are errors in historical data at the new location. In my case, symbol WBA has incorrect Open and Low entries (i.e: $1.00) on these dates:
in 2012 - 5/29, 7/5, 9/4, 10/31, 11/23, 12/26
in 2013 - 1/2, 1/22, 2/19, 4/1, 5/28, 7/5
We conclude that this data base is unreliable, especially for back testing purposes. and plan to investigate another source.

Thanks to Joshua Ulrich for the timely response to Yahoo's sudden disconnect. I was able to implement his getSymbols fix and obtain price data. However, there are errors in historical data at the new location. In my case, symbol WBA has incorrect Open and Low entries (i.e: $1.00) on these dates:
in 2012 - 5/29, 7/5, 9/4, 10/31, 11/23, 12/26
in 2013 - 1/2, 1/22, 2/19, 4/1, 5/28, 7/5
We conclude that this data base is unreliable, especially for back testing purposes. and plan to investigate another source.

@joshuaulrich

This comment has been minimized.

Show comment
Hide comment
@joshuaulrich

joshuaulrich May 23, 2017

Owner

@leekaishing get0 was introduced in R-3.2.0, so you need to be running a version of R that was released sometime in the past 2 years.

Owner

joshuaulrich commented May 23, 2017

@leekaishing get0 was introduced in R-3.2.0, so you need to be running a version of R that was released sometime in the past 2 years.

joshuaulrich added a commit that referenced this issue May 23, 2017

Attempt to (re-)establish a session up to 5 times
The attempt to establish a session could result in a curl handle that
does not have a cookie, in which case the user will not be able to
import data.

Establish a connection and check if it resulted in a cookie. Try again,
up to 5 times, if it does not. Throw an error if the attempt fails 5
times.

Hopefully most users will not receive errors because of this, combined
with the second attempt to import data if the first attempt fails,

See #157.
@mlb92

This comment has been minimized.

Show comment
Hide comment
@mlb92

mlb92 May 23, 2017

@joshuaulrich I have ran your updates

remotes::install_github("joshuaulrich/quantmod")
# or
devtools::install_github("joshuaulrich/quantmod")

and I am still getting the 502 Bad Gateway issue...is there anything you recommend (with an example) for how I can get it to work. Thank you.

mlb92 commented May 23, 2017

@joshuaulrich I have ran your updates

remotes::install_github("joshuaulrich/quantmod")
# or
devtools::install_github("joshuaulrich/quantmod")

and I am still getting the 502 Bad Gateway issue...is there anything you recommend (with an example) for how I can get it to work. Thank you.

@joshuaulrich

This comment has been minimized.

Show comment
Hide comment
@joshuaulrich

joshuaulrich May 23, 2017

Owner

@mlb92 You have not run the commands to install the patch. The commands you used will only work once the patch is merged, which it has not been. You need to run these commands:

remotes::install_github("joshuaulrich/quantmod", ref="157_yahoo_502")
# or
devtools::install_github("joshuaulrich/quantmod", ref="157_yahoo_502")
Owner

joshuaulrich commented May 23, 2017

@mlb92 You have not run the commands to install the patch. The commands you used will only work once the patch is merged, which it has not been. You need to run these commands:

remotes::install_github("joshuaulrich/quantmod", ref="157_yahoo_502")
# or
devtools::install_github("joshuaulrich/quantmod", ref="157_yahoo_502")

@joshuaulrich joshuaulrich added this to the 0.4-9 milestone May 23, 2017

@mlb92

This comment has been minimized.

Show comment
Hide comment
@mlb92

mlb92 May 23, 2017

@joshuaulrich I am very new to programming, could you explain what you mean by "You have not run the commands to install the patch."? I have run these lines so far. Thank you for your help:

install.packages("remotes")
install.packages("devtools")
library(remotes)
library(devtools)

After running these, I get the following errors

remotes::install_github("joshuaulrich/quantmod")
Downloading GitHub repo joshuaulrich/quantmod@master

  • installing source package ‘quantmod’ ...
    ** libs
    gfortran-4.8 -fPIC -g -O2 -c ohlcq.f -o ohlcq.o
    make: gfortran-4.8: No such file or directory
    make: *** [ohlcq.o] Error 1
    ERROR: compilation failed for package ‘quantmod’
  • removing ‘/Library/Frameworks/R.framework/Versions/3.3/Resources/library/quantmod’
  • restoring previous ‘/Library/Frameworks/R.framework/Versions/3.3/Resources/library/quantmod’
    Warning message:
    In utils::install.packages(...) :
    installation of package ‘/var/folders/41/p78vz8291wx4f9k6gpb2c_1c0000gn/T//RtmpK1HTpO/remotesd8e77ec9991/joshuaulrich-quantmod-9305034’ had non-zero exit status

devtools::install_github("joshuaulrich/quantmod")
Downloading GitHub repo joshuaulrich/quantmod@master
from URL https://api.github.com/repos/joshuaulrich/quantmod/zipball/master
Installing quantmod
'/Library/Frameworks/R.framework/Resources/bin/R' --no-site-file --no-environ --no-save --no-restore
--quiet CMD INSTALL
'/private/var/folders/41/p78vz8291wx4f9k6gpb2c_1c0000gn/T/RtmpK1HTpO/devtoolsd8e1f379e94/joshuaulrich-quantmod-9305034'
--library='/Library/Frameworks/R.framework/Versions/3.3/Resources/library' --install-tests

  • installing source package ‘quantmod’ ...
    ** libs
    gfortran-4.8 -fPIC -g -O2 -c ohlcq.f -o ohlcq.o
    make: gfortran-4.8: No such file or directory
    make: *** [ohlcq.o] Error 1
    ERROR: compilation failed for package ‘quantmod’
  • removing ‘/Library/Frameworks/R.framework/Versions/3.3/Resources/library/quantmod’
  • restoring previous ‘/Library/Frameworks/R.framework/Versions/3.3/Resources/library/quantmod’
    Installation failed: Command failed (1)

mlb92 commented May 23, 2017

@joshuaulrich I am very new to programming, could you explain what you mean by "You have not run the commands to install the patch."? I have run these lines so far. Thank you for your help:

install.packages("remotes")
install.packages("devtools")
library(remotes)
library(devtools)

After running these, I get the following errors

remotes::install_github("joshuaulrich/quantmod")
Downloading GitHub repo joshuaulrich/quantmod@master

  • installing source package ‘quantmod’ ...
    ** libs
    gfortran-4.8 -fPIC -g -O2 -c ohlcq.f -o ohlcq.o
    make: gfortran-4.8: No such file or directory
    make: *** [ohlcq.o] Error 1
    ERROR: compilation failed for package ‘quantmod’
  • removing ‘/Library/Frameworks/R.framework/Versions/3.3/Resources/library/quantmod’
  • restoring previous ‘/Library/Frameworks/R.framework/Versions/3.3/Resources/library/quantmod’
    Warning message:
    In utils::install.packages(...) :
    installation of package ‘/var/folders/41/p78vz8291wx4f9k6gpb2c_1c0000gn/T//RtmpK1HTpO/remotesd8e77ec9991/joshuaulrich-quantmod-9305034’ had non-zero exit status

devtools::install_github("joshuaulrich/quantmod")
Downloading GitHub repo joshuaulrich/quantmod@master
from URL https://api.github.com/repos/joshuaulrich/quantmod/zipball/master
Installing quantmod
'/Library/Frameworks/R.framework/Resources/bin/R' --no-site-file --no-environ --no-save --no-restore
--quiet CMD INSTALL
'/private/var/folders/41/p78vz8291wx4f9k6gpb2c_1c0000gn/T/RtmpK1HTpO/devtoolsd8e1f379e94/joshuaulrich-quantmod-9305034'
--library='/Library/Frameworks/R.framework/Versions/3.3/Resources/library' --install-tests

  • installing source package ‘quantmod’ ...
    ** libs
    gfortran-4.8 -fPIC -g -O2 -c ohlcq.f -o ohlcq.o
    make: gfortran-4.8: No such file or directory
    make: *** [ohlcq.o] Error 1
    ERROR: compilation failed for package ‘quantmod’
  • removing ‘/Library/Frameworks/R.framework/Versions/3.3/Resources/library/quantmod’
  • restoring previous ‘/Library/Frameworks/R.framework/Versions/3.3/Resources/library/quantmod’
    Installation failed: Command failed (1)
@joshuaulrich

This comment has been minimized.

Show comment
Hide comment
@joshuaulrich

joshuaulrich May 23, 2017

Owner

@mlb92 Then you should just wait until this is fixed. I told you to run remotes::install_github("joshuaulrich/quantmod", ref="157_yahoo_502") and you ran remotes::install_github("joshuaulrich/quantmod"). You also need to make sure you have the necessary tools needed to install packages with compiled code on Mac (which I have mentioned in a few other comments in this thread).

Owner

joshuaulrich commented May 23, 2017

@mlb92 Then you should just wait until this is fixed. I told you to run remotes::install_github("joshuaulrich/quantmod", ref="157_yahoo_502") and you ran remotes::install_github("joshuaulrich/quantmod"). You also need to make sure you have the necessary tools needed to install packages with compiled code on Mac (which I have mentioned in a few other comments in this thread).

@umeshnathani

This comment has been minimized.

Show comment
Hide comment
@umeshnathani

umeshnathani May 24, 2017

I ran devtools::install_github("joshuaulrich/quantmod", ref="157_yahoo_502") and ran getSymbols("AAPL"). I get the follwoing:

Error in download.file(paste(yahoo.URL, "s=", Symbols.name, "&a=", from.m, :
cannot open URL 'https://ichart.finance.yahoo.com/table.csv?s=AAPL&a=0&b=01&c=2007&d=4&e=24&f=2017&g=d&q=q&y=0&z=AAPL&x=.csv'
In addition: Warning message:
In download.file(paste(yahoo.URL, "s=", Symbols.name, "&a=", from.m, :
cannot open URL 'https://ichart.finance.yahoo.com/table.csv?s=AAPL&a=0&b=01&c=2007&d=4&e=24&f=2017&g=d&q=q&y=0&z=AAPL&x=.csv': HTTP status was '502 Connection refused'

I am using windows and latest version - R 3.4.0 (also got the same error previously with R 3.3.2)

Is yahoo! down? I tried opening https://chart.finance.yahoo.com/ and I get an error message from Yahoo! saying "Will be right back.."

umeshnathani commented May 24, 2017

I ran devtools::install_github("joshuaulrich/quantmod", ref="157_yahoo_502") and ran getSymbols("AAPL"). I get the follwoing:

Error in download.file(paste(yahoo.URL, "s=", Symbols.name, "&a=", from.m, :
cannot open URL 'https://ichart.finance.yahoo.com/table.csv?s=AAPL&a=0&b=01&c=2007&d=4&e=24&f=2017&g=d&q=q&y=0&z=AAPL&x=.csv'
In addition: Warning message:
In download.file(paste(yahoo.URL, "s=", Symbols.name, "&a=", from.m, :
cannot open URL 'https://ichart.finance.yahoo.com/table.csv?s=AAPL&a=0&b=01&c=2007&d=4&e=24&f=2017&g=d&q=q&y=0&z=AAPL&x=.csv': HTTP status was '502 Connection refused'

I am using windows and latest version - R 3.4.0 (also got the same error previously with R 3.3.2)

Is yahoo! down? I tried opening https://chart.finance.yahoo.com/ and I get an error message from Yahoo! saying "Will be right back.."

@selgamal

This comment has been minimized.

Show comment
Hide comment
@selgamal

selgamal May 24, 2017

@umeshnathani I think you might need to restart your R session after installing the branch (157_yahoo_502) or just remove quantmod package you already have and then run:
devtools::install_github("joshuaulrich/quantmod", ref="157_yahoo_502")

I just tested AAPL and it worked fine, here are the results and session info:

>  library(quantmod)

> getSymbols("AAPL")

[1] "AAPL"

> head(AAPL)

           AAPL.Open AAPL.High AAPL.Low AAPL.Close AAPL.Volume AAPL.Adjusted
2007-01-03     86.29     86.58    81.90   83.80000   309579900      11.97143
2007-01-04     84.05     85.95    83.82   85.66000   211815100      12.23714
2007-01-05     85.77     86.20    84.40   85.04999   208685400      12.15000
2007-01-08     85.96     86.53    85.28   85.47000   199276700      12.21000
2007-01-09     86.45     92.98    85.15   92.57000   837324600      13.22429
2007-01-10     94.75     97.80    93.45   97.00000   738220000      13.85714
> sessionInfo()
R version 3.4.0 (2017-04-21)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] quantmod_0.4-8 TTR_0.23-1     xts_0.9-7      zoo_1.8-0     

loaded via a namespace (and not attached):
[1] compiler_3.4.0  tools_3.4.0     curl_2.6        grid_3.4.0      lattice_0.20-35

selgamal commented May 24, 2017

@umeshnathani I think you might need to restart your R session after installing the branch (157_yahoo_502) or just remove quantmod package you already have and then run:
devtools::install_github("joshuaulrich/quantmod", ref="157_yahoo_502")

I just tested AAPL and it worked fine, here are the results and session info:

>  library(quantmod)

> getSymbols("AAPL")

[1] "AAPL"

> head(AAPL)

           AAPL.Open AAPL.High AAPL.Low AAPL.Close AAPL.Volume AAPL.Adjusted
2007-01-03     86.29     86.58    81.90   83.80000   309579900      11.97143
2007-01-04     84.05     85.95    83.82   85.66000   211815100      12.23714
2007-01-05     85.77     86.20    84.40   85.04999   208685400      12.15000
2007-01-08     85.96     86.53    85.28   85.47000   199276700      12.21000
2007-01-09     86.45     92.98    85.15   92.57000   837324600      13.22429
2007-01-10     94.75     97.80    93.45   97.00000   738220000      13.85714
> sessionInfo()
R version 3.4.0 (2017-04-21)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] quantmod_0.4-8 TTR_0.23-1     xts_0.9-7      zoo_1.8-0     

loaded via a namespace (and not attached):
[1] compiler_3.4.0  tools_3.4.0     curl_2.6        grid_3.4.0      lattice_0.20-35
@walts

This comment has been minimized.

Show comment
Hide comment
@walts

walts May 25, 2017

@joshuaulrich The current build of getSymbols.yahoo on github is removing downloaded rows which have a date but only NA prices and volumes. An example is VDIGX for 2016-06-29. This seems to be a change from last week's version which didn't remove these rows. I had been using na.approx(..., maxgap = 2) to deal with a small number of rows missing in the download and that worked for my purposes. Could removing missing rows be made an option with a warning if rows were missing rather than the fixed behavior?

walts commented May 25, 2017

@joshuaulrich The current build of getSymbols.yahoo on github is removing downloaded rows which have a date but only NA prices and volumes. An example is VDIGX for 2016-06-29. This seems to be a change from last week's version which didn't remove these rows. I had been using na.approx(..., maxgap = 2) to deal with a small number of rows missing in the download and that worked for my purposes. Could removing missing rows be made an option with a warning if rows were missing rather than the fixed behavior?

@joshuaulrich

This comment has been minimized.

Show comment
Hide comment
@joshuaulrich

joshuaulrich May 26, 2017

Owner

@walts That's a fair point. I made the change to address @ghost's comment. While you could use xts::na.restore() to put the missing values back, I think it makes more sense to return them with a warning and let the user deal with them.

Owner

joshuaulrich commented May 26, 2017

@walts That's a fair point. I made the change to address @ghost's comment. While you could use xts::na.restore() to put the missing values back, I think it makes more sense to return them with a warning and let the user deal with them.

joshuaulrich added a commit that referenced this issue May 26, 2017

Only warn about missing values, do not remove
Removing missing values assumes that's what the user prefers. The
object contains the na.omit attribute, so they could call na.restore()
to put the missing values back. However, that's more work and makes it
easier to ignore the fact that the data are suspect.

Update the warning to explain why the missing values might be a problem
and suggest possible solutions.

See #157.

joshuaulrich added a commit that referenced this issue May 26, 2017

Add 'type' argument to .yahooURL()
This allows the .yahooURL() function to be used in getDividends and
getSplits(), which reduces duplicated code. Also use the .dateToUNIX()
function to calculate the from and to POSIX dates in both functions.

See #157.

joshuaulrich added a commit that referenced this issue May 26, 2017

Add warning to first getSymbols.yahoo call
Similar to the first call to getSymbols, the first call to
getSymbols.yahoo should throw a warning alerting users to the changes
in the Yahoo Finance data. Include more details in ?getSymbols.yahoo.

Also update the message printed after the first call to getSymbols().

See #157.

@joshuaulrich joshuaulrich self-assigned this May 26, 2017

@joshuaulrich

This comment has been minimized.

Show comment
Hide comment
@joshuaulrich

joshuaulrich May 26, 2017

Owner

The 157_yahoo_502 branch is now merged to master. There may still be some edge-case weirdness, but the main issues appear to be resolved, so I'm closing this as fixed.

Thanks for everyone's feedback, testing, and support!

Owner

joshuaulrich commented May 26, 2017

The 157_yahoo_502 branch is now merged to master. There may still be some edge-case weirdness, but the main issues appear to be resolved, so I'm closing this as fixed.

Thanks for everyone's feedback, testing, and support!

@Lois47ed

This comment has been minimized.

Show comment
Hide comment
@Lois47ed

Lois47ed May 29, 2017

Hi Mr Joshualrich, I´m Lois.

Since past may 12 of 2017, WannaCry event, I can´t see this chart (Nike Inc. stok) in my browser:

http://ichart.finance.yahoo.com/z?s=NKE&t=6m&q=c&l=on&z=m&p=e50,e20&a=v,r14

Dear Joshualrich, ¿Is there any way to fix this problem?

You said that was a change of URL, this is, ¨https¨ instead of ¨http¨

Thanks for your time.

Bye for now

Hi Mr Joshualrich, I´m Lois.

Since past may 12 of 2017, WannaCry event, I can´t see this chart (Nike Inc. stok) in my browser:

http://ichart.finance.yahoo.com/z?s=NKE&t=6m&q=c&l=on&z=m&p=e50,e20&a=v,r14

Dear Joshualrich, ¿Is there any way to fix this problem?

You said that was a change of URL, this is, ¨https¨ instead of ¨http¨

Thanks for your time.

Bye for now

ceschi added a commit to ceschi/UnemplTaylor that referenced this issue May 30, 2017

Fixed issue with yahoo API
Use temporary fix from 'quantmod' pkg, hopefully to be included by
default soon.

Ref: joshuaulrich/quantmod#157

joshuaulrich added a commit that referenced this issue Jun 13, 2017

Set from to "1900-01-01" again
Setting from="1949-01-01" in 05e87ea
worked because it avoided date-to-POSIX conversion error that was fixed
in 97e4df6. The issue is that the
POSIXct value for 1900-01-01 is just slightly less than the 32-bit
integer minimum.

Return the defaults to their original value, since this isn't a problem
for the Yahoo Finance servers.

See #157.
@Harrymon12

This comment has been minimized.

Show comment
Hide comment
@Harrymon12

Harrymon12 Jul 5, 2017

Yahoo finance API is not available anymore. I have moved to MarketXLS after this change, much more reliable data.

Yahoo finance API is not available anymore. I have moved to MarketXLS after this change, much more reliable data.

@joshuaulrich

This comment has been minimized.

Show comment
Hide comment
@joshuaulrich

joshuaulrich Jul 5, 2017

Owner

@Harrymon12 It simply isn't true that the Yahoo Finance API is not available. I just ran this code:

R> quantmod::getSymbols("SPY", src = "yahoo")
[1] "SPY"
R> head(SPY)
           SPY.Open SPY.High SPY.Low SPY.Close SPY.Volume SPY.Adjusted
2007-01-03  176.779  177.537 174.691    141.37   94807600     113.7571
2007-01-04  175.512  176.531 174.741    141.67   69620600     113.9985
2007-01-05  175.636  175.723 174.455    140.54   76645300     113.0893
2007-01-08  175.002  175.735 174.294    141.19   71655000     113.6122
2007-01-09  175.611  175.971 174.480    141.07   75680100     113.5157
2007-01-10  174.704  175.934 174.356    141.54   72428000     113.8939
R> packageVersion("quantmod")
[1] '0.4.10'
Owner

joshuaulrich commented Jul 5, 2017

@Harrymon12 It simply isn't true that the Yahoo Finance API is not available. I just ran this code:

R> quantmod::getSymbols("SPY", src = "yahoo")
[1] "SPY"
R> head(SPY)
           SPY.Open SPY.High SPY.Low SPY.Close SPY.Volume SPY.Adjusted
2007-01-03  176.779  177.537 174.691    141.37   94807600     113.7571
2007-01-04  175.512  176.531 174.741    141.67   69620600     113.9985
2007-01-05  175.636  175.723 174.455    140.54   76645300     113.0893
2007-01-08  175.002  175.735 174.294    141.19   71655000     113.6122
2007-01-09  175.611  175.971 174.480    141.07   75680100     113.5157
2007-01-10  174.704  175.934 174.356    141.54   72428000     113.8939
R> packageVersion("quantmod")
[1] '0.4.10'
@Harrymon12

This comment has been minimized.

Show comment
Hide comment
@Harrymon12

Harrymon12 Aug 2, 2017

@joshuaulrich Oh I see. It came back.

@joshuaulrich Oh I see. It came back.

@chadworth

This comment has been minimized.

Show comment
Hide comment
@chadworth

chadworth Aug 27, 2017

It looks like same underlying data source is used for both the query1 and query2 APIs. E.g. the WBA issue that @lowcsharp mentioned (#157 (comment)) is an issue in both endpoints

It looks like same underlying data source is used for both the query1 and query2 APIs. E.g. the WBA issue that @lowcsharp mentioned (#157 (comment)) is an issue in both endpoints

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment