getYahooData no longer works #43

humbletrader123 · 2017-04-19T01:53:29Z

Hi for last two days (started April 17, 2017) the TTR::getYahooData returns the following errors

> getYahooData("IBM", 19990404, 20050607)
Error in file(file, "rt") : cannot open the connection

There is absolutely no issue with my internet.

My TTR package version is 0.23-1
and R version is 2.15.2

I'd very much appreciate if someone takes a quick look and fix. This get function has been working daily for last 5 years.

Thanks

The text was updated successfully, but these errors were encountered:

joshuaulrich · 2017-04-19T01:56:44Z

Thanks for the report! This is likely the same as joshuaulrich/quantmod#149. Will patch this evening, but might not be on CRAN for a few more days.

Out of curiosity, why are you using a version of R that is nearly 5 years old?

humbletrader123 · 2017-04-19T02:03:03Z

Thank you for your quick response I need to upgrade it. I have tons of programs written under this version and it is very time consuming to test. Yes I will need to upgrade.

…

On Tue, Apr 18, 2017 at 9:56 PM, Joshua Ulrich ***@***.***> wrote: Thanks for the report! This is likely the same as joshuaulrich/quantmod#149 <joshuaulrich/quantmod#149>. Will patch this evening, but might not be on CRAN for a few more days. Out of curiosity, why are you using a version of R that is nearly 5 years old? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#43 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AadzdoyCMAb23Ab0OJF62I-VQybbqV7Lks5rxWndgaJpZM4NBGkK> .

Yahoo Finance is now redirecting HTTP requests to HTTPS. Some users' R installations are not able to follow this redirect, so they error. Use HTTPS directly for Yahoo Finance. Fixes #43.

joshuaulrich · 2017-04-19T02:54:02Z

Okay, just pushed a patch to branch 43_yahoo_https. Note that HTTPS wasn't supported and used as a default on all platforms until R-3.3.0. So it's quite possible you will still have issues until you upgrade...

humbletrader123 · 2017-05-18T01:09:18Z

Hi this does not work starting two days ago I've since upgraded to 3.3.3 and also applied two patches "joshuaulrich/quantmod", ref="157_yahoo_502" "joshuaulrich/TTR", ref="43_yahoo_https" But I got the following errors:

getYahooData('FB',start='20170101',end='20170501')

Error in file(file, "rt") : cannot open the connection In addition: Warning message: In file(file, "rt") : cannot open URL ' https://ichart.finance.yahoo.com/table.csv?s=FB&a=0&b=1&c=2017&d=4&e=17&f=2017&g=d&ignore=.csv': HTTP status was '502 Connection refused' Error in download.file(paste(yahoo.URL, "s=", Symbols.name, "&a=", from.m, : cannot open URL 'https://ichart.finance.yahoo.com/table.csv?s= ^GSPC&a=0&b=01&c=2007&d=4&e=17&f=2017&g=d&q=q&y=0&z=^GSPC&x=.csv' In addition: Warning message: In download.file(paste(yahoo.URL, "s=", Symbols.name, "&a=", from.m, : cannot open URL 'https://ichart.finance.yahoo.com/table.csv?s=^GSPC&a=0&b=01&c=2007&d=4&e=17&f=2017&g=d&q=q&y=0&z=^GSPC&x=.csv': HTTP status was '502 Connection refused'

sessionInfo()

R version 3.3.3 (2017-03-06) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows 7 x64 (build 7601) Service Pack 1 locale: [1] LC_COLLATE=English_Canada.1252 LC_CTYPE=English_Canada.1252 LC_MONETARY=English_Canada.1252 LC_NUMERIC=C [5] LC_TIME=English_Canada.1252 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] PerformanceAnalytics_1.4.3541 nortest_1.0-4 quantmod_0.4-8 xts_0.9-7 [5] zoo_1.8-0 moments_0.14 TTR_0.23-1 loaded via a namespace (and not attached): [1] tools_3.3.3 remotes_1.0.0 grid_3.3.3 lattice_0.20-35

…

On Tue, Apr 18, 2017 at 10:54 PM, Joshua Ulrich ***@***.***> wrote: Okay, just pushed a patch to branch 43_yahoo_https. Note that HTTPS wasn't supported and used as a default on all platforms until R-3.3.0. So it's quite possible you will still have issues until you upgrade... — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#43 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AadzdhzYB3mSC3Mc5vvsGMipP3KDyoYNks5rxXdLgaJpZM4NBGkK> .

joshuaulrich · 2017-05-18T02:08:25Z

@humbletrader123 It's the same issue as quantmod:157_yahoo_502, but I have to patch it in TTR too. The fix in quantmod doesn't apply to the TTR functions.

ghost · 2017-05-22T12:11:35Z

Joshua, thank you, fix 157 works well for downloading data off Yahoo again.. Somehow it did break something, when calculating historical volatility on a vector of logreturns: xts(apply(retGSPC,2,runSD,n=30),index(retGSPC))*sqrt(252) it gives a Error in runCov(x, x, n, use = "all.obs", sample = sample, cumulative) : Series contain non-leading NAs. The runCov function can't handle NAs and how did the quantmod patch change that ?

joshuaulrich · 2017-05-29T15:21:40Z

@ghost The running/rolling functions in TTR cannot handle missing values. Those are called inhomogeneous indicators, and they aren't simple to implement. While one single missing value may be easy to handle, missing values are hard to handle in these functions in a robust and general way.

humbletrader123 · 2017-05-29T19:28:59Z

Thanks Joshua for supporting this package for last few years I realized the days of free yahoo quotes are over.

…

Sent from my Samsung device

-------- Original message -------- From: Joshua Ulrich <notifications@github.com> Date: 05-29-2017 11:21 AM (GMT-05:00) To: joshuaulrich/TTR <TTR@noreply.github.com> Cc: humbletrader123 <lawrence.sum@gmail.com>, Mention <mention@noreply.github.com> Subject: Re: [joshuaulrich/TTR] getYahooData no longer works (#43) @ghost The running/rolling functions in TTR cannot handle missing values. Those are called inhomogeneous indicators, and they aren't simple to implement. While one single missing value may be easy to handle, missing values are hard to handle in these functions in a robust and general way. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread. {"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/joshuaulrich/TTR","title":"joshuaulrich/TTR","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/joshuaulrich/TTR"}},"updates":{"snippets":[{"icon":"PERSON","message":"@joshuaulrich in #43: @ghost The running/rolling functions in TTR cannot handle missing values. Those are called inhomogeneous indicators, and they aren't simple to implement. While one single missing value may be easy to handle, missing values are hard to handle in these functions in a robust and general way."}],"action":{"name":"View Issue","url":"#43 (comment)"}}}

This commit simply uses code from the quantmod::getSymbols.yahoo fix from the quantmod repository. See these commits for details: 793601805c5d7291d5cda22ce4df4001a40ff312..aa19272d170d9b35be4279e5abbe310182e08c4f Fixes #43.

getYahooData() does essentially the same thing as getSymbols.yahoo(), and it doesn't make sense to maintain two interfaces to the same data source. This will also make it clear that quantmod is the package for data management. See #43.

This commit simply uses code from the quantmod::getSymbols.yahoo fix from the quantmod repository. See these commits for details: 793601805c5d7291d5cda22ce4df4001a40ff312..aa19272d170d9b35be4279e5abbe310182e08c4f Fixes #43.

getYahooData() does essentially the same thing as getSymbols.yahoo(), and it doesn't make sense to maintain two interfaces to the same data source. This will also make it clear that quantmod is the package for data management. See #43.

Trying to fetch the Yahoo Finance! homepage multiple times can result in getting a cached response from a proxy. The cached response does not have a "Set-cookie" header, so the curl handle will still not have a cookie and the connection will be retried. This StackOverflow Q/A has two approaches to avoid a cache hit: https://stackoverflow.com/q/31653271/271616 1) Add a "Cache-control: no-cache" header 2) Add a random query to the URL. The first approach didn't work because it appears the Yahoo proxies simply ignore the request. The second approach did work though. Replace get0() call, so dependency on R>=3.2.0 isn't required. See #43.

Looks like Yahoo does not provide data for IBM, so switch to using GE instead. Also remove the 'dontrun' from the example and set adjust = FALSE so the examples run faster. See #43.

The new Yahoo URL uses UNIX timestamps, but the OHLC subset right before exiting the function assumed the 'start' and 'end' objects are Date. See #43.

joshuaulrich added the bug label Apr 19, 2017

joshuaulrich self-assigned this Apr 19, 2017

joshuaulrich closed this as completed in 310ffd9 May 29, 2017

joshuaulrich added a commit that referenced this issue Jul 10, 2017

Use GE instead of IBM in examples

f5faf17

Looks like Yahoo does not provide data for IBM, so switch to using GE instead. Also remove the 'dontrun' from the example and set adjust = FALSE so the examples run faster. See #43.

joshuaulrich added a commit that referenced this issue Jul 10, 2017

Fix subsetting bug related to date->POSIXct

0c39061

The new Yahoo URL uses UNIX timestamps, but the OHLC subset right before exiting the function assumed the 'start' and 'end' objects are Date. See #43.

joshuaulrich added this to the 0.23-2 milestone Sep 9, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

getYahooData no longer works #43

getYahooData no longer works #43

humbletrader123 commented Apr 19, 2017 •

edited by joshuaulrich

Loading

joshuaulrich commented Apr 19, 2017

humbletrader123 commented Apr 19, 2017 via email

joshuaulrich commented Apr 19, 2017

humbletrader123 commented May 18, 2017 via email

joshuaulrich commented May 18, 2017

ghost commented May 22, 2017 •

edited by ghost

Loading

joshuaulrich commented May 29, 2017

humbletrader123 commented May 29, 2017 via email

getYahooData no longer works #43

getYahooData no longer works #43

Comments

humbletrader123 commented Apr 19, 2017 • edited by joshuaulrich Loading

joshuaulrich commented Apr 19, 2017

humbletrader123 commented Apr 19, 2017 via email

joshuaulrich commented Apr 19, 2017

humbletrader123 commented May 18, 2017 via email

joshuaulrich commented May 18, 2017

ghost commented May 22, 2017 • edited by ghost Loading

joshuaulrich commented May 29, 2017

humbletrader123 commented May 29, 2017 via email

humbletrader123 commented Apr 19, 2017 •

edited by joshuaulrich

Loading

ghost commented May 22, 2017 •

edited by ghost

Loading