Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ImmediateDeprecationError: Yahoo Actions has been immediately deprecated due to large breaks in the API without the introduction of a stable replacement. #487

Closed
david2588e opened this issue Feb 14, 2018 · 38 comments · Fixed by #543
Milestone

Comments

@david2588e
Copy link

@david2588e david2588e commented Feb 14, 2018

Yahoo Finance data looks not working well not 2018/2/14:

ImmediateDeprecationError:
Yahoo Actions has been immediately deprecated due to large breaks in the API without the
introduction of a stable replacement. Pull Requests to re-enable these data
connectors are welcome.

@bashtage

This comment has been minimized.

Copy link
Contributor

@bashtage bashtage commented Feb 14, 2018

Yahoo changed their API so that the old pandas 0.5 code did not work. If you would like to fix this, PRs are welcome.

@stoffprof

This comment has been minimized.

Copy link
Contributor

@stoffprof stoffprof commented Feb 14, 2018

I am willing to work on this, but want to check if there's a known reason why my approach isn't good.

It looks like a loading a page like this:

https://finance.yahoo.com/quote/%5EGSPC/history?period1=-630961200&period2=1518498000&interval=1d&filter=history&frequency=1d

has all of the historical price data for the requested ticker stored in JSON format. (Search for "HistoricalPriceStore" in the source.) Can't we just pull that down? No need for a crumb, reading CSVs, etc.

Am I missing something?

@bashtage

This comment has been minimized.

Copy link
Contributor

@bashtage bashtage commented Feb 15, 2018

Normally PDF has avoided scraping since this seems to be especially fragile. This said I think that Yahoo was a very important contribution and so if you can reliably scrape the data, then this would be welcome in this instance.

@sudoFerraz

This comment has been minimized.

Copy link

@sudoFerraz sudoFerraz commented Feb 15, 2018

What are you guys using for FOREX currency tickers?
I was using the yahoo data for my system, and now it is all broke since then...
It was the only FREE source of data of FOREX from pdf...

@stoffprof

This comment has been minimized.

Copy link
Contributor

@stoffprof stoffprof commented Feb 17, 2018

I've been having trouble implementing this within the existing code. (I'm not an experienced coder.) But the code below works if someone can modify it as needed. Otherwise, I'll keep trying to get a useable PR.

https://gist.github.com/stoffprof/719ae8d876d3a5539112e74d292339c9

@addisonlynch

This comment has been minimized.

Copy link
Contributor

@addisonlynch addisonlynch commented Feb 17, 2018

@sudoFerraz AlphaVantage could be an option for this. Haven't gotten around to writing a PR to implement, but would be a good idea.

See their docs for more info

@luferazth

This comment has been minimized.

Copy link

@luferazth luferazth commented Feb 18, 2018

I've been coding in python and i want to repair this. What should i do?

@bashtage

This comment has been minimized.

Copy link
Contributor

@bashtage bashtage commented Feb 18, 2018

@luferazth See #489 which has what looks like a reasonable fix, given the current API available.

@sudoFerraz

This comment has been minimized.

Copy link

@sudoFerraz sudoFerraz commented Feb 19, 2018

Just updated pandas today, and datareader at least from the FOREX ticker I was using is now back to normal!
Thank you @stoffprof , that saved my skin with my client 👯‍♂

@leong0803

This comment has been minimized.

Copy link

@leong0803 leong0803 commented Feb 21, 2018

Thanks @stoffprof , that works perfectly with stocks.
Is there any other way to get currency data from yahoo please?

@david2588e

This comment has been minimized.

Copy link
Author

@david2588e david2588e commented Feb 21, 2018

Thanks for stoffprof great job !
The Yahoo API issue partly fixed, only forex data has some issues.

@stoffprof

This comment has been minimized.

Copy link
Contributor

@stoffprof stoffprof commented Feb 21, 2018

I will try to add Forex data in the next few days.

@bashtage bashtage added this to the 0.7.0 milestone Feb 21, 2018
@rsvp

This comment has been minimized.

Copy link

@rsvp rsvp commented Feb 21, 2018

@stoffprof Nice, thanks!

How are FX symbols constructed at Yahoo, say, e.g. EURJPY
for the Euro cross-rate against Japanese Yen ?

@sudoFerraz

This comment has been minimized.

Copy link

@sudoFerraz sudoFerraz commented Feb 23, 2018

What happened? It just stopped working again o.o

@bashtage

This comment has been minimized.

Copy link
Contributor

@bashtage bashtage commented Feb 23, 2018

The perils of scraping.

@stoffprof

This comment has been minimized.

Copy link
Contributor

@stoffprof stoffprof commented Feb 23, 2018

What's not working? Still fine for me.

@stoffprof stoffprof referenced this issue Feb 26, 2018
3 of 4 tasks complete
@stoffprof

This comment has been minimized.

Copy link
Contributor

@stoffprof stoffprof commented Feb 27, 2018

FX data can be accessed using YahooFXReader with PR #498
@leong0803
@david2588e
@rsvp

@sudoFerraz

This comment has been minimized.

Copy link

@sudoFerraz sudoFerraz commented Mar 2, 2018

You can reproduce the error querying the ticker
df = pandas_datareader.DataReader('BRL=X', 'yahoo')

output:
Traceback (most recent call last):
File "", line 1, in
File "/Users/ferraz/anaconda2/lib/python2.7/site-packages/pandas_datareader/data.py", line 291, in DataReader
raise ImmediateDeprecationError(DEP_ERROR_MSG.format('Yahoo Daily'))
pandas_datareader.exceptions.ImmediateDeprecationError:
Yahoo Daily has been immediately deprecated due to large breaks in the API without the
introduction of a stable replacement. Pull Requests to re-enable these data
connectors are welcome.

See https://github.com/pydata/pandas-datareader/issues

@bashtage

This comment has been minimized.

Copy link
Contributor

@bashtage bashtage commented Mar 2, 2018

That looks like 0.6.0. What is the version string of the pdr you are using?

@sudoFerraz

This comment has been minimized.

Copy link

@sudoFerraz sudoFerraz commented Mar 2, 2018

That looks like 0.6.0. What is the version string of the pdr you are using?

Yes it is :
>>> pandas_datareader.__version__ u'0.6.0'

But it is not the last one?
When I try to upgrade :
Requirement already up-to-date: pandas_datareader in /Users/ferraz/anaconda2/lib/python2.7/site-packages Requirement already up-to-date: requests-ftp in /Users/ferraz/anaconda2/lib/python2.7/site-packages (from pandas_datareader) Requirement already up-to-date: requests-file in /Users/ferraz/anaconda2/lib/python2.7/site-packages (from pandas_datareader) Requirement already up-to-date: lxml in /Users/ferraz/anaconda2/lib/python2.7/site-packages (from pandas_datareader) Requirement already up-to-date: wrapt in /Users/ferraz/anaconda2/lib/python2.7/site-packages (from pandas_datareader) Requirement already up-to-date: requests>=2.3.0 in /Users/ferraz/anaconda2/lib/python2.7/site-packages (from pandas_datareader) Requirement already up-to-date: pandas>=0.19.2 in /Users/ferraz/anaconda2/lib/python2.7/site-packages (from pandas_datareader) Requirement already up-to-date: six in /Users/ferraz/anaconda2/lib/python2.7/site-packages (from requests-file->pandas_datareader) Requirement already up-to-date: chardet<3.1.0,>=3.0.2 in /Users/ferraz/anaconda2/lib/python2.7/site-packages (from requests>=2.3.0->pandas_datareader) Requirement already up-to-date: idna<2.7,>=2.5 in /Users/ferraz/anaconda2/lib/python2.7/site-packages (from requests>=2.3.0->pandas_datareader) Requirement already up-to-date: urllib3<1.23,>=1.21.1 in /Users/ferraz/anaconda2/lib/python2.7/site-packages (from requests>=2.3.0->pandas_datareader) Requirement already up-to-date: certifi>=2017.4.17 in /Users/ferraz/anaconda2/lib/python2.7/site-packages (from requests>=2.3.0->pandas_datareader) Requirement already up-to-date: numpy>=1.9.0 in /Users/ferraz/anaconda2/lib/python2.7/site-packages (from pandas>=0.19.2->pandas_datareader) Requirement already up-to-date: python-dateutil in /Users/ferraz/anaconda2/lib/python2.7/site-packages (from pandas>=0.19.2->pandas_datareader) Requirement already up-to-date: pytz>=2011k in /Users/ferraz/anaconda2/lib/python2.7/site-packages (from pandas>=0.19.2->pandas_datareader)

@bashtage

This comment has been minimized.

Copy link
Contributor

@bashtage bashtage commented Mar 2, 2018

Yahoo was broken when 0.6 was released. Since 0.6 a patch using scraping has been merged into master. If you want to use Yahoo then you will need to install from master.

@bashtage

This comment has been minimized.

Copy link
Contributor

@bashtage bashtage commented Mar 2, 2018

pip install git+https://github.com/pydata/pandas-datareader

@sudoFerraz

This comment has been minimized.

Copy link

@sudoFerraz sudoFerraz commented Mar 2, 2018

Unfortunately that did not work

>>> pandas_datareader.__version__ u'0.6.0+4.g0debb02'

>>> df = web.DataReader('BRL=X', 'yahoo') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/ferraz/anaconda2/lib/python2.7/site-packages/pandas_datareader/data.py", line 290, in DataReader raise ImmediateDeprecationError(DEP_ERROR_MSG.format('Yahoo Daily')) pandas_datareader.exceptions.ImmediateDeprecationError: Yahoo Daily has been immediately deprecated due to large breaks in the API without the introduction of a stable replacement. Pull Requests to re-enable these data connectors are welcome.

@BrianDAllen

This comment has been minimized.

Copy link

@BrianDAllen BrianDAllen commented Mar 4, 2018

As a temporary fix, here is a wrapper I put together that utilizes scraping yahoo finance to return a pandas DataFrame of historical stock prices or dividends.

https://github.com/BrianDAllen/pandas_yahoo_historical_stocks

@jasonhei1

This comment has been minimized.

Copy link

@jasonhei1 jasonhei1 commented Mar 7, 2018

@BrianDAllen I have received an error message upon importing historical_prices_and_dividends. Would you advise what will be the possible problem? Thanks

TypeError Traceback (most recent call last)
in ()
2 df_stocks = historical_prices_and_dividends.get_historical_prices(ticker="IBM",
3 start_date=datetime.datetime(2017,1,1),
----> 4 end_date=datetime.datetime.now())
5
6 df_stocks.head()

~.....historical_prices_and_dividends.py in get_historical_prices(ticker, start_date, end_date)
33
34 # find the json object in the response, format it, and load it
---> 35 index_s = response.find("HistoricalPriceStore")
36 index_e = response.find("isPending", index_s)
37

TypeError: a bytes-like object is required, not 'str'

@sudoFerraz

This comment has been minimized.

Copy link

@sudoFerraz sudoFerraz commented Mar 13, 2018

@BrianDAllen
>>> df_stocks = historical_prices_and_dividends.get_historical_prices(ticker="BRL=X", start_date=datetime.datetime(2017,1,1), end_date=datetime.datetime.now()) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "historical_prices_and_dividends.py", line 41, in get_historical_prices mm = json.loads(json_string) File "/Users/ferraz/anaconda2/lib/python2.7/json/__init__.py", line 339, in loads return _default_decoder.decode(s) File "/Users/ferraz/anaconda2/lib/python2.7/json/decoder.py", line 364, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/Users/ferraz/anaconda2/lib/python2.7/json/decoder.py", line 382, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded

@jingsupo

This comment has been minimized.

Copy link

@jingsupo jingsupo commented Mar 19, 2018

I get data by this way:
baba

@jaycode

This comment has been minimized.

Copy link

@jaycode jaycode commented Mar 24, 2018

Hi there, some updates to empyrical module should fix this issue. See my fix commit above.

@prayank66

This comment has been minimized.

Copy link

@prayank66 prayank66 commented Mar 24, 2018

I tried with below code in Python 3:

import pandas_datareader as pdr
import datetime
start = datetime.datetime(2018,1,1)
end = datetime.date.today()
baba = pdr.get_data_yahoo('BABA',start,end)

Still getting this error:
Yahoo Actions has been immediately deprecated due to large breaks in the API without the
introduction of a stable replacement. Pull Requests to re-enable these data
connectors are welcome.

@jmsblah

This comment has been minimized.

Copy link

@jmsblah jmsblah commented Apr 4, 2018

Is there a solution for this? I am just starting with pandas_datareader.

@JeremieGauthier

This comment has been minimized.

Copy link

@JeremieGauthier JeremieGauthier commented Apr 14, 2018

from pandas_datareader import data as pdr
import plotly.offline as py_offline
import plotly.graph_objs as go
import fix_yahoo_finance as yf

py_offline.init_notebook_mode(connected=True)

yf.pdr_override()
mcd = pdr.get_data_yahoo("MCD", start="2005-07-01", end="2005-07-31")
mcd_candle = go.Candlestick(x=mcd.index,
                            open=mcd.Open,
                            high=mcd.High,
                            low=mcd.Low,
                            close=mcd.Close,
                            increasing=dict(line=dict(color= '#00FF00')),
                            decreasing=dict(line=dict(color= '#FF0000'))
                           )
data = [mcd_candle]

layout = go.Layout(
    plot_bgcolor='rgb(59,68,75)'
)

fig = go.Figure(data=data, layout=layout)

That code worked fine for me with yf.pdr_override()

@rkohli3

This comment has been minimized.

Copy link

@rkohli3 rkohli3 commented Apr 18, 2018

Could we have options Data too?

@boromirmonk

This comment has been minimized.

Copy link

@boromirmonk boromirmonk commented Apr 19, 2018

I'm having this issue. Do you have any recommendations on how I can do it differently?

@david2588e

This comment has been minimized.

Copy link
Author

@david2588e david2588e commented May 1, 2018

Thanks a lot for JeremieGauthier sharing the solution! It looks perfectly solved this issue.

@gyfeather

This comment has been minimized.

Copy link

@gyfeather gyfeather commented May 15, 2018

import pandas_datareader as pdr
import datetime
start = datetime.datetime(2018,1,1)
end = datetime.date.today()
baba = pdr.get_data_yahoo('BABA',start,end)

line 146, in _read_one_data
prices.columns = map(str.capitalize, prices.columns)
TypeError: descriptor 'capitalize' requires a 'str' object but received a 'unicode'

@bashtage

This comment has been minimized.

Copy link
Contributor

@bashtage bashtage commented May 15, 2018

Use Python 3.

@gyfeather

This comment has been minimized.

Copy link

@gyfeather gyfeather commented May 15, 2018

Thanks a lot.when i try to use Python 3,then:
File "pandas_libs\index.pyx", line 117, in pandas._libs.index.IndexEngine.get_loc
File "pandas_libs\index.pyx", line 139, in pandas._libs.index.IndexEngine.get_loc
File "pandas_libs\hashtable_class_helper.pxi", line 1265, in pandas._libs.hashtable.PyObjectHashTable.get_item
File "pandas_libs\hashtable_class_helper.pxi", line 1273, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'Data'
what is wrong with this?

@wza47

This comment has been minimized.

Copy link

@wza47 wza47 commented May 22, 2018

Had the same issue as you gyfeather. Fixed it with 2 lines of code. Add the below lines and your script should work.

import fix_yahoo_finance as yf
yf.pdr_override()

Make sure you have pip installed fix_yahoo_finance already.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.