Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Try to workaround Yahoo! finance 404 errors, better error messages #826

Closed
wesm opened this issue Feb 24, 2012 · 2 comments

Comments

@wesm
Copy link
Member

commented Feb 24, 2012

randomly get this crap


import pandas
import pandas.io.data as pd

def get_close(ticker, start, end):

    stock = pd.DataReader(ticker, 'yahoo', start, end)
    adj_close = stock['Adj Close']

    return adj_close[-1]


start = datetime.datetime(2010,1,1)
end = datetime.datetime(2012,1,24)
ticker = ['TLT', 'IEF', 'IYR', 'XLY', 'XLK', 'XLU', 'XLV', 'XLP', 'XLI', 'XLF', 'XLB', 'XLE', 'SPY', 'EFA', 'EEM', 'EWJ', 'IWM', 'QQQ', 'LQD', 'JNK']

for t in ticker:
    result = get_close(t, start, end)
    print t,result
## -- End pasted text --
TLT 116.17
IEF 104.15
IYR 59.52
XLY 41.25
XLK 26.87
XLU 34.47
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
/home/wesm/code/pandas/<ipython-input-7-eb9190ce71e6> in <module>()
     17 
     18 for t in ticker:
---> 19     result = get_close(t, start, end)
     20     print t,result

/home/wesm/code/pandas/<ipython-input-7-eb9190ce71e6> in get_close(ticker, start, end)
      6 def get_close(ticker, start, end):
      7 
----> 8     stock = pd.DataReader(ticker, 'yahoo', start, end)
      9     adj_close = stock['Adj Close']
     10 

/home/wesm/code/pandas/pandas/io/data.pyc in DataReader(name, data_source, start, end)
     51 
     52     if(data_source == "yahoo"):
---> 53         return get_data_yahoo(name=name, start=start, end=end)
     54     elif(data_source == "fred"):
     55         return get_data_fred(name=name, start=start, end=end)

/home/wesm/code/pandas/pandas/io/data.pyc in get_data_yahoo(name, start, end)
    133 
    134     lines = urllib.urlopen(url).read()
--> 135     return read_csv(StringIO(lines), index_col=0, parse_dates=True)[::-1]
    136 
    137 def get_data_fred(name=None, start=dt.datetime(2010, 1, 1),

/home/wesm/code/pandas/pandas/io/parsers.pyc in read_csv(filepath_or_buffer, sep, header, index_col, names, skiprows, na_values, parse_dates, date_parser, nrows, iterator, chunksize, skip_footer, converters, verbose, delimiter, encoding)
    125         return parser
    126 
--> 127     return parser.get_chunk()
    128 
    129 @Appender(_read_table_doc)

/home/wesm/code/pandas/pandas/io/parsers.pyc in get_chunk(self, rows)
    483 
    484         if len(self.columns) != len(zipped_content):
--> 485             raise Exception('wrong number of columns')
    486 
    487         data = dict((k, v) for k, v in izip(self.columns, zipped_content))

Exception: wrong number of columns

In [8]: %debug
> /home/wesm/code/pandas/pandas/io/parsers.py(485)get_chunk()
    484         if len(self.columns) != len(zipped_content):
--> 485             raise Exception('wrong number of columns')
    486 

ipdb> u
> /home/wesm/code/pandas/pandas/io/parsers.py(127)read_csv()
    126 
--> 127     return parser.get_chunk()
    128 

ipdb> 
> /home/wesm/code/pandas/pandas/io/data.py(135)get_data_yahoo()
    134     lines = urllib.urlopen(url).read()
--> 135     return read_csv(StringIO(lines), index_col=0, parse_dates=True)[::-1]
    136 

ipdb> lines
'<!doctype html public "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">\n<html><head><title>Yahoo! - 404 Not Found</title><style>\n/* nn4 hide */ \n/*/*/\nbody {font:small/1.2em arial,helvetica,clean,sans-serif;font:x-small;text-align:center;}table {font-size:inherit;font:x-small;}\nhtml>body {font:83%/1.2em arial,helvetica,clean,sans-serif;}input {font-size:100%;vertical-align:middle;}p, form {margin:0;padding:0;}\np {padding-bottom:6px;margin-bottom:10px;}#doc {width:48.5em;margin:0 auto;border:1px solid #fff;text-align:center;}#ygma {text-align:right;margin-bottom:53px}\n#ygma img {float:left;}#ygma div {border-bottom:1px solid #ccc;padding-bottom:8px;margin-left:152px;}#bd {clear:both;text-align:left;width:75%;margin:0 auto 20px;}\nh1 {font-size:135%;text-align:center;margin:0 0 15px;}legend {display:none;}fieldset {border:0 solid #fff;padding:.8em 0 .8em 4.5em;}\nform {position:relative;background:#eee;margin-bottom:15px;border:1px solid #ccc;border-width:1px 0;}\n#s1p {width:15em;margin-right:.1em;}\nform span {position:absolute;left:70%;top:.8em;}form a {font:78%/1.2em arial;display:block;padding-left:.8em;white-space:nowrap;background: url(http://l.yimg.com/a/i/s/bullet.gif) no-repeat left center;} \nform .sep {display:none;}.more {text-align:center;}#ft {padding-top:10px;border-top:1px solid #999;}#ft p {text-align:center;font:78% arial;}\n/* end nn4 hide */\n</style></head>\n<body><div id="doc">\n<div id="ygma"><a href="http://us.rd.yahoo.com/404/*http://www.yahoo.com"><img\nsrc=http://l.yimg.com/a/i/yahoo.gif\nwidth=147 height=31 border=0 alt="Yahoo!"></a><div><a\nhref="http://us.rd.yahoo.com/404/*http://www.yahoo.com">Yahoo!</a>\n - <a href="http://us.rd.yahoo.com/404/*http://help.yahoo.com">Help</a></div></div>\n<div id="bd"><h1>Sorry, the page you requested was not found.</h1>\n<p>Please check the URL for proper spelling and capitalization. If\nyou\'re having trouble locating a destination on Yahoo!, try visiting the\n<strong><a\nhref="http://us.rd.yahoo.com/404/*http://www.yahoo.com">Yahoo! home\npage</a></strong> or look through a list of <strong><a\nhref="http://us.rd.yahoo.com/404/*http://docs.yahoo.com/docs/family/more/">Yahoo!\'s\nonline services</a></strong>. Also, you may find what you\'re looking for\nif you try searching below.</p>\n<form name="s1" action="http://us.rd.yahoo.com/404/*-http://search.yahoo.com/search"><fieldset>\n<legend><label for="s1p">Search the Web</label></legend>\n<input type="text" size=30 name="p" id="s1p" title="enter search terms here">\n<input type="submit" value="Search">\n<span><a href="http://us.rd.yahoo.com/404/*http://search.yahoo.com/search/options?p=">advanced search</a> <span class=sep>|</span> <a href="http://us.rd.yahoo.com/404/*http://buzz.yahoo.com">most popular</a></span>\n</fieldset></form>\n<p class="more">Please try <strong><a\nhref="http://us.rd.yahoo.com/404/*http://help.yahoo.com">Yahoo!\nHelp Central</a></strong> if you need more assistance.</p>\n</div><div id="ft"><p>Copyright &copy; 2012 Yahoo! Inc.\nAll rights reserved. <a\nhref="http://us.rd.yahoo.com/404/*http://privacy.yahoo.com">Privacy\nPolicy</a> - <a\nhref="http://us.rd.yahoo.com/404/*http://docs.yahoo.com/info/terms/">Terms\nof Service</a></p></div>\n</div></body></html>\n'
ipdb> 
@paddymul

This comment has been minimized.

Copy link
Contributor

commented Jun 9, 2012

What is the desired behavior for calling code when yahoo returns a 404? Should this call fail? Retry?

@wesm

This comment has been minimized.

Copy link
Member Author

commented Jun 9, 2012

It should probably retry once or twice and then raise an exception if all 2-3 attempts fail

paddymul added a commit to paddymul/pandas that referenced this issue Jun 10, 2012

@wesm wesm closed this in 0cbee12 Jun 11, 2012

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.