Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Try to workaround Yahoo! finance 404 errors, better error messages #826

wesm opened this issue Feb 24, 2012 · 2 comments


Copy link

commented Feb 24, 2012

randomly get this crap

import pandas
import as pd

def get_close(ticker, start, end):

    stock = pd.DataReader(ticker, 'yahoo', start, end)
    adj_close = stock['Adj Close']

    return adj_close[-1]

start = datetime.datetime(2010,1,1)
end = datetime.datetime(2012,1,24)
ticker = ['TLT', 'IEF', 'IYR', 'XLY', 'XLK', 'XLU', 'XLV', 'XLP', 'XLI', 'XLF', 'XLB', 'XLE', 'SPY', 'EFA', 'EEM', 'EWJ', 'IWM', 'QQQ', 'LQD', 'JNK']

for t in ticker:
    result = get_close(t, start, end)
    print t,result
## -- End pasted text --
TLT 116.17
IEF 104.15
IYR 59.52
XLY 41.25
XLK 26.87
XLU 34.47
Exception                                 Traceback (most recent call last)
/home/wesm/code/pandas/<ipython-input-7-eb9190ce71e6> in <module>()
     18 for t in ticker:
---> 19     result = get_close(t, start, end)
     20     print t,result

/home/wesm/code/pandas/<ipython-input-7-eb9190ce71e6> in get_close(ticker, start, end)
      6 def get_close(ticker, start, end):
----> 8     stock = pd.DataReader(ticker, 'yahoo', start, end)
      9     adj_close = stock['Adj Close']

/home/wesm/code/pandas/pandas/io/data.pyc in DataReader(name, data_source, start, end)
     52     if(data_source == "yahoo"):
---> 53         return get_data_yahoo(name=name, start=start, end=end)
     54     elif(data_source == "fred"):
     55         return get_data_fred(name=name, start=start, end=end)

/home/wesm/code/pandas/pandas/io/data.pyc in get_data_yahoo(name, start, end)
    134     lines = urllib.urlopen(url).read()
--> 135     return read_csv(StringIO(lines), index_col=0, parse_dates=True)[::-1]
    137 def get_data_fred(name=None, start=dt.datetime(2010, 1, 1),

/home/wesm/code/pandas/pandas/io/parsers.pyc in read_csv(filepath_or_buffer, sep, header, index_col, names, skiprows, na_values, parse_dates, date_parser, nrows, iterator, chunksize, skip_footer, converters, verbose, delimiter, encoding)
    125         return parser
--> 127     return parser.get_chunk()
    129 @Appender(_read_table_doc)

/home/wesm/code/pandas/pandas/io/parsers.pyc in get_chunk(self, rows)
    484         if len(self.columns) != len(zipped_content):
--> 485             raise Exception('wrong number of columns')
    487         data = dict((k, v) for k, v in izip(self.columns, zipped_content))

Exception: wrong number of columns

In [8]: %debug
> /home/wesm/code/pandas/pandas/io/
    484         if len(self.columns) != len(zipped_content):
--> 485             raise Exception('wrong number of columns')

ipdb> u
> /home/wesm/code/pandas/pandas/io/
--> 127     return parser.get_chunk()

> /home/wesm/code/pandas/pandas/io/
    134     lines = urllib.urlopen(url).read()
--> 135     return read_csv(StringIO(lines), index_col=0, parse_dates=True)[::-1]

ipdb> lines
'<!doctype html public "-//W3C//DTD HTML 4.01//EN" "">\n<html><head><title>Yahoo! - 404 Not Found</title><style>\n/* nn4 hide */ \n/*/*/\nbody {font:small/1.2em arial,helvetica,clean,sans-serif;font:x-small;text-align:center;}table {font-size:inherit;font:x-small;}\nhtml>body {font:83%/1.2em arial,helvetica,clean,sans-serif;}input {font-size:100%;vertical-align:middle;}p, form {margin:0;padding:0;}\np {padding-bottom:6px;margin-bottom:10px;}#doc {width:48.5em;margin:0 auto;border:1px solid #fff;text-align:center;}#ygma {text-align:right;margin-bottom:53px}\n#ygma img {float:left;}#ygma div {border-bottom:1px solid #ccc;padding-bottom:8px;margin-left:152px;}#bd {clear:both;text-align:left;width:75%;margin:0 auto 20px;}\nh1 {font-size:135%;text-align:center;margin:0 0 15px;}legend {display:none;}fieldset {border:0 solid #fff;padding:.8em 0 .8em 4.5em;}\nform {position:relative;background:#eee;margin-bottom:15px;border:1px solid #ccc;border-width:1px 0;}\n#s1p {width:15em;margin-right:.1em;}\nform span {position:absolute;left:70%;top:.8em;}form a {font:78%/1.2em arial;display:block;padding-left:.8em;white-space:nowrap;background: url( no-repeat left center;} \nform .sep {display:none;}.more {text-align:center;}#ft {padding-top:10px;border-top:1px solid #999;}#ft p {text-align:center;font:78% arial;}\n/* end nn4 hide */\n</style></head>\n<body><div id="doc">\n<div id="ygma"><a href="*"><img\nsrc=\nwidth=147 height=31 border=0 alt="Yahoo!"></a><div><a\nhref="*">Yahoo!</a>\n - <a href="*">Help</a></div></div>\n<div id="bd"><h1>Sorry, the page you requested was not found.</h1>\n<p>Please check the URL for proper spelling and capitalization. If\nyou\'re having trouble locating a destination on Yahoo!, try visiting the\n<strong><a\nhref="*">Yahoo! home\npage</a></strong> or look through a list of <strong><a\nhref="*">Yahoo!\'s\nonline services</a></strong>. Also, you may find what you\'re looking for\nif you try searching below.</p>\n<form name="s1" action="*-"><fieldset>\n<legend><label for="s1p">Search the Web</label></legend>\n<input type="text" size=30 name="p" id="s1p" title="enter search terms here">\n<input type="submit" value="Search">\n<span><a href="*">advanced search</a> <span class=sep>|</span> <a href="*">most popular</a></span>\n</fieldset></form>\n<p class="more">Please try <strong><a\nhref="*">Yahoo!\nHelp Central</a></strong> if you need more assistance.</p>\n</div><div id="ft"><p>Copyright &copy; 2012 Yahoo! Inc.\nAll rights reserved. <a\nhref="*">Privacy\nPolicy</a> - <a\nhref="*">Terms\nof Service</a></p></div>\n</div></body></html>\n'

This comment has been minimized.

Copy link

commented Jun 9, 2012

What is the desired behavior for calling code when yahoo returns a 404? Should this call fail? Retry?


This comment has been minimized.

Copy link
Member Author

commented Jun 9, 2012

It should probably retry once or twice and then raise an exception if all 2-3 attempts fail

paddymul added a commit to paddymul/pandas that referenced this issue Jun 10, 2012

@wesm wesm closed this in 0cbee12 Jun 11, 2012

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
2 participants
You can’t perform that action at this time.