Error retrieving H15 interest rates - ValueError: 'Time Period' is not in list

[rudolf-bauer]

Dear Zipline Maintainers,

Environment

• Operating System: Linux 3d62a3c1924c 4.4.0-66-generic Fixes #83, update a msgpack if it is out of date #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
• Python Version: Python 3.5.2 :: Anaconda 4.3.0 (64-bit)
• Python Bitness: 64
• How did you install Zipline: pip
• Python packages:

alabaster==0.7.9
alembic==0.9.2
anaconda-client==1.6.0
anaconda-navigator==1.4.3
appdirs==1.4.3
APScheduler==3.3.1
arctic==1.38.0
argcomplete==1.0.0
astroid==1.4.9
astropy==1.3
Babel==2.3.4
backports.shutil-get-terminal-size==1.0.0
bayesian-optimization==0.4.0
bcolz==0.12.1
beautifulsoup4==4.5.3
bitarray==0.8.1
blaze==0.10.1
bokeh==0.12.4
boto==2.45.0
Bottleneck==1.2.1
butils==0.2.13
cachetools==2.0.0
cffi==1.9.1
chardet==2.3.0
chest==0.2.3
click==6.7
cloudpickle==0.2.2
clyent==1.2.2
colorama==0.3.7
conda==4.3.9
conda-build==2.1.3
conda-verify==2.0.0
configobj==5.0.6
contextlib2==0.5.5
coverage==4.3.4
coveralls==1.1
cryptography==1.7.1
cycler==0.10.0
cyordereddict==1.0.0
Cython==0.25.2
cytoolz==0.8.2
dask==0.13.0
data==0.3.7
datashape==0.5.4
deap==1.1.0
decorator==4.0.11
dill==0.2.5
docopt==0.6.2
docutils==0.13.1
dynd==0.7.3.dev1
empyrical==0.2.2
enum34==1.1.6
et-xmlfile==1.0.1
fastcache==1.0.2
filelock==2.0.7
Flask==0.12
Flask-Cors==3.0.2
future==0.16.0
gevent==1.2.1
greenlet==0.4.11
h5py==2.6.0
HeapDict==1.0.0
hmmlearn==0.2.0
idna==2.2
imagesize==0.7.1
inspyred==1.0.1
intervaltree==2.1.0
ipykernel==4.5.2
ipython==5.1.0
ipython-genutils==0.1.0
ipywidgets==5.2.2
isort==4.2.5
itsdangerous==0.24
jdcal==1.3
jedi==0.9.0
Jinja2==2.9.4
jsonschema==2.5.1
jupyter==1.0.0
jupyter-client==4.4.0
jupyter-console==5.0.0
jupyter-core==4.2.1
jupyterhub==0.7.2
lazy-object-proxy==1.2.2
learning==0.4.19
llvmlite==0.15.0
locket==0.2.0
Logbook==1.0.0
lru-dict==1.1.6
lxml==3.7.2
lz4==0.8.2
Mako==1.0.6
MarkupSafe==1.0
matplotlib==2.0.0
minepy==1.2.0
mistune==0.7.3
mockextras==1.0.2
mpmath==0.19
multipledispatch==0.4.9
nb-anacondacloud==1.2.0
nb-conda==2.0.0
nb-conda-kernels==2.0.0
nbconvert==4.2.0
nbformat==4.2.0
nbpresent==3.0.2
networkx==1.11
nltk==3.2.2
nose==1.3.7
notebook==4.3.1
numba==0.30.1
numexpr==2.6.2
numpy==1.12.1
numpydoc==0.6.0
odo==0.5.0
openpyxl==2.4.1
packaging==16.8
pamela==0.3.0
pandas==0.18.1
pandas-datareader==0.4.0
partd==0.3.7
pathlib2==2.2.0
patsy==0.4.1
pep8==1.7.0
pexpect==4.2.1
pickleshare==0.7.4
Pillow==4.0.0
pkginfo==1.4.1
ply==3.9
prompt-toolkit==1.0.9
psutil==5.0.1
ptyprocess==0.5.1
py==1.4.32
pyasn1==0.1.9
PyBrain==0.3.3
pycosat==0.6.1
pycparser==2.17
pycrypto==2.6.1
pycurl==7.43.0
pyflakes==1.5.0
pyfolio==0.7.0
Pygments==2.1.3
pylint==1.6.4
pymongo==3.4.0
pyOpenSSL==16.2.0
pypandoc==1.3.3
pyparsing==2.2.0
pytest==3.0.5
python-dateutil==2.6.0
python-editor==1.0.3
pytz==2017.2
PyWavelets==0.5.2
PyYAML==3.12
pyyawt==0.1.1
pyzmq==16.0.2
QtAwesome==0.4.3
qtconsole==4.2.1
QtPy==1.2.1
redis==2.10.5
requests==2.14.2
requests-file==1.4.2
requests-ftp==0.3.1
rope-py3k==0.9.4.post1
rpy2==2.8.5
scikit-image==0.12.3
scikit-learn==0.18.1
scipy==0.19.0
seaborn==0.7.1
simplegeneric==0.8.1
singledispatch==3.4.0.3
six==1.10.0
snowballstemmer==1.2.1
sockjs-tornado==1.0.3
sortedcontainers==1.5.7
Sphinx==1.5.1
sphinx-rtd-theme==0.1.9
spyder==3.1.2
SQLAlchemy==1.1.10
statsmodels==0.8.0
sympy==1.0
TA-Lib==0.4.10
tables==3.4.2
terminado==0.6
toolz==0.8.2
tornado==4.4.2
TPOT==0.6.8
tqdm==4.11.2
traitlets==4.3.1
tsfresh==0.5.0
tzlocal==1.3
unicodecsv==0.14.1
update-checker==0.16
wcwidth==0.1.7
Werkzeug==0.11.15
widgetsnbextension==1.2.6
wrapt==1.10.8
xgboost==0.6a2
xlrd==1.0.0
XlsxWriter==0.9.6
xlwt==1.2.0
zipline==1.1.0

Error retrieving H15 interest rates - ValueError: 'Time Period' is not in list

• If I try to instantiate a TradingEnvironment, I get a ValueError. The code worked until yesterday and stopped some hours ago. I'm not sure whether on Th or Fr (CEST).

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)

...
env = TradingEnvironment(bm_symbol=self.benchmark, exchange_tz=self.exchange_tz,
--> 539                                  trading_calendar=cal)
...

/opt/conda/envs/develop/lib/python3.5/site-packages/zipline/finance/trading.py in __init__(self, load, bm_symbol, exchange_tz, trading_calendar, asset_db_path)
     94             trading_calendar.day,
     95             trading_calendar.schedule.index,
---> 96             self.bm_symbol,
     97         )
     98 

/opt/conda/envs/develop/lib/python3.5/site-packages/zipline/data/loader.py in load_market_data(trading_day, trading_days, bm_symbol)
    169         first_date,
    170         last_date,
--> 171         now,
    172     )
    173     benchmark_returns = br[br.index.slice_indexer(first_date, last_date)]

/opt/conda/envs/develop/lib/python3.5/site-packages/zipline/data/loader.py in ensure_treasury_data(bm_symbol, first_date, last_date, now)
    317 
    318     try:
--> 319         data = loader_module.get_treasury_data(first_date, last_date)
    320         data.to_csv(path)
    321     except (OSError, IOError, HTTPError):

/opt/conda/envs/develop/lib/python3.5/site-packages/zipline/data/treasuries.py in get_treasury_data(start_date, end_date)
     74         parse_dates=['Time Period'],
     75         na_values=['ND'],  # Presumably this stands for "No Data".
---> 76         index_col=0,
     77     ).loc[
     78         start_date:end_date

/opt/conda/envs/develop/lib/python3.5/site-packages/pandas/io/parsers.py in parser_f(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, escapechar, comment, encoding, dialect, tupleize_cols, error_bad_lines, warn_bad_lines, skipfooter, skip_footer, doublequote, delim_whitespace, as_recarray, compact_ints, use_unsigned, low_memory, buffer_lines, memory_map, float_precision)
    644                     skip_blank_lines=skip_blank_lines)
    645 
--> 646         return _read(filepath_or_buffer, kwds)
    647 
    648     parser_f.__name__ = name

/opt/conda/envs/develop/lib/python3.5/site-packages/pandas/io/parsers.py in _read(filepath_or_buffer, kwds)
    387 
    388     # Create the parser.
--> 389     parser = TextFileReader(filepath_or_buffer, **kwds)
    390 
    391     if (nrows is not None) and (chunksize is not None):

/opt/conda/envs/develop/lib/python3.5/site-packages/pandas/io/parsers.py in __init__(self, f, engine, **kwds)
    728             self.options['has_index_names'] = kwds['has_index_names']
    729 
--> 730         self._make_engine(self.engine)
    731 
    732     def close(self):

/opt/conda/envs/develop/lib/python3.5/site-packages/pandas/io/parsers.py in _make_engine(self, engine)
    921     def _make_engine(self, engine='c'):
    922         if engine == 'c':
--> 923             self._engine = CParserWrapper(self.f, **self.options)
    924         else:
    925             if engine == 'python':

/opt/conda/envs/develop/lib/python3.5/site-packages/pandas/io/parsers.py in __init__(self, src, **kwds)
   1434                 raise ValueError("Usecols do not match names.")
   1435 
-> 1436         self._set_noconvert_columns()
   1437 
   1438         self.orig_names = self.names

/opt/conda/envs/develop/lib/python3.5/site-packages/pandas/io/parsers.py in _set_noconvert_columns(self)
   1484                         _set(k)
   1485                 else:
-> 1486                     _set(val)
   1487 
   1488         elif isinstance(self.parse_dates, dict):

/opt/conda/envs/develop/lib/python3.5/site-packages/pandas/io/parsers.py in _set(x)
   1474 
   1475             if not is_integer(x):
-> 1476                 x = names.index(x)
   1477 
   1478             self._reader.set_noconvert(x)

ValueError: 'Time Period' is not in list

What steps have you taken to resolve this already?

• If I run the corresponding code manually, I get the same error.
  https://github.com/quantopian/zipline/blob/master/zipline/data/treasuries.py#L61

df = pd.read_csv(
        "http://www.federalreserve.gov/datadownload/Output.aspx"
        "?rel=H15"
        "&series=bf17364827e38702b42a58cf8eaa3f78"
        "&lastObs="
        "&from="  # An unbounded query is ~2x faster than specifying dates.
        "&to="
        "&filetype=csv"
        "&label=omit"
        "&layout=seriescolumn"
        "&type=package",
        skiprows=1,  # First row is a useless header.
        parse_dates=['Time Period'],
        na_values=['ND'],  # Presumably this stands for "No Data".
        index_col=0,
)

• The request to the hard encoded link (to obtain the H15 interest rates) fails: "Unable to find the output file. Please contact administrator for assistance."
  http://www.federalreserve.gov/datadownload/Output.aspx?rel=H15&series=bf17364827e38702b42a58cf8eaa3f78&lastObs=&from=&to=&filetype=csv&label=omit&layout=seriescolumn&type=package

• I played around with the Data Download Program and compared the parameters of the generated request strings.
  https://www.federalreserve.gov/releases/h15/

• It seems as if label=omit is not accepted anymore. If I omit the label parameter or set it to include, it seems to work.
  http://www.federalreserve.gov/datadownload/Output.aspx?rel=H15&series=bf17364827e38702b42a58cf8eaa3f78&lastObs=&from=&to=&filetype=csv&label=include&layout=seriescolumn&type=package

• I am curious if I am the only one having this problem. If not, how can we fix this problem permanently?

Sincerely,

Rudi

[ivanlen]

I am having the same problem as you, I didn't change anything in the zipline code.

[utsavkesharwani]

Same issue here as well

[sjhddh]

same here

[pbharrin]

+1

[justinlent]

Me too… perhaps the source data URL modified it’s format? -Justin

…

On Fri, May 26, 2017 at 6:34 AM, Peter Harrington ***@***.***> wrote: +1 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1817 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGPO3lDYPyKF5g17Wxr3Y7s4x_fQK7d8ks5r9tT_gaJpZM4NnCoj> .

[freddiev4]

I've just merged #1818 and also ran get_treasury_data() in master and didn't have any issues. If nobody else is seeing this I'll go ahead and close this issue 😃 (feel free to re-open if you're still seeing problems)

Also #1812

[rudolf-bauer]

Hi @freddiev4, thx a lot, this solves my problems.

[nmurray941]

Im having the same problems but in the treasury script its already set to "label=include", what else can I do to resolve this issue?

[nicolasferrari]

Hello, I'm having the same issue. While I run a backtest , I got the value error 'Time Period' is not in list.

The code work before, but now have that trouble. Any clue to solve this?

[pbharrin]

I just got this error and the treasury data did not get updated today. I copied the data from yesterday in the file ~/.zipline/data/treasury_curves.csv and it started working. (I made t-1's data exactly the same as t-2's) This isn't the best solution, but it isn't the worst thing in the world. Need to find the root cause.

[nicolasferrari]

Thanks for the response pbharrin. Yes, it was surprising because few days ago I can run the strategy and have the desire output. But today, I got that error all times.

I will try with your advice!!
Nicolás

[nicolasferrari]

Just for curious: why the source of zipline(treasury function) has that data to retrieve(the h15 interest rate), in order to run the zipline strategy?

[pbharrin]

The calculation of Sharpe ratio is (Your mean return - risk free rate)/std. So they use the treasury data to get the risk free rate. The risk free rate has been effectively 0 for so long that many calculations leave out the risk free rate. Many of the Zipline problems are caused by this and the SPY data which is only used to show a comparison of your algorithm to a benchmark, however SPY may not be the right benchmark. What if you are trading Japanese stocks, or small caps? If you are trading a long/short strategy than the SPY isn't a good benchmark as assumes long only.

[nicolasferrari]

Hello pbharrin, thanks for the explanation. I suspect now, that I have problems with the SPY data. That is my sense after reading the trace of the error that I'm receiving now. The output error is: "IndexError: index 0 is out of bounds for axis 0 with size 0". It seems that the data of SPY is not load properly.

The trace of the error is :

IndexError Traceback (most recent call last)
in ()
74 capital_base = 100000,
75 handle_data = handle_data,
---> 76 data = panel)
77

c:\users\nicolas\lib\site-packages\zipline\utils\run_algo.py in run_algorithm(start, end, initialize, capital_base, handle_data, before_trading_start, analyze, data_frequency, data, bundle, bundle_timestamp, trading_calendar, metrics_set, default_extension, extensions, strict_extensions, environ)
396 metrics_set=metrics_set,
397 local_namespace=False,
--> 398 environ=environ,
399 )

c:\users\nicolas\lib\site-packages\zipline\utils\run_algo.py in _run(handle_data, initialize, before_trading_start, analyze, algofile, algotext, defines, data_frequency, capital_base, data, bundle, bundle_timestamp, start, end, output, trading_calendar, print_algo, metrics_set, local_namespace, environ)
173 )
174 else:
--> 175 env = TradingEnvironment(environ=environ)
176 choose_loader = None
177

c:\users\nicolas\lib\site-packages\zipline\finance\trading.py in init(self, load, bm_symbol, exchange_tz, trading_calendar, asset_db_path, future_chain_predicates, environ)
97 trading_calendar.day,
98 trading_calendar.schedule.index,
---> 99 self.bm_symbol,
100 )
101

c:\users\nicolas\lib\site-packages\zipline\data\loader.py in load_market_data(trading_day, trading_days, bm_symbol, environ)
154 last_date,
155 now,
--> 156 environ,
157 )
158

c:\users\nicolas\lib\site-packages\zipline\data\loader.py in ensure_treasury_data(symbol, first_date, last_date, now, environ)
263
264 data = _load_cached_data(filename, first_date, last_date, now, 'treasury',
--> 265 environ)
266 if data is not None:
267 return data

c:\users\nicolas\lib\site-packages\zipline\data\loader.py in _load_cached_data(filename, first_date, last_date, now, resource_name, environ)
309 data = from_csv(path)
310 data.index = data.index.to_datetime().tz_localize('UTC')
--> 311 if has_data_for_dates(data, first_date, last_date):
312 return data
313

c:\users\nicolas\lib\site-packages\zipline\data\loader.py in has_data_for_dates(series_or_df, first_date, last_date)
84 if not isinstance(dts, pd.DatetimeIndex):
85 raise TypeError("Expected a DatetimeIndex, but got %s." % type(dts))
---> 86 first, last = dts[[0, -1]]
87 return (first <= first_date) and (last >= last_date)
88

c:\users\nicolas\lib\site-packages\pandas\tseries\base.py in getitem(self, key)
173 attribs['freq'] = freq
174
--> 175 result = getitem(key)
176 if result.ndim > 1:
177 return result

IndexError: index 0 is out of bounds for axis 0 with size 0

[nicolasferrari]

Any help about this?

Thanks!

[pbharrin]

IT still looks like it is an issue with the Treasury data as you have this line c:\users\nicolas\lib\site-packages\zipline\data\loader.py in ensure_treasury_data(symbol, first_date, last_date, now, environ) in the traceback.

Do you really need the treasury or SPY data? There are quick hacks that you can do to get rid of these errors.

[pbharrin]

Also I heard that the latest Zipline build no longer uses treasury data. Are you using the latest build?

[nicolasferrari]

My version of Zipline is 1.2 I think.
Is an issue with the treasury data yes, as the treasury_curves.csv file is empty, after I run the algorithm.

Im not need the treasury_data, how I can get rid of this?

[kanatm287]

Looks like fed changed request params and csv file again.

def get_treasury_data(start_date, end_date):
return pd.read_csv(
"https://www.federalreserve.gov/datadownload/Output.aspx"
"?rel=H15"
"&series=bf17364827e38702b42a58cf8eaa3f78"
"&lastObs="
"&from=" # An unbounded query is ~2x faster than specifying dates.
"&to="
"&filetype=csv"
"&label=omit"
"&layout=seriescolumn"
"&type=package",
skiprows=1, # First row are useless headers.
parse_dates=['Time Period'],
na_values=['ND'], # Presumably this stands for "No Data".
index_col=0,
).loc[
start_date:end_date
].dropna(
how='all'
).rename(
columns=parse_treasury_csv_column
).tz_localize('UTC') * 0.01 # Convert from 2.57% to 0.0257.

works with above request params as usual.

[Galsor]

Thanks kanatm287. Seems to work with me !

[pbharrin]

Thanks kanatm287 you are a hero to a generation. It looks like the param skiprows was changed from 5 to 1, did I miss anything else?

[redallica]

@pbharrin label has been changed to omit again ?
&label=omit"

Just changed skiprows to 1 and label to omit, and it is still not working for me.

Edit: actually, I just changed skiprows = 1 and kept label = include and it is still not working.
Don't know what to do else. Any help would be greatly appreciated

[nateGeorge]

Currently I'm getting this error, I think because the website isn't working properly right now. Shouldn't we be also catching the ValueError in the get_treasure_data() function in treasuries.py?
Maybe the data should be cached elsewhere? Or a copy updated with zipline with each release? Don't be able to seem to run zipline backtesting at all if we can't download this file at least once.

The URL seems to be

https://www.federalreserve.gov/datadownload/Output.aspx?rel=H15&series=bf17364827e38702b42a58cf8eaa3f78&lastObs=&from=&to=&filetype=csv&label=include&layout=seriescolumn&type=package

[imhgchoi]

Same issue here... Does this happen periodically?

[nateGeorge]

Actually, this seems to be happening on another machine I have with custom CSV data where it was working before. I guess hopefully the fed site is fixed tomorrow, but it would be good to have a better solution for handling this problem in the future.

Backtesting seems to work for the quandl dataset, but not for a custom csv dataset I have on a computer where zipline was installed and I ran the last backtest last october sometime.

[imhgchoi]

That's exactly what is going on for me too. I'm as well using my custom CSV file for the data bundle, and it worked just fine until yesterday. I hope this isn't a long term issue.

And by the way, anybody knows if it's necessary to download data from the fed site? I'm using a dataset totally irrelevant to the US fed, so maybe I can somehow block the transaction with the site? I'm not so sure what kind of data the zipline code was trying to import from fed, since I haven't looked into it previously, but I'm wondering if there are any get arounds if possible.

[ksyme99]

I have just this morning started getting this error also - not sure what started causing it since one minute it worked and the next it didn't. I'm using a custom data bundle and have no need to actually use the treasury data. Going to try and hack it out for now but having this fail more gracefully would be good.

[praghavan039]

I am facing same issue. 'Time Period' is not in list

[pbharrin]

Just a reminder the treasury data gives us the "risk free rate" which is used in the Sharpe calculation.

[nateGeorge]

Fix for the future times when the Fed's site is down: https://github.com/nateGeorge/treasury_data_backup