New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: Adds quantopian-quandl bundle as new default. #1178

Merged
merged 6 commits into from May 6, 2016

Conversation

Projects
None yet
4 participants
@llllllllll
Member

llllllllll commented May 4, 2016

This data bundle will use the quantopian mirror of the quandl WIKI data
instead of downloading from quandl directly. This dramatically improves
the speed because we do not pay the rate limiting for quandl and we can
send the data in the format zipline expects.

NOTE: I need to rebuild the quandl set in s3. To test locally you can change the path to point to s3://.../test/ which is a smaller sample of data which works. I will run the load overnight tonight.

@@ -140,8 +140,8 @@ on OSX):
As you can see there are a couple of flags that specify where to find your
algorithm (``-f``) as well as parameters specifying which data to use,
defaulting to the :ref:`quandl-data-bundle`. There are also arguments for the
date range to run the algorithm over (``--start`` and ``--end``). Finally,
defaulting to the :ref:`quantopian-quandl-mirror`. There are also arguments for

This comment has been minimized.

@richafrank

richafrank May 4, 2016

Member

Is mirror part of the name somewhere?

This comment has been minimized.

@llllllllll

llllllllll May 4, 2016

Member

a ref link without an explicit link name will use the name of the next section. this renders as 'Quantopian Quandl WIKI Mirror': https://github.com/quantopian/zipline/pull/1178/files/265f77bf3e0ea53535fa6f025d44f5d05e0b2f48#diff-d2e46d2591b3fdb2539b3947867f60a5R106

This comment has been minimized.

@richafrank

richafrank May 4, 2016

Member

Aha, awesome.

@llllllllll llllllllll force-pushed the quantopian-quandl branch 2 times, most recently from 9ad3900 to fc04cd0 May 4, 2016

# use closing for py2 compat
with closing(urlopen(QUANTOPIAN_QUANDL_URL)) as f, \
tarfile.open('r', fileobj=BytesIO(f.read())) as tar:
tar.extractall(output_dir)

This comment has been minimized.

@richafrank

richafrank May 4, 2016

Member

Since this isn't instantaneous, would it be possible to show progress?

This comment has been minimized.

@llllllllll

llllllllll May 4, 2016

Member

idk how to get a progress bar around this operation.

This comment has been minimized.

@richafrank

richafrank May 4, 2016

Member

Ok, too bad.

where ``<bundle>`` is the name of the bundle to ingest.
where ``<bundle>`` is the name of the bundle to ingest, defaulting to
:ref:`quantipian-quandl <quantopian-quandl-mirror>`.

This comment has been minimized.

@ssanderson

ssanderson May 5, 2016

Member

quantipian

@ssanderson

This comment has been minimized.

Member

ssanderson commented May 5, 2016

The CLI crashed when I tried to use run from the command line with this:

$ python -m zipline ingest  # Wait 10 seconds or so.  Would be nice to have some progress indication here.
$ python -m zipline run -f zipline/examples/dual_moving_average.py -s 2014 -e 2015

Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/ssanderson/quantopian/qexec/zipline_repo/zipline/__main__.py", line 345, in <module>
    cli()
  File "/home/ssanderson/.virtualenvs/qexec/local/lib/python2.7/site-packages/click/core.py", line 664, in __call__
    return self.main(*args, **kwargs)
  File "/home/ssanderson/.virtualenvs/qexec/local/lib/python2.7/site-packages/click/core.py", line 644, in main
    rv = self.invoke(ctx)
  File "/home/ssanderson/.virtualenvs/qexec/local/lib/python2.7/site-packages/click/core.py", line 991, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ssanderson/.virtualenvs/qexec/local/lib/python2.7/site-packages/click/core.py", line 837, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ssanderson/.virtualenvs/qexec/local/lib/python2.7/site-packages/click/core.py", line 464, in invoke
    return callback(*args, **kwargs)
  File "/home/ssanderson/quantopian/qexec/zipline_repo/zipline/__main__.py", line 95, in _
    return f(*args, **kwargs)
  File "/home/ssanderson/quantopian/qexec/zipline_repo/zipline/__main__.py", line 238, in run
    environ=os.environ,
  File "zipline/utils/run_algo.py", line 115, in _run
    bundle_timestamp,
  File "zipline/data/bundles/core.py", line 406, in load
    asset_db_path(name, timestr, environ=environ),
  File "zipline/assets/assets.py", line 108, in __init__
    metadata = sa.MetaData(bind=engine)
  File "/home/ssanderson/.virtualenvs/qexec/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 3381, in __init__
    self.bind = bind
  File "<string>", line 1, in <lambda>
  File "/home/ssanderson/.virtualenvs/qexec/local/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 3477, in _bind_to
    self._bind = sqlalchemy.create_engine(bind)
  File "/home/ssanderson/.virtualenvs/qexec/local/lib/python2.7/site-packages/sqlalchemy/engine/__init__.py", line 386, in create_engine
    return strategy.create(*args, **kwargs)
  File "/home/ssanderson/.virtualenvs/qexec/local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 49, in create
    u = url.make_url(name_or_url)
  File "/home/ssanderson/.virtualenvs/qexec/local/lib/python2.7/site-packages/sqlalchemy/engine/url.py", line 186, in make_url
    return _parse_rfc1738_args(name_or_url)
  File "/home/ssanderson/.virtualenvs/qexec/local/lib/python2.7/site-packages/sqlalchemy/engine/url.py", line 235, in _parse_rfc1738_args
    "Could not parse rfc1738 URL from string '%s'" % name)
sqlalchemy.exc.ArgumentError: Could not parse rfc1738 URL from string '/home/ssanderson/.zipline/data/quantopian-quandl/1462468523226428000/assets-3.sqlite'
@llllllllll

This comment has been minimized.

Member

llllllllll commented May 5, 2016

@ssanderson ah, I can repro that in py2. weird that it would change, looking into that now

@coveralls

This comment has been minimized.

coveralls commented May 5, 2016

Coverage Status

Coverage increased (+0.3%) to 81.469% when pulling 2565d4f on quantopian-quandl into 8756bf2 on master.

@llllllllll llllllllll force-pushed the quantopian-quandl branch 2 times, most recently from 47b2baa to f7a522e May 5, 2016

llllllllll added some commits May 4, 2016

ENH: Adds quantopian-quandl bundle as new default.
This data bundle will use the quantopian mirror of the quandl WIKI data
instead of downloading from quandl directly. This dramatically improves
the speed because we do not pay the rate limiting for quandl and we can
send the data in the format zipline expects.
ENH: fix the quality of life issues in the CLI
Fixes the issues presented in #1181 by @ssanderson around the new
command line interface.
ENH: use more human readable format for bundle ingest directories
We are now using isoformats with ':' replaced with ';'. We cannot use a
normal isoformat because windows does not allow files or directories
with ':' in the name.
@coveralls

This comment has been minimized.

coveralls commented May 5, 2016

Coverage Status

Coverage increased (+0.02%) to 81.429% when pulling 120d60f on quantopian-quandl into 6997299 on master.

@llllllllll llllllllll merged commit 0562179 into master May 6, 2016

2 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@llllllllll llllllllll deleted the quantopian-quandl branch May 6, 2016

@llllllllll llllllllll referenced this pull request May 6, 2016

Closed

CLI QOL Improvements #1181

0 of 4 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment