ENH: Implement csvdir bundle #1860

bartosh · 2017-06-26T19:19:56Z

This bundle is for loading historical price data from the
set of .csv files. It supports daily and minute time frames.

coveralls · 2017-06-29T17:13:01Z

Coverage decreased (-0.4%) to 87.281% when pulling ca076c7 on bartosh:csvdir into 5e6110e on quantopian:master.

coveralls · 2017-06-30T07:16:24Z

Coverage decreased (-0.4%) to 87.281% when pulling 5c18613 on bartosh:csvdir into 5e6110e on quantopian:master.

coveralls · 2017-09-24T19:58:09Z

Coverage decreased (-0.04%) to 87.16% when pulling 2cbfa9b on bartosh:csvdir into 64af3d3 on quantopian:master.

coveralls · 2017-09-24T20:44:25Z

Coverage decreased (-0.04%) to 87.16% when pulling 17ff27c on bartosh:csvdir into 64af3d3 on quantopian:master.

coveralls · 2017-09-24T21:41:13Z

Coverage increased (+0.03%) to 87.232% when pulling 6255331 on bartosh:csvdir into 64af3d3 on quantopian:master.

coveralls · 2017-09-27T08:59:48Z

Coverage increased (+0.01%) to 87.215% when pulling 2340d1a on bartosh:csvdir into fbdfaf8 on quantopian:master.

coveralls · 2017-09-27T09:18:53Z

Coverage increased (+0.01%) to 87.215% when pulling 2340d1a on bartosh:csvdir into fbdfaf8 on quantopian:master.

coveralls · 2017-09-27T09:58:51Z

Coverage increased (+0.01%) to 87.215% when pulling 21337b7 on bartosh:csvdir into fbdfaf8 on quantopian:master.

bartosh · 2017-10-17T10:16:17Z

@freddiev4 FWIW, my plan is to review this sometime this week 🙂
Thank you. That would be great to have it reviewed!

freddiev4

Awesome work @bartosh! I have some feedback for you below 🙂

freddiev4 · 2017-10-05T14:11:11Z

zipline/data/bundles/csvdir.py

+
+    Parameters
+    ----------
+    tframe: tuple, optional


Small typo here. Looks like it should be tframes

freddiev4 · 2017-10-05T14:15:28Z

zipline/data/bundles/__init__.py

@@ -1,5 +1,7 @@
 # These imports are necessary to force module-scope register calls to happen.
 from . import quandl  # noqa
+from . import csvdir  # noqa
+
 from .core import (


Should we also add csvdir to __all__?

I don't think we should as quandl is not there.

freddiev4 · 2017-10-19T18:22:33Z

zipline/data/bundles/csvdir.py

+
+from . import core as bundles
+
+logger = logbook.Logger(__name__)


I think we can make the output from the logger a little cleaner by removing the timestamp and log level. We can do this with a CustomHandler

handler = StreamHandler(sys.stdout, format_string=" | {record.message}") logger = Logger(__name__) logger.handlers.append(handler)

Output before hander:

Loading custom pricing data: [#########---------------------------] 25%[2017-10-19 18:00:46.369496] INFO: zipline.data.bundles.csvdir: AAPL: sid 0 Loading custom pricing data: [##################------------------] 50%[2017-10-19 18:00:46.405959] INFO: zipline.data.bundles.csvdir: IBM: sid 1 Loading custom pricing data: [###########################---------] 75%[2017-10-19 18:00:46.447738] INFO: zipline.data.bundles.csvdir: KO: sid 2 Loading custom pricing data: [####################################] 100%[2017-10-19 18:00:46.481477] INFO: zipline.data.bundles.csvdir: MSFT: sid 3 Loading custom pricing data: [####################################] 100% Merging daily equity files: [####################################]

Output after handler

Loading custom pricing data: [#########---------------------------] 25% | AAPL: sid 0 Loading custom pricing data: [##################------------------] 50% | IBM: sid 1 Loading custom pricing data: [###########################---------] 75% | KO: sid 2 Loading custom pricing data: [####################################] 100% | MSFT: sid 3 Loading custom pricing data: [####################################] 100% Merging daily equity files: [####################################]

If we want to actually log this data that might require some other Handler and a default directory in which to put those logs (like ~/.zipline/logs/)

done. Thank you for the suggestion!

freddiev4 · 2017-10-19T18:44:52Z

zipline/data/bundles/csvdir.py

+    -------
+    ingest : callable
+        The bundle ingest function
+    Examples


Newline above Examples

freddiev4 · 2017-10-19T19:42:54Z

zipline/data/bundles/csvdir.py

+        # Hardcode the exchange to "CSVDIR" for all assets and (elsewhere)
+        # register "CSVDIR" to resolve to the NYSE calendar, because these
+        # are all equities and thus can use the NYSE calendar.
+        metadata['exchange'] = "CSVDIR"


I think we can leave this for another PR, as long as we document this somewhere in the code with a TODO

Users can pass in their own trading calendar using the --trading-calendar flag; if they want to make a bundle with their own csv data, the dates in that bundle might not map correctly to the dates in the NYSE calendar.

I think the solution to this is to change the calendar in zipline.data.bundles.core in our register() function

@curry def register(name, f, calendar_name='NYSE', start_session=None, end_session=None, minutes_per_day=390, create_writers=True):

And add an option to zipline ingest that works in the same way that we pass --trading-calendar to zipline run. #1800 shows how we do that.

I implemented csvdir_equities function especially for this purpose. It can be used in extension.py to use another calendar or other custom parameters, e.g.

register('csvdir-custom', csvdir_equities(['daily', 'minute'], "/Users/test/csvdir"), calendar_name='CUSTOM', minutes_per_day=24*60)

Ah good call. That slipped past me.

Code you share the code of your 'CUSTOM' calendar? :)

freddiev4 · 2017-10-19T19:54:56Z

zipline/data/bundles/csvdir.py

+       register('custom-csvdir-bundle',
+                csvdir_equities(["daily", "minute"],
+                '/full/path/to/the/csvdir/directory'))
+    """


Maybe not in this PR but I think we should add an example of how this works, with expected output and such; the best place for that would be http://www.zipline.io/bundles.html

Thank you for the suggestion. I'll create pr for that.

freddiev4 · 2017-10-19T20:01:00Z

zipline/data/bundles/csvdir.py

+    for tframe in tframes:
+        ddir = os.path.join(csvdir, tframe)
+
+        symbols = sorted(item.split('.csv')[0]


👍 for consistent symbol order

freddiev4 · 2017-10-21T13:45:45Z

tests/data/bundles/test_csvdir.py

+        def per_symbol(symbol):
+            df = pd.read_csv(
+                test_resource_path('csvdir_samples', 'csvdir',
+                                   'daily', symbol + '.csv'),


Can we gzip these files?

coveralls · 2017-10-23T17:27:00Z

Coverage increased (+0.02%) to 87.237% when pulling d0dcd2d on bartosh:csvdir into 0770d3b on quantopian:master.

coveralls · 2017-10-23T17:27:00Z

Coverage increased (+0.02%) to 87.237% when pulling d0dcd2d on bartosh:csvdir into 0770d3b on quantopian:master.

freddiev4 · 2017-10-27T15:07:37Z

@bartosh can you gzip the csv files?

bartosh · 2017-10-27T15:09:33Z

Sure, will do.

coveralls · 2017-11-05T12:48:47Z

Coverage increased (+0.006%) to 87.248% when pulling 1b51ace on bartosh:csvdir into 856313c on quantopian:master.

coveralls · 2017-11-05T13:48:03Z

Coverage increased (+0.006%) to 87.248% when pulling 688aa2b on bartosh:csvdir into 856313c on quantopian:master.

coveralls · 2017-11-05T13:48:03Z

Coverage decreased (-0.003%) to 87.239% when pulling 688aa2b on bartosh:csvdir into 856313c on quantopian:master.

bartosh · 2017-11-06T12:01:19Z

@freddiev4 Thank you for asking me to gzip test data. As a nice side effect of this the bundle now supports compressed csv files!

@freddiev4 AppVeyor failure is not caused by my changes I believe:

WARNING: could not import _license.show_info
# try:
# $ conda install -n root _license
conda install conda=4.1.11 conda-build=1.21.11 anaconda-client=1.5.1 --yes -q
Fetching package metadata: ..
Solving package specifications: 
Error: Could not find some dependencies for conda 4.1.11*: vc 9.*, vc 10.*, vc 14.*

Let me know when it's fixed in master. I'll rebase my branch.

freddiev4 · 2017-11-09T19:54:04Z

This should be good to merge 👍 don't worry about the Py34 build failure here as we removed Py34 package builds on master.

freddiev4 · 2017-11-09T19:54:44Z

However, could you actually rebase your branch @bartosh? As it looks like I can't merge it b/c of that failure in particular.

bartosh · 2017-11-09T19:56:26Z

Only Python 3.4 build failed. It still looks like infra error or something else not related to this PR. Can you rebuild it again?

bartosh · 2017-11-09T19:56:58Z

OK, I'll rebase it.

The bundle for loading historical price data from the set of .csv files. It supports daily and minute time frames.

bartosh · 2017-11-09T20:00:24Z

rebased

coveralls · 2017-11-09T20:47:49Z

Coverage increased (+0.006%) to 87.239% when pulling 30c2431 on bartosh:csvdir into 505c191 on quantopian:master.

bartosh · 2017-11-09T20:54:23Z

@freddiev4 thank you for merging it!

freddiev4 · 2017-11-09T20:57:56Z

Thanks for your awesome work! @bartosh

cemal95 · 2020-06-01T14:19:25Z

Does anyone know how to solve this problem. Its data from Quandl, EoD data I purchased. But I get this:

bartosh force-pushed the csvdir branch from bfad4c2 to ca076c7 Compare June 29, 2017 16:38

bartosh force-pushed the csvdir branch from ca076c7 to 5c18613 Compare June 30, 2017 06:39

This was referenced Jul 7, 2017

Solution for someone who relied solely on load_from_yahoo for price data for backtesting? #1876

Closed

Write example script that shows use of custom data. #174

Closed

Any examples with csv data #713

Closed

bartosh force-pushed the csvdir branch from 5c18613 to 6ef0ca0 Compare July 21, 2017 21:27

freddiev4 mentioned this pull request Aug 3, 2017

Methods of accessing Data in zipline (For Beginners) #1903

Closed

pvillenave approved these changes Aug 31, 2017

View reviewed changes

bartosh force-pushed the csvdir branch 2 times, most recently from 71a93a9 to 2cbfa9b Compare September 24, 2017 19:34

bartosh force-pushed the csvdir branch from 2cbfa9b to 17ff27c Compare September 24, 2017 20:11

bartosh force-pushed the csvdir branch from 17ff27c to 6255331 Compare September 24, 2017 21:07

bartosh changed the title ~~implement csvdir bundle~~ ENH: implement csvdir bundle Sep 25, 2017

bartosh changed the title ~~ENH: implement csvdir bundle~~ ENH: Implement csvdir bundle Sep 25, 2017

bartosh force-pushed the csvdir branch 2 times, most recently from da7c3cb to 2340d1a Compare September 27, 2017 08:33

bartosh force-pushed the csvdir branch from 2340d1a to 21337b7 Compare September 27, 2017 09:27

tibkiss mentioned this pull request Oct 6, 2017

Continuous ingestion of realtime data from live brokers zipline-live/zipline#26

Open

bartosh force-pushed the csvdir branch 4 times, most recently from 83cbf8c to 5f7b851 Compare October 14, 2017 21:14

freddiev4 reviewed Oct 21, 2017

View reviewed changes

bartosh force-pushed the csvdir branch from 5f7b851 to d0dcd2d Compare October 23, 2017 16:49

freddiev4 mentioned this pull request Nov 3, 2017

Some basic question about relationship between zipline and quantopian #2006

Closed

bartosh force-pushed the csvdir branch 3 times, most recently from 445c683 to 1b51ace Compare November 5, 2017 12:19

bartosh force-pushed the csvdir branch from 1b51ace to 688aa2b Compare November 5, 2017 13:12

bartosh added 2 commits November 9, 2017 21:58

ENH: implement csvdir bundle

52820cd

The bundle for loading historical price data from the set of .csv files. It supports daily and minute time frames.

TST: add test case for csvdir bundle

30c2431

bartosh force-pushed the csvdir branch from 688aa2b to 30c2431 Compare November 9, 2017 19:59

freddiev4 merged commit 67d983b into quantopian:master Nov 9, 2017

vonpupp mentioned this pull request Nov 14, 2017

Ensure that external price data bundles are supported scrtlabs/catalyst#65

Closed


		from . import core as bundles

		logger = logbook.Logger(__name__)

ENH: Implement csvdir bundle #1860

ENH: Implement csvdir bundle #1860

Conversation

bartosh commented Jun 26, 2017 • edited Loading

coveralls commented Jun 29, 2017 • edited Loading

coveralls commented Jun 30, 2017 • edited Loading

coveralls commented Sep 24, 2017

coveralls commented Sep 24, 2017

coveralls commented Sep 24, 2017 • edited Loading

coveralls commented Sep 27, 2017 • edited Loading

coveralls commented Sep 27, 2017 • edited Loading

coveralls commented Sep 27, 2017 • edited Loading

bartosh commented Oct 17, 2017

freddiev4 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freddiev4 Oct 19, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Oct 23, 2017

coveralls commented Oct 23, 2017 • edited Loading

freddiev4 commented Oct 27, 2017 • edited Loading

bartosh commented Oct 27, 2017

coveralls commented Nov 5, 2017 • edited Loading

coveralls commented Nov 5, 2017

coveralls commented Nov 5, 2017 • edited Loading

bartosh commented Nov 6, 2017

freddiev4 commented Nov 9, 2017 • edited Loading

freddiev4 commented Nov 9, 2017

bartosh commented Nov 9, 2017

bartosh commented Nov 9, 2017

bartosh commented Nov 9, 2017

coveralls commented Nov 9, 2017 • edited Loading

bartosh commented Nov 9, 2017

freddiev4 commented Nov 9, 2017

cemal95 commented Jun 1, 2020

bartosh commented Jun 26, 2017 •

edited

Loading

coveralls commented Jun 29, 2017 •

edited

Loading

coveralls commented Jun 30, 2017 •

edited

Loading

coveralls commented Sep 24, 2017 •

edited

Loading

coveralls commented Sep 27, 2017 •

edited

Loading

coveralls commented Sep 27, 2017 •

edited

Loading

coveralls commented Sep 27, 2017 •

edited

Loading

freddiev4 Oct 19, 2017 •

edited

Loading

coveralls commented Oct 23, 2017 •

edited

Loading

freddiev4 commented Oct 27, 2017 •

edited

Loading

coveralls commented Nov 5, 2017 •

edited

Loading

coveralls commented Nov 5, 2017 •

edited

Loading

freddiev4 commented Nov 9, 2017 •

edited

Loading

coveralls commented Nov 9, 2017 •

edited

Loading