Skip to content

Commit

Permalink
DEV: Overhaul core history logic.
Browse files Browse the repository at this point in the history
Overhaul the core HistoryContainer logic to be more robust to changing
universes.

Major Changes
-------------
* Remove `return_frame` cache.  The original purpose of using
  return_frames was to avoid having to create new DataFrames on each
  iteration of handle_data, but we ended up having to copy the return
  frames anyway because user code could mutate the frames in place.
  Removing the return_frames reduces unnecessary copying, and reduces
  the logic of `get_history` to just forward-filling and concatenating
  two DataFrames.

* Use a `MultiIndex`ed DataFrame to represent
  `last_known_prior_values`.  This makes lookups faster and greatly
  simplifies the logic of adding and dropping sids.

* HistoryContainer no longer attempts to determine its universe based on
  the contents of its internal buffers.  The TradingAlgorithm
  controlling the container is now responsible for explicitly calling
  `add_sids` or `drop_sids` when securities enter or leave the
  algorithm's universe.  These methods, along with the internal
  `_realign` method, provide a clean interface for changing the universe
  of securities managed by the container.

* Refactor index mutation logic in `RollingPanel` into a
  `MutableIndexRollingPanel` subclass.  Maintenance of the old behavior
  is regrettably necessary to support `BatchTransform`.

* Refactor shared logic from `roll` and `get_history` into a single
  `aggregate_ohlcv_panel` method that's responsible for collapsing an
  OHLCV buffer into a frame.
  • Loading branch information
ssanderson committed Sep 29, 2014
1 parent b3c7c26 commit 235954d
Show file tree
Hide file tree
Showing 4 changed files with 353 additions and 282 deletions.
8 changes: 4 additions & 4 deletions tests/test_rolling_panel.py
Expand Up @@ -22,16 +22,16 @@
import pandas as pd
import pandas.util.testing as tm

from zipline.utils.data import RollingPanel
from zipline.utils.data import MutableIndexRollingPanel


class TestRollingPanel(unittest.TestCase):
class TestMutableIndexRollingPanel(unittest.TestCase):

def test_basics(self, window=10):
items = ['bar', 'baz', 'foo']
minor = ['A', 'B', 'C', 'D']

rp = RollingPanel(window, items, minor, cap_multiple=2)
rp = MutableIndexRollingPanel(window, items, minor, cap_multiple=2)

dates = pd.date_range('2000-01-01', periods=30, tz='utc')

Expand Down Expand Up @@ -69,7 +69,7 @@ def test_adding_and_dropping_items(self, n_items=5, n_minor=10, window=10,
add_items = np.arange(first_non_existant, first_non_existant + periods)
np.random.shuffle(add_items)

rp = RollingPanel(window, items, minor, cap_multiple=2)
rp = MutableIndexRollingPanel(window, items, minor, cap_multiple=2)

dates = pd.date_range('2000-01-01', periods=periods, tz='utc')

Expand Down

1 comment on commit 235954d

@ehebert
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great to me. /signoff

Please sign in to comment.