risk rework #2081

llllllllll · 2018-01-17T06:41:03Z

posting my current work for feedback

prsutherland

I'm not through the whole thing, but sharing comments on parts that I've reviewed.

prsutherland · 2018-01-24T02:13:06Z

tests/metrics/test_metrics.py

+        )
+
+    @parameter_space(direction=['long', 'short'])
+    def test_future_single_position(self, direction):


Can this test and test_equity_single_position be parameterized/combined? If not a comment to describe the expected differences between the two would be helpful for understanding.

I would prefer not to mix these because the value/exposure calculations are different.

prsutherland · 2018-01-24T02:13:06Z

tests/metrics/test_metrics.py

+        ]
+        if direction == 'long':
+            all_zero_fields.extend((
+                'short_value',


the *_value are defined above

prsutherland · 2018-01-24T02:13:06Z

tests/test_algorithm.py

-                        for asset, shares in iteritems(OrderedDict(zip(
-                            context.assets, {share_counts}
-                        ))):
+                        it = zip(context.assets, {share_counts})


Looks like a lot of this test pre-dates this PR, but it is odd to see the assets hard coded and the share counts dynamic on a template that is only used once.

yeah, these tests are very weird. I am not going to change this now though.

prsutherland · 2018-01-24T02:13:06Z

tests/test_algorithm.py

@@ -2454,11 +2431,11 @@ def order_stuff(context, data):
            0.0,
            0.0,
            # 1000 shares * gain of 1
-            (100000.0 + 1000.0)/100000.0 - 1.0,
+            (100000.0 + 1000.0) / 100000.0 - 1.0,
            # 2000 shares * gain of 1, capital change of +5000


Comment should read, "... capital change of +50,000"

prsutherland · 2018-01-24T02:13:06Z

tests/test_tradesimulation.py

        )

+        algo.metrics_tracker = algo._create_metrics_tracker()
+        benchmark_source = algo._create_benchmark_source()
+        algo.metrics_tracker.handle_start_of_simulation(benchmark_source)


Does AlgorithmSimulator not call handle_start_of_simulation?

Ah, I see TradingAlgorithm calls it because it needs the benchmark source. Still, this a method with simulation in the name on object that is largely "owned" by the AlgorithmSimulator. Intuitively, this feels like it should be called by the AlgorithmSimulator.

There is nothing intuitive about the current TradingAlgorithm vs AlgorithmSimulator flow. I will look to see if we can have all of this info somewhere in the tradesim loop.

prsutherland · 2018-01-24T02:13:06Z

zipline/algorithm.py

@@ -1599,34 +1593,39 @@ def order_value(self,
    def recorded_vars(self):
        return copy(self._recorded_vars)

+    def _sync_last_sale_prices(self, dt=None):


This is sooooo much cleaner than the updated_portfolio stuff before!

prsutherland · 2018-01-24T02:13:06Z

zipline/finance/metrics/__init__.py

@@ -0,0 +1,162 @@
+#


There appears to be an empty zipline/finance/metrics.py in this pull request that github won't let me comment on. Looks like an accidental commit?

yeah, I will kill that

prsutherland · 2018-01-24T02:13:06Z

zipline/testing/core.py

@@ -970,10 +983,11 @@ def wrapped(*args, **kwargs):
                scope = tuple(scope)
                try:
                    f(*args + scope, **kwargs)
-                except Exception as e:
+                except Exception:
+                    info = sys.exc_info()


What does this change?

Right now, when you use parameter_space or subtest, you don't get the full traceback of each subtest failure. in the past you would set __fail_fast=True to not run subtests and get the traceback; however, that is really annoying. Now, if you have a subtest failure you will see the full traceback (indented) for each parameter set that failed.

prsutherland · 2018-01-24T02:13:06Z

zipline/sources/benchmark_source.py

+                                         asset,
+                                         trading_calendar,
+                                         trading_days,
+                                         data_portal):


This is probably in progress as you write more tests, but it looks like the docstring needs the new return type and if self.emission_rate == "minute" the return type is only a Series

prsutherland

At this point I've reviewed everything except zipline/finance/metrics/*

prsutherland · 2018-01-24T20:31:12Z

zipline/data/data_portal.py

+            elif field == "contract":
+                return None
+            elif field != "last_traded":
+                return np.NaN


Looks like this is relocated code, but do you know if "last_traded" should return anything here?

I assume this is not used anymore. ping @ehebert

It appears that last_traded is still expected to work. See: https://www.quantopian.com/help#overview-gettingdata

This is the case where the dt is outside of the assets lifetime, where np.NaN was chosen to represent that null value. (However pd.NaT may be more appropriate.)

prsutherland · 2018-01-24T20:31:12Z

zipline/finance/ledger.py

+        return stats
+
+
+if PY2:


Should this live in compat.py?

we discussed this in person, but this is pretty specialized. we can reconsider relocating this in the future

prsutherland · 2018-01-24T20:31:12Z

zipline/finance/ledger.py

+
+        Parameters
+        ----------
+        dividends: iterable of (asset, amount, pay_date) namedtuples


What do you think of renaming dividends to cash_dividends?

prsutherland · 2018-01-24T20:31:12Z

zipline/finance/ledger.py

+    def __init__(self, data_frequency):
+        # asset => position object
+        self.positions = OrderedDict()
+        self._unpaid_dividends = {}


I'm guessing the answer is performance/object creation overhead, but why do we not use defaultdicts for the unpaid dividends?

default dicts make it very easy to hide a programming error, setdefault is more explicit

prsutherland · 2018-01-24T20:31:12Z

zipline/finance/ledger.py

+                    div_owed,
+                ]
+
+    def pay_dividends(self, next_trading_day):


Should earn_dividends or pay_dividends update self._dirty_stats?

prsutherland · 2018-01-24T20:31:12Z

zipline/protocol.py

+    """
+
+    def __init__(self, portfolio):
+        dict_ = vars(self)


It is a little confusing that Portfolio uses a MutableView and Account uses a vars dict.

prsutherland

We largely discussed this in person yesterday, but writing it down so it doesn't get lost:

We should probably have at least some documentation about how to write/add metrics particularly around how to find what you need on the ledger. If this ends up looking like it will take a non-trivial amount of time, @abhijeetkalyan may have opinions on priority/timing.

Tests to include / scenarios we may want to worry about:

Slippage and commission tests (particularly that the new metrics tracker propagates the amount sign consistently with the perf tracker in shorts)

These tests may be exercised elsewhere, but the ledger might be impacted by:

Order fills over multiple bars
Orders being canceled or not filled by end of day
Splits/dividends applied correctly

prsutherland · 2018-01-25T21:07:33Z

zipline/finance/metrics/portfolio.py

+from zipline.protocol import Portfolio
+
+
+class PortfolioFieldMetric(object):


I think this is unused.

I thought I killed this, I replaced it with the ledgerfield stuff since you can get that from ledger.portfolio.*

prsutherland · 2018-01-25T21:07:33Z

zipline/finance/metrics/metric.py

+
+
+class AlphaBeta(object):
+    """End of simulation alpha and beta to the benchmark.


The docstring says "end of simulation" but there is no end_of_simulation method.

there may be a few others like this, I will clean these up.

coveralls · 2018-01-31T01:55:21Z

Coverage decreased (-0.5%) to 87.138% when pulling 5d45c7a on risk-rework into 783cacd on master.

prsutherland · 2018-02-02T19:54:44Z

docs/source/risk-and-perf-metrics.rst

@@ -0,0 +1,274 @@
+.. _metrics:


This is excellent! Very clear!

prsutherland · 2018-02-02T19:54:44Z

docs/source/risk-and-perf-metrics.rst

+
+``next_session_label`` is an :class:`int` which is the index of the next trading
+session to be run. This is provided to allow for efficient access to the daily
+returns through ``ledger.daily_returns_array[:next_session_ix]``.


Is the session index continuous so that you can access specific returns by subtracting N days?

yes, the reason it is next_session_ix instead of current_session_ix is to make it easy to do this slice though.

prsutherland · 2018-02-02T19:54:44Z

tests/test_algorithm.py

@@ -4491,7 +4471,7 @@ def transactions_for_date(date):
            first_auto_close_transaction,
            {
                'amount': -order_size,
-                'commission': 0.0,
+                'commission': None,


When should commissions be None vs 0.0?

orders have commission, transactions do not.

prsutherland · 2018-02-02T19:54:44Z

zipline/data/data_portal.py

@@ -198,7 +198,7 @@ def __init__(self,
                if reader is not None
            ]
            if last_minutes:
-                self._last_available_minute = min(last_minutes)
+                self._last_available_minute = max(last_minutes)


I would have guessed that this was originally min because that would be the last time we have data for both equity_minute_reader and future_minute_reader. Do we want to run backtests after one reader ends, but while the other still has data?

We already do! On prod right now, you cannot use history after the equities market closes on the current day. imo that should just fill NaN for equities or ffill the price field.

ssanderson · 2018-02-22T21:56:49Z

zipline/utils/sentinel.py

@@ -26,21 +26,43 @@ def sentinel(name, doc=None):
        if doc == value.__doc__:
            return value

+        if value._created_at is None:


Seems like it would be simpler to move this logic down to creation time?

prsutherland reviewed Jan 24, 2018

View changes

prsutherland reviewed Jan 25, 2018

View changes

llllllllll force-pushed the risk-rework branch 2 times, most recently from 122ead7 to 5214a71 Jan 30, 2018

llllllllll changed the title ~~(WIP) risk rework~~ risk rework Jan 30, 2018

llllllllll force-pushed the risk-rework branch 7 times, most recently from ada2746 to 1f955cf Jan 31, 2018

ssanderson mentioned this pull request Feb 2, 2018

Correct Roll Related Tests, then Fix Roll Functions quantopian/empyrical#81

Merged

prsutherland reviewed Feb 2, 2018

View changes

llllllllll added Waiting on Downstream Enhancement Refactoring API Design Performance Metrics labels Feb 2, 2018

llllllllll force-pushed the risk-rework branch 5 times, most recently from fb0e8bb to c8d215e Feb 13, 2018

ssanderson reviewed Feb 22, 2018

View changes

llllllllll force-pushed the risk-rework branch 3 times, most recently from 33b7b3c to b3890d3 Feb 22, 2018

llllllllll force-pushed the risk-rework branch from b3890d3 to 5503af4 Feb 23, 2018

llllllllll added 5 commits Jan 6, 2018

ENH: new risk metrics API

543a299

ENH: make sentinel track the created at location to debug conflicts

9a4f1b1

TST: fakedataportal support for minute mode

7bf39d9

DEV: update empyrical conda package

9e459c0

DOC: whatsnew

Loading status checks…

5d45c7a

llllllllll force-pushed the risk-rework branch from b600ebe to 5d45c7a Feb 23, 2018

llllllllll merged commit bd24e0d into master Feb 26, 2018
1 check passed

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

llllllllll deleted the risk-rework branch Feb 26, 2018

		from zipline.protocol import Portfolio


		class PortfolioFieldMetric(object):



		class AlphaBeta(object):
		"""End of simulation alpha and beta to the benchmark.

quantopian / zipline

risk rework #2081

risk rework #2081

Conversation

llllllllll commented Jan 17, 2018

prsutherland left a comment

This comment has been minimized.

prsutherland Jan 24, 2018 Contributor

This comment has been minimized.

llllllllll Jan 24, 2018 Author Contributor

This comment has been minimized.

prsutherland Jan 24, 2018 Contributor

This comment has been minimized.

prsutherland Jan 24, 2018 Contributor

This comment has been minimized.

llllllllll Jan 24, 2018 Author Contributor

This comment has been minimized.

prsutherland Jan 24, 2018 Contributor

This comment has been minimized.

prsutherland Jan 24, 2018 Contributor

This comment has been minimized.

prsutherland Jan 24, 2018 Contributor

This comment has been minimized.

llllllllll Jan 24, 2018 Author Contributor

This comment has been minimized.

prsutherland Jan 24, 2018 Contributor

This comment has been minimized.

prsutherland Jan 24, 2018 Contributor

This comment has been minimized.

llllllllll Jan 24, 2018 Author Contributor

This comment has been minimized.

prsutherland Jan 24, 2018 Contributor

This comment has been minimized.

llllllllll Jan 24, 2018 Author Contributor

This comment has been minimized.

prsutherland Jan 24, 2018 Contributor

This comment has been minimized.

llllllllll Jan 24, 2018 Author Contributor

prsutherland left a comment

This comment has been minimized.

prsutherland Jan 24, 2018 Contributor

This comment has been minimized.

llllllllll Jan 31, 2018 Author Contributor

This comment has been minimized.

ehebert Feb 3, 2018 Contributor

This comment has been minimized.

prsutherland Jan 24, 2018 Contributor

This comment has been minimized.

llllllllll Jan 30, 2018 Author Contributor

This comment has been minimized.

prsutherland Jan 24, 2018 Contributor

This comment has been minimized.

prsutherland Jan 24, 2018 Contributor

This comment has been minimized.

llllllllll Jan 30, 2018 Author Contributor

This comment has been minimized.

prsutherland Jan 24, 2018 Contributor

This comment has been minimized.

prsutherland Jan 24, 2018 Contributor

prsutherland left a comment

This comment has been minimized.

prsutherland Jan 25, 2018 Contributor

This comment has been minimized.

llllllllll Jan 25, 2018 Author Contributor

This comment has been minimized.

prsutherland Jan 25, 2018 Contributor

This comment has been minimized.

llllllllll Jan 25, 2018 Author Contributor

coveralls commented Jan 31, 2018 • edited

This comment has been minimized.

prsutherland Feb 2, 2018 Contributor

This comment has been minimized.

prsutherland Feb 2, 2018 Contributor

This comment has been minimized.

llllllllll Feb 2, 2018 Author Contributor

This comment has been minimized.

prsutherland Feb 2, 2018 Contributor

This comment has been minimized.

llllllllll Feb 2, 2018 Author Contributor

This comment has been minimized.

prsutherland Jan 24, 2018
Contributor

llllllllll Jan 24, 2018
Author Contributor

prsutherland Jan 24, 2018
Contributor

prsutherland Jan 24, 2018
Contributor

llllllllll Jan 24, 2018
Author Contributor

prsutherland Jan 24, 2018
Contributor

prsutherland Jan 24, 2018
Contributor

prsutherland Jan 24, 2018
Contributor

llllllllll Jan 24, 2018
Author Contributor

prsutherland Jan 24, 2018
Contributor

prsutherland Jan 24, 2018
Contributor

llllllllll Jan 24, 2018
Author Contributor

prsutherland Jan 24, 2018
Contributor

llllllllll Jan 24, 2018
Author Contributor

prsutherland Jan 24, 2018
Contributor

llllllllll Jan 24, 2018
Author Contributor

prsutherland Jan 24, 2018
Contributor

llllllllll Jan 31, 2018
Author Contributor

ehebert Feb 3, 2018
Contributor

prsutherland Jan 24, 2018
Contributor

llllllllll Jan 30, 2018
Author Contributor

prsutherland Jan 24, 2018
Contributor

prsutherland Jan 24, 2018
Contributor

llllllllll Jan 30, 2018
Author Contributor

prsutherland Jan 24, 2018
Contributor

prsutherland Jan 24, 2018
Contributor

prsutherland Jan 25, 2018
Contributor

llllllllll Jan 25, 2018
Author Contributor

prsutherland Jan 25, 2018
Contributor

llllllllll Jan 25, 2018
Author Contributor

coveralls commented Jan 31, 2018 •

edited

prsutherland Feb 2, 2018
Contributor

prsutherland Feb 2, 2018
Contributor

llllllllll Feb 2, 2018
Author Contributor

prsutherland Feb 2, 2018
Contributor

llllllllll Feb 2, 2018
Author Contributor

prsutherland Feb 2, 2018
Contributor

llllllllll Feb 2, 2018
Author Contributor

ssanderson Feb 22, 2018
Contributor