WIP: (DO NOT MERGE) New ExchangeCalendar API

[ssanderson]

WIP for ExchangeCalendar redesign. Still in the process of fixing up the tests and porting over the non NYSE calendars, but I think something like this API is what we want.

The essential thought is that, rather than passing an entire module (which among other things, causes us to do substantial computation at import time, slows down our tests, and makes simulation startup noticeably slower), you now pass a subclass of ExchangeCalendar to TradingEnvironment. The environment then instantiates the calendar, which provides three public attributes:

• day (an instance of pandas.CustomBusinessDay)
• schedule (DataFrame with the same structure as the old open_and_closes)
• native_timezone (a timezone)

I haven't yet re-implemented any of the date logic that's done manually in TradingEnvironment, but much of it could be thrown out in favor of using the new CustomBusinessDay. For example, next_trading_day(dt) can be written as just (dt + calendar.day), and it's also possible to compute ranges of trading days by passing calendar.day as freq to pd.date_range.

Implementing a new ExchangeCalendar can be done by subclassing and implementing the following abstract properties (in practice I expect most of them to be implemented as class-level attributes: note that this doesn't have the same issue of doing work at module scope as the old implementation, because instantiating an AbstractHolidayCalendar doesn't do any work until you call calendar.dates to actually extract its dates):

• native_timezone: A timezone in which to interpret open_time and close_time.
• open_time: A datetime.time representing the first tradable minute of the exchange on a normal day).
• close_time: A datetime.time representing the final tradable minute of the exchange on a normal day.
• holidays_calendar: A pandas AbstractHolidayCalendar instance defining days on which the exchange is not open.
• special_opens_calendars: A (possibly empty) iterable of pairs of (datetime.time, AbstractHolidayCalendar). Each calendar is used to define days on which the exchange regularly opens at a time other than the standard open. Multiple calendars are allowed to account for the possibility that the exchange opens at different nonstandard times under different circumstances.
• special_closes_calendars: Same structure as special_opens_calendars, but defines nonstandard closing times. For the main NYSE calendar, this is used to define the various regular early closes.
• holidays_adhoc: An iterable of dates defining days on which the exchange was closed due to historical accident (e.g. the hurricane sandy closings for NYSE, or the world cup closings for BMF).
• special_opens_adhoc: A (possibly empty) iterable of pairs of (datetime.time, list of Timestamp). Each entry defines a list of additional dates on which the exchange opened at a nonstandard time due to historical accident.
• special_closes_adhoc: Same structure as special_opens_adhoc, but defines anomalous closes.

I've already re-implemented the NYSE calendar in terms of this interface, and I think it's significantly cleaner than the old version. It's particularly nice to be able to use pandas' built-in holiday functionality to write something like:

Christmas = Holiday(
    'Christmas',
    month=12,
    day=25,
    observance=nearest_workday,
)

and not have to do all the special-casing logic for what happens when Christmas falls on a weekend. Implementing Black Friday (the day after the Thanksgiving, i.e. the 4th thursday in November) as

USBlackFriday = Holiday(
    'Black Friday',
    month=11,
    day=1,
    offset=[DateOffset(weekday=TH(4)), Day(1)],
)

is also quite nice.

@warren-oneill, @jfkirk, @ehebert, @twiecki I'd like to get your guys' feedback on the API presented here, both for creating new calendars and for consuming them in TradingEnvironment while I'm wrangling the remaining tests.

[ehebert]

ACK'd will read through by end of day.

[warren-oneill]

Sounds great but imo there lacks a connection between the calendar and other marketplace specific variables such as the benchmark, commission and asset metadata. I think we need another level of abstraction, maybe an Exchange class, that links all these together. I know this is supposed to be solved by the TradingEnviroment but does it makes sense to use, for example, the LSE calendar with a NYSE benchmark? When the LSE ExchangeCalendar is chosen shouldn't an LSE specific benchmark and commission automatically be chosen too?
A better solution would be a class of type Exchange such as LseExchange which initializes everything and that way the user can see all market relevant info in one class rather than have to search all over the code.
The way it currently works the user must make sure they initialize all of these aspects before running the algo which I think is too much to ask and also since the benchmark is currently yahoo dependent I had to write my own benchmark loader which gives a constant return rate just to get the algo running. For example:

trading.environment = TradingEnvironment(bm_symbol='^EEX',
                                         exchange_tz="Europe/Berlin",
                                         env_trading_calendar=calendar,
                                         load=load_market_data_eex)

algo = TradingAlgorithm(initialize=initialize,
                            handle_data=handle_data,
                            commission=PerShare(0.0075))

The benchmark loader is a massive barrier (as big as the calendar problems) for anyone like myself looking to develop Zipline for other marketplaces. Is is possible to make this more user-friendly?

It would also be great to see a connect between @jfkirk's current work on the Asset class incorporated, for example linking product metadata to an Exchange.

[warren-oneill]

On another note, it would be great to be able to have some flexibility with schedule. For example we plan to implement auction-type markets and need to record the auction time in some way.

[jfkirk]

Warren's feedback about wrapping this all together in exchange-level information makes sense to me - it will be valuable as more Assets in more exchanges are added to our system.

@warren-oneill The Asset object currently has a field for its exchange, but we don't have it set anywhere. Are you looking to perform one simulation with Assets from many different exchanges and, therefore, many different trading hours? This is on my roadmap, but was not my immediate plan.

[warren-oneill]

@jfkirk far down the line we would like to trade the same asset on different exchanges but what I'd like in the short term is to just have to define the exchange and then have the assets automatically linked/initialized but presumably our list of assets is a lot shorter than your list of assets.

[ehebert]

TIL about the @ symbol in the query syntax. Nice.

[ehebert]

@ssanderson 👍 on using the Holiday syntax, as well as 👍 on instantiating with-in the trading environment. So looks good to move forward with.
Though if it happens that one or the other needs to be paused, I think they orthogonal?

@warren-oneill @jfkirk I could see where there would be an environment with that tighter exchange binding, and perhaps multiple exchanges and calendars. This design doesn't solve that problem now, but I think is a good step along the way.

[ssanderson]

A better solution would be a class of type Exchange such as LseExchange which initializes everything and that way the user can see all market relevant info in one class rather than have to search all over the code.

I'm on board with this. I suppose the question is whether it's worth having the top-level exchange class that's separate from the calendar class, or whether we should add "default_exchange" or somesuch to ExchangeCalendar.

A sketch of what an Exchange class would look like would be something like:

class NYSE(Exchange):
    symbol = "NYSE"
    name = "New York Stock Exchange"
    calendar_class = NYSEExchangeCalendar
    default_benchmark = "^GSPC"

@warren-oneill @jfkirk what other kinds of metadata would you want to be associated with an exchange?

[warren-oneill]

@ssanderson I would def prefer to have an exchange class in the vain that you illustrated above. Personally I would also like to see a default commission and an AssetMetaData class as part of the Exchange class.

[warren-oneill]

hey @ssanderson is this on ice?

[ssanderson]

hey @ssanderson is this on ice?

@warren-oneill been superceded as an immediate need by #584, though I'm trying to allocate ~20% time to Zipline community needs. Are there features here that would be particularly important to the work you're doing with Zipline?

[warren-oneill]

@ssanderson it's not a barrier anymore now that I have everything setup (I made my own exchange class locally) but for would be a big help for new users to have everything in one place. What's still unclear to me what is the correct order of initialization of env, asset_finder and source but perhaps the order doesn't matter. @jfkirk?

[warren-oneill]

FYI here's a snippet of my exchange base class

class Exchange(object, metaclass=ABCMeta):
    def __init__(self):
        self.bm_symbol = self.insert_bm()
        self.exchange_tz = "Europe/Berlin"
        self.calendar = self.insert_calendar()
        self.load = load_market_data
        self.commission = PerShare(self.insert_commission())
        self.env = self.insert_env()
        self.asset_finder = self.insert_asset_finder()
        self.source = self.insert_source()

    def insert_env(self):
        return TradingEnvironment(bm_symbol=self.bm_symbol,
                                  exchange_tz=self.exchange_tz,
                                  env_trading_calendar=self.calendar,
                                  load=self.load)

any functions not defined are abstractmethods and are defined in the subclasses and load_market_data is a function I wrote to create a constant benchmark based on the trading calendar and also uses ffill to fill days that are missing from the treasury curves.

[ssanderson]

@warren-oneill thanks for the snippet. I'm hoping to get back to this some time soon.

[ssanderson]

Closing this in deference to @jfkirk's current work on bringing this in for real.

[warren-oneill]

@ssanderson @jfkirk is there a branch or PR?

[jfkirk]

@warren-oneill #1138