Long term plan for adding operators

[ezyang]

Right now, we have been adding operators to the AST one-by-one to support various operations we may be interested in. There are problems with doing this long term: every new operation you add, is another case that must be handled in any IR_IF case statement. IR_IF works well if you don't have too many forms in your language, not if you have a separate subkind for every single one of PyTorch's operations!

The object-oriented way to solve this problem is to keep subclasses, stop writing IR_IFs and move methods into the class definition. In GHC, we solved this problem differently, by maintaining an environment of "known" ops and recording with them information necessary to do analyses and optimizations with them (e.g., their types, strictness, unfoldings, etc.) I think this latter approach scales better with lots of ops (and is similar to how NNVM is structuring their operators: https://github.com/dmlc/nnvm/blob/master/docs/overview.md) but we should make a decision one way or another before we start adding tons and tons of operator definitions to fill out our PyTorch support.

Unlike NNVM, one thing I do NOT advocate is making this information public (at least for now.) So, no user-written ops that support fusion. That's a very tricky API to get right and there should be some concerted design that happens before we let people rely on it.

[ezyang]

One thing to note: until we start porting operators into direct C++, translation of an operator like Sigmoid into a a plain IR node is a pessimization, because the only way we can actually run it is by reconstruction the appropriate Python function and then running it.

[ezyang]

@zdevito, @killeent, and I have decided that we will solve this problem by introducing a new Primitive node type, which corresponds directly to ToffeeIR nodes (semantics defined by ToffeeIR). User-defined autograd Functions which correspond to a ToffeeIR node will define how they convert into ToffeeIR, and the optimizer will work directly with ToffeeIR nodes.

See also https://fb.quip.com/bg3MAJe0MStd