[C++ API] Using new registration mechanism

[goldsborough]

To solve the problem of references to parameters/submodules in the base module getting invalidated on clone() and reassignment of variables in submodules, this PR implements a change discussed by @ezyang, @ebetica and I, which, instead of storing references, stores functions that can be passed the this pointer to access the members. You can think of this as a fancy way of storing the offsets into the subclass.

Posting as MVP to discuss. It's a little batshit crazy, but only a little.

Review order:

1. torch/detail/member_ref.h
2. torch/nn/module.h
3. torch/nn/modules/linear.cpp and other modules
4. Other stuff

Pros:

• Solves reference invalidation
• Allows use of constructors instead of reset() function (not implemented yet!)

Cons:

• Complicated code
• Unsafe, I accidentally passed the wrong object to access the member and fortunately (!) it segfaulted. This could be improved by using our std::any shim instead of void* in the implementation of MemberRef
• Registration is a little verbose: register_parameter("weight", &MyModule::weight, at::ones(at::CPU(at::kFloat), {3, 3})

@ezyang @ebetica @apaszke

[apaszke]

This syntax is quite verbose. What if we did it more like

reqister_module(l3, Linear(3, 5).build());

also, a helper for building modules might end up being useful as well

build_module(l3, Linear(3, 5));

---

Ideally we would do a Catch-style thing like

add(l3 = Linear(3, 5));

This is almost good, because the expression actually evaluates to an lvalue, so we can get access to the module too (and e.g. detect if it's been constructed, and call build() if it hasn't). This would still work if we changed MemberRef to use pointer offsets. I think we could just store lvalue_addr - this in that case.

The only problem with it, is that we don't know how to assign a name to it. This should be still solvable, although I don't have a pretty solution in mind just yet. A dirty one would be to turn the expression into the string and slice out the part that stands before the equal sign at run time.

Note that this macro could evaluate to a different function based on the lvalue type, so you could also have

add(weight = torch::CPU(torch::kFloat).ones({num_features}));

WDYT?

[goldsborough]

Where would you get the parameter name from in all your suggestions? We could just name them with progressive numbers, but it might be nice to have names for parameters, at least in our core modules

[ebetica]

And progressive numbers is not very like pytorch, and is not super backwards compatible: you always have to add new parameters to the end of your list. I'm generally happy with a macro for ease of naming for users, like

#define REGISTER(x, expr) register(#x, &(decltype(*this):: ## x), expr);

usage:
REGISTER(weight, at::ones(at::CPU(at::kFloat), {3, 3})));

though it does add a little bit of magic, and you can't have unbound parameters. However, I think this is the same case in pytorch.

[apaszke]

Yep, I've been thinking of doing #x to turn the first argument into a string at compile time. The

REGISTER(weight = at::ones(...))

syntax is nice, although that would be more hacky because it would create a compile-time string of weight = at::ones(...), and we'd need to slice it at the first equal sign at run-time. It doesn't seem to bad either.

[goldsborough]

I will add a nice macro to make registration less verbose in another PR. I want to merge this now to make progress on my other in-flight PRs.