Question about RNN search spaces

[ucmnd-pp]

Hi, I was playing around a little with your framework and wanted to create a search space for a RNN but i failed to get a working RNN out-of-box: I could create a cell and plug that cell manually into an vanilla RNN that is encapsulating the cell and iterating over the temporal dimension.

Now I'm wondering if there is any way to get a complete RNN (not just the cell) as the result of an architecture search with your framework.

I tried various approaches like expressing the outer temporal loop as a substitution module and tried to instantiate the cell within the compile function. That wasn't working (probably because not all variables of the cell's search space have been set before the call to compile_fn()).

Another idea (that is quickly rejected) was to connect the cell's output with the cell's input, but that would produce a cyclic graph...

To me it seems that with the current framework it is not possible to create true recursive search spaces. The kind of recursion you show in your paper is no true recursive search space, the result is a linear/sequential one (referring to fig. 18 in your paper) .

Is what I'm trying to do possible?
Could you help me to make this work somehow (in an elegant, non-hacky way)?
And how would you define e.g. a multi-layer RNN with individual, independent cell search spaces for each layer, so your don't reuse the same cell on every layer?

FYI: I have also posted this on the google groups board.

[negrinho]

Hi Pratik,

If you want to consume multiple time steps instead of a single one, the best way is to do this directly, i.e., to consume the whole tensor (batch_size x input_dim x num_steps, or whatever ordering of the dimensions you are using) and output whole processed tensors your computation performs. I would probably take that route. In the future, I want to define modules that can have other search spaces inside, which would help in this use-case, but right now, there is a flat nesting hierarchy in the search space, i.e., eventually the search space bottoms out to basic modules and then you run that. I've thought about the nested case but haven't implemented yet.

The way to do this would be either to search over the cell, compile it to Pytorch code, and then use that, or search directly over something that consumes all time steps and does the computation you want. After the implementation of the recursive case, wrapping search spaces will be possible. I've thought about it, but haven't got around implementing it. Additionally, while it might not be super relevant here, the most recent branch is this one. Haven't updated the documentation yet though.

For the multi-layer example that you mention, I would do the same thing: stack the RNN that consume the whole input that contains all time steps and outputs the results of processing all time steps. Inside the module, you can enforce the locality of the computation by just looking at the relevant positions.

I'll check the code on Google Groups.

All the best,
Renato