10_may_2017

Learning to learn by gradient descent by gradient descent

Questions:

[@priba] No acabo d'entendre com s'optimitza el 'optimizee'. Diu que que fa gradient descend sobre phi, pero no veig perque calcula el loss d'aquesta forma.

[@gcucurull] In section 2.1, they say that the optimizer operates coordinatewise on the parameters, so that an LSTM is shared (same weights but different hidden state) for all the parameters. What do they mean with "Different behavior on each coordinate is achieved by using separate activations for each objective function parameter"?

[@gitabcworld] This is a more general question. There is to main problems scalability and time complexity. I have read in a forum this paragraph: "It takes 4000 steps for even our toy problem to converge, we had to train a network completely just for one step of optimization for the meta-learner. We would have to optimize a large problem many more times and would take a very long time. Also unrolling the entire training loop in the graph is not feasible for larger problems, although in the paper they only unroll the BPTT to 20 steps. There is also evidence in the paper that the RNN optimizer can generalize from smaller problem to larger problems". Question 1: The time to train the network completely it is refering to the LSTM? How much time could be if you want to learn and LSTM optimizer for MNIST gradients? or for ImageNet gradients? Question 2: I am not an expert of LSTM, why increasing the BPTT it is not feasible?

Learned Optimizers that Scale and Generalize

Questions:

[@priba]