How does W and b randomly initialized?

[RyanCV]

Hi, in the file vl_nnlstm_params.m, line 40-41:

W = Param('value', -noise + 2 * noise * rand(4 * d, d + m, 'single')) ;
 b = Param('value', -noise + 2 * noise * rand(4 * d, 1, 'single'));

where does the dimension of W and b from? It seems W is in 4d x (d+m), and b is 4d x 1? Thanks.

[jotaf98]

Hi, yes that's correct. M is the input dimension and D the number of LSTM units.

As for why the W size is (4d)x(d+m), it's because an LSTM contains several gates, and W is partitioned into 4x2 blocks of weights where each block contains the parameters that control each gate.

The exact partition is described in the help for vl_nnlstm:

W = [Wxi   Whi
     Wxf   Whf
     Wxo   Who
     Wxc   Whc]

where i/f/o/c stand for input, forget gate, output gate, and cell respectively, while x and h are the input and hidden state, respectively.