Enas_lm child model

[HaleyQH]

In the child.py file, why w_skip[0] is not used for L2 weight reg ?
var_s = [w_prev] + w_skip[1:]

google-research/enas_lm/src/child.py

Line 69 in 4305853

# extract the relevant variables, so that you only do L2-reg on them.

Does w_skip[0] means the weight of node 1 and node 2 (as the red arrow shows )?

[ZhiqingXiao]

I also noticed bugs in calculating the L2 regularization.

In child.py:
https://github.com/google-research/google-research/blob/master/enas_lm/src/child.py#L318

self.l2_reg_loss = tf.add_n([tf.nn.l2_loss(w ** 2) for w in var_s])

The regularization is implemented as "l4" regularization.

However in fixed_lib is correct:
https://github.com/google-research/google-research/blob/master/enas_lm/src/tpu/fixed_lib.py#L409

@phthhieu
@melodyguan