A question on two_layer_net

[huangzl19]

Hello! First great thanks to your brilliant answers of these assignments. It really helps me a lot!

I would like to ask a question in neural_net.py. When db1 is calculated, I noticed that 2 * reg * b1 hasn't been added. db2 has the the same problem. The original code is as below.

# Backprop dRelu1 to calculate db1. db1 = dRelu1 * 1 grads['b1'] = np.sum(db1, axis=0)

However, according to the forward process, when calculating the loss we have added reg * np.sum(b1 * b1). The original code is as below.

loss += reg * (np.sum(W1*W1) + np.sum(W2*W2) + np.sum(b1*b1) + np.sum(b2*b2))

Is this part missing during back propogation? I add the 2 * reg * b1 to my code, and the analytic gradients is still less than 1e-8 for b1, and b2.

[Burton2000]

Good spot thank you!

Looking at the notes again I believe we should not typically include the bias term when we do L2 regularization (From what is also said in the notes it probably has a negligible effect if we do or do not regularize biases).

But you are right that if we do include them in regularization we should also include the regularization terms in the backprop.
I will update the code to not include the bias terms in the L2 regularization so we dont need to backprop it.

[Burton2000]

Done

[xzh-scut]

where is my comment? it disappera???