The question in tutorial optimizers_conditionalgradient frobenius_norm graph and pytorch version of me

[johnnylin110]

I am looking for the tutorial https://github.com/tensorflow/addons/blob/master/docs/tutorials/optimizers_conditionalgradient.ipynb
And the graph about frobenius norm calculate in this way

def frobenius_norm(m):
    """This function is to calculate the frobenius norm of the matrix of all
    layer's weight.
    Args:
        m: is a list of weights param for each layers.
    """
    total_reduce_sum = 0
    for i in range(len(m)):
        total_reduce_sum = total_reduce_sum + tf.math.reduce_sum(m[i]**2)
    norm = total_reduce_sum**0.5
    return norm

and which is obtain by

CG_get_weight_norm = tf.keras.callbacks.LambdaCallback(
    on_epoch_end=lambda batch, logs: CG_frobenius_norm_of_weight.append(
        frobenius_norm(model.trainable_weights).numpy()))

And I want to ask is that
1.when in the network update , we calculate the frobenius norm by gt(gradient value) , is the "model.trainable_weights" equal to gradient value?
2. The update formula is
by the paper
And this means that each layer update by this rule, and why the definition above calculate frobenius norm by add all layer together not sperately? (which means each epoch has 6 frobenius norm , beacuse there have 6 parameter according to network architecture)

[johnnylin110]

I want to ask this because I implement Frobenius norm constranit in pytorch which has the different calculate way as this and this is my implement way

[seanpmorgan]

++ @pkan2 codeowner.

@lokhande-vishnu

[pkan2]

The "model.trainable_weights" equals to the parameters value rather than gradient. And the Frobenius norm function you pointed out is used to present the frobenius norm of the parameter after each epoch's training. It has nothing to do with training process but to present the regularization effect of this epoch's training.

[johnnylin110]

Got it , And i want to ask that my frobenius norm of gradient in each layer is always between 0 to 1
is this a common value?
@pkan2

[Squadrick]

Closing this as the issue is resolved. You can continue the discussion here.