Ploting Gradients to Tensorboard and Console #31542
Comments
oanush
added
2.0.0-beta0
comp:tensorboard
|
Issue replicating for TF version 2.0beta,please find the Gist of Colab. Thanks! |
gowthamkpr
added
the
stat:awaiting tensorflower
jvishnuvardhan
added
TF 2.0
|
I also find it challenging to plot gradients to Tensorboard in TF 2.2.0-rc3 on Colab. My case is different from @SPP3000 in that instead of if self.console:
print('Tensor: {}'.format(t.name))
print('{}\n'.format(K.get_value(t)[:10]))
else:
tf.summary.histogram(t.name, data=t)I simply have AttributeError: 'Sequential' object has no attribute 'total_loss'Using |
|
Im also affected by the issue |
|
Here is a workaround where the gradients are explicitly calculated. It avoids the class ExtendedTensorBoard(tf.keras.callbacks.TensorBoard):
def _log_gradients(self, epoch):
step = tf.cast(tf.math.floor((epoch+1)*num_instance/batch_size), dtype=tf.int64)
writer = self._get_writer(self._train_run_name)
with writer.as_default(), tf.GradientTape() as g:
# here we use test data to calculate the gradients
_x_batch = x_te[:100]
_y_batch = y_te[:100]
g.watch(_x_batch)
_y_pred = self.model(_x_batch) # forward-propagation
loss = self.model.loss(y_true=_y_batch, y_pred=_y_pred) # calculate loss
gradients = g.gradient(loss, self.model.trainable_weights) # back-propagation
# In eager mode, grads does not have name, so we get names from model.trainable_weights
for weights, grads in zip(self.model.trainable_weights, gradients):
tf.summary.histogram(
weights.name.replace(':', '_')+'_grads', data=grads, step=step)
writer.flush()
def on_epoch_end(self, epoch, logs=None):
# This function overwrites the on_epoch_end in tf.keras.callbacks.TensorBoard
# but we do need to run the original on_epoch_end, so here we use the super function.
super(ExtendedTensorBoard, self).on_epoch_end(epoch, logs=logs)
if self.histogram_freq and epoch % self.histogram_freq == 0:
self._log_gradients(epoch)
|
|
Thanks for posting the code. I am getting and error saying: |
you can try this code import tensorflow as tf
from tensorflow.python.keras import backend as K
mnist = tf.keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
model = tf.keras.models.Sequential([
tf.keras.layers.Flatten(input_shape=(28, 28)),
tf.keras.layers.Dense(128, activation='relu', name='l_1st'),
tf.keras.layers.Dense(128, activation='relu', name='l_2nd'),
tf.keras.layers.Dense(128, activation='relu', name='l_3rd'),
tf.keras.layers.Dense(128, activation='relu', name='l_4th'),
tf.keras.layers.Dense(128, activation='relu', name='l_5th'),
tf.keras.layers.Dense(128, activation='relu', name='l_6th'),
tf.keras.layers.Dense(128, activation='relu', name='l_7th'),
tf.keras.layers.Dense(128, activation='relu', name='l_8th'),
tf.keras.layers.Dense(128, activation='relu', name='l_9th'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10, activation='softmax', name='dense10')
])
l = tf.keras.losses.SparseCategoricalCrossentropy()
opt = tf.keras.optimizers.Adam(0.001)
model.compile(optimizer=opt, loss=l, metrics=['accuracy'])
class ExtendedTensorBoard(tf.keras.callbacks.TensorBoard):
def _log_gradients(self, epoch):
step = tf.cast(epoch, dtype=tf.int64)
writer = self._train_writer
# writer = self._get_writer(self._train_run_name)
with writer.as_default(), tf.GradientTape() as g:
# here we use test data to calculate the gradients
_x_batch = x_train[:100]
_y_batch = y_train[:100]
g.watch(tf.convert_to_tensor(_x_batch))
_y_pred = self.model(_x_batch) # forward-propagation
loss = self.model.loss(y_true=_y_batch, y_pred=_y_pred) # calculate loss
gradients = g.gradient(loss, self.model.trainable_weights) # back-propagation
# In eager mode, grads does not have name, so we get names from model.trainable_weights
for weights, grads in zip(self.model.trainable_weights, gradients):
tf.summary.histogram(
weights.name.replace(':', '_')+'_grads', data=grads, step=step)
writer.flush()
def on_epoch_end(self, epoch, logs=None):
# def on_train_batch_end(self, batch, logs=None):
# This function overwrites the on_epoch_end in tf.keras.callbacks.TensorBoard
# but we do need to run the original on_epoch_end, so here we use the super function.
super(ExtendedTensorBoard, self).on_epoch_end(epoch, logs=logs)
# super(ExtendedTensorBoard, self).on_train_batch_end(batch, logs=logs)
if self.histogram_freq and epoch % self.histogram_freq == 0:
self._log_gradients(epoch)
ee = ExtendedTensorBoard(histogram_freq=1, write_images=True, update_freq='batch')
model.fit(x_train, y_train, epochs=10, callbacks=[ee], validation_data=(x_test, y_test), )
# model.fit(x_train, y_train, epochs=5, callbacks=[gradient_cb, tensorboard_cb]) |
|
I tried to run the code on colab using tf v2.5 and faced attribute error ,please find the gist here..Thanks ! |
|
Now i'm able to get specific error message in the recent Tensorflow version, please find the gist here and confirm the same. Thanks! |
sachinprasadhs
added
stat:awaiting response
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you. |
google-ml-butler
bot
added
the
stale
|
Closing as stale. Please reopen if you'd like to work on this further. |
System information
Describe the current behavior
Program ends with an unclear error, while trying to retrieve the bias gradients of the two dense layers in the model.
Describe the expected behavior
Writing to tensorboard (console parameter = False)
Tensorboard event file which contains the distribution and histograms of gradients
derived from the total loss that has been accumulated over the last epoch.
Priniting to console(console parameter = True)
The program should print the first 10 gradient bias values of each of the two dense layer
to the console.
If the exceptions produced here are the expected behavior due to errors in the developers code
a more meaningful error message would be appriciated.
In that case a correction of the code would be useful for me and other people as well who had to update their code as the write_grads parameter has been removed from the tensorboard callback in version 2.0.
Code to reproduce the issue
The text was updated successfully, but these errors were encountered: