Activation histograms

[abinthomasonline]

keras-team/tf-keras#118

design based on:- https://stackoverflow.com/a/70948849

does not support subclassed models

[fchollet]

Thanks for the PR! It's a neat design, but it would involve adding quite a bit of complexity (overriding call methods on all layers in particular) and I'm not sure the value of the feature justifies it. It would also represent a fair amount of overhead on accelerator, especially TPU, due to the need to transfer activation values back to host memory.

I would recommend that you factor this as a standalone easily-reusable callback, rather than adding it to the TensorBoard callback (ActivationHistograms(Callback)) and that you surface it via a 3rd party distribution channel like StackOverflow or your own block post.

[stellarpower]

Admittedly I am hotpatching the version of Keras I have installed, but, the early return here meant that just NaNs were logged:

https://github.com/abinthomasonline/keras/blob/c5a4d842a057433fb7793e5216f34da1a76953fd/keras/callbacks.py#L2850

I simply commented that out should anyone benefit from it.

I'm not sure the value of the feature justifies it.

Agree that it is a bit convoluted however it appears people are asking for this feature. It has proved valuable for me to be able to verify if the activation is becoming saturated or not when the model does not converge, and I don't think it should be difficult for the end user to do this. For me, any performance concerns are secondary to having a working model, given that this is a feature primarily used for debugging. I think adding it as an opt-in feature that won't harm the performance for any users who don't want it would be a viable idea.