Visualise Self-Attention weights response

[tallamjr]

It would desirable to be able to plot the weight responses of an input sequence (light curve)

This has benefits of being more explainable and would help justify the use of this mechanism for this problem.

Something akin to the image below from https://github.com/cordeirojoao/ECG_Processing/blob/master/Ecg_keras_v9-Raphael.ipynb would be useful

This might be able to be achieved by branching off the TransformerBlock and inspecting the weights at this level. See Autoencoders notebook in https://github.com/astro-informatics/course_mlbd/blob/master/Lectures/Lecture21_AutoEncoders.ipynb for example of plotting the weights

Possible useful links:

• https://github.com/HarisIqbal88/PlotNeuralNet
• https://medium.com/datalogue/attention-in-keras-1892773a4f22
• https://jacobgil.github.io/deeplearning/class-activation-maps
• https://gist.github.com/RaphaelMeudec/e9a805fa82880876f8d89766f0690b54
• https://github.com/sicara/tf-explain
• https://stackoverflow.com/questions/58356868/how-visualize-attention-lstm-using-keras-self-attention-package
• https://www.researchgate.net/figure/The-heat-map-visualization-of-the-learned-attention-weights-by-our-spatial-attention_fig4_321306916
• https://www.machinecurve.com/index.php/2019/11/25/visualizing-keras-cnn-attention-saliency-maps/

[tallamjr]

Closed with #59