How to visualize 2d multihead attention?

[anglil]

Given the js and python code for visualizing 1d multihead atttention, what would be the easiest way to achieve 2d attention visualization? (in terms of arranging a plane instead of a sequence for multiple heads in an ipython notebook environment)

[dimitry12]

@anglil have you found any good visualization methods?

[anglil]

@dimitry12 I ended up flattening out the 2d matrix (m by n) and visualizing the vector (1 by m*n) instead. I'd be interested in learning about ways to visualize 2d attention modules directly, too.

[Zminghua]

@anglil what is your method for visualizing 1d multihead atttention ?

[anglil]

@Zminghua i used the provided visualization code.

[Zminghua]

@anglil oh, thanks!
but I implemented the transformer myself, how can i use the provided visualization code?

[anglil]

@Zminghua that might add some complication, but in general this ipython notebook is a good starting point: https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/visualization/TransformerVisualization.ipynb
basically you'd want to let the visualizer find the attention weights from your implementation of transformer, and be able to take some input sentence and generate an output sentence.

[Zminghua]

@anglil Thanks very much.
i have adapted the provided visualization code to my "transformer".