Question

[tchaton]

Hello guys,

Very nice piece of work.
I was wondering why you didn't use a
einsum implementation of the bilinear attention in order to speed up training.

This equation is perfect for it. U should have a significant gain, and it would be nice for once to have highly optimized code available on github.

Best,
T.C

[tchaton]

Hello there,

I have implemented the module with einsum in order to optimize the code.
Please tell me if you think it is correct ? Tensor-wise it works, however, the output values are huge, which suggest something wrong.

[tchaton]

Hello guys,

I have corrected a bit the code.

However, I had to normalize by the max for softmax not to saturate.

Could you please correct this version and also tell me how you solve the softmax issue ?

Best,
T.C

[linjieli222]

Softmax is applied to both dimensions in attention.py, not only the columns (dim = 1).

[tchaton]

Ok, I will change that :)

https://github.com/Zhaoyi-Yan/Shift-Net_pytorch/blob/sobel/models/bilinear_shift_net/innerBilinearShiftTripleModule.py#L68

I have implemented the module there. Could you please have a quick look to tell me if you think it is correct ?
Using einsum, it is very fast.

Best,
T.C

[jnhwkim]

@tchaton in our implementation, the computational speed is not much different from the previous. I doubt that the other factors, missing nonlinear activations, regularization, and the other implementational details, may be related. However, we found that memory consumption is hugely reduced (around 30%) due to its efficiency in dealing with computational temporaries. Please refer to our implementation in #23. Thank you for the heads-up.