DL4J: LRN doesn't match MKL-DNN implementation, libnd4j implementation incorrect

[AlexDBlack]

When implementing MKL-DNN support for DL4J, I found that MKL-DNN implementation doesn't provide the same result as DL4J for backprop - forward pass is OK.
After digging further, I'm unable to rule out the possibility that both the DL4J LRN backprop implementation and the libnd4j LRN backprop implementations are incorrect here.
DL4J: For example, setting alpha=1, k=0, beta=1 (about the simplest possible case for LRN) results in gradient checks failing when MKL-DNN is not used. Smaller values of alpha (<=1e-4) pass gradient checks. I suspect with a small enough value for alpha, the incorrect component of the gradient calculation just doesn't cause enough of a problem to trigger a test failure.

When MKL-DNN is enabled, I'm not seeing any evidence it is used when set to double precision (as required for gradient checks) - only for float precision. i.e., no output when MKLDNN_VERBOSE=1 env variable is set. Which suggests the libnd4j helper implementation is being used instead, and this is also incorrect - as gradient checks fail when it is used (including for cases that DL4J passes for).

http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf Section 3.3

Aha! Link: https://skymindai.aha.io/features/DL4J-5

[raver119]

libnd4j LRN BP being incorrect is known issue, and is a wip.

[AlexDBlack]

Right, I'm aware, I need to derive math for that. DL4J also being wrong is main new thing here.