Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

libnd4j: Layer norm only supports largest axis, backprop doesn't support 4d? #8008

Closed
AlexDBlack opened this issue Jul 12, 2019 · 2 comments

Comments

@AlexDBlack
Copy link
Contributor

commented Jul 12, 2019

java.lang.RuntimeException: Op [layer_norm] execution failed

	at org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.exec(NativeOpExecutioner.java:1669)
	at org.nd4j.linalg.factory.Nd4j.exec(Nd4j.java:7431)
	at org.nd4j.Temp.testLayerNormNCHW(Temp.java:127)
Caused by: java.lang.RuntimeException: NDArray::applyTrueBroadcast method: the shapes of this and other arrays are not suitable for broadcast operation !
	at org.nd4j.nativeblas.Nd4jCpu$NativeOps.execCustomOp(Native Method)
	at

This broadcast will only work when gain/bias is the last value? i.e., [mb,h,w,c] + [c] works, but obviously [mb,c,h,w] + [c] doesn't... we need [mb,c,h,w]+[c,1,1] at least (or equivalently +[1,c,1,1])

Backprop looks off too for the 4D case:
https://github.com/eclipse/deeplearning4j/blob/master/libnd4j/include/ops/declarable/generic/transforms/layer_norm.cpp#L79

Consider the [n,c,h,w] axis 1 case - we should be summing over axes [0,2,3] not just [0].

And I can't find any 4d test cases in libnd4j either.

@AlexDBlack AlexDBlack added this to the 1.0.0-beta5 milestone Jul 27, 2019

@AlexDBlack

This comment has been minimized.

Copy link
Contributor Author

commented Aug 27, 2019

Layer norm 4d case forward pass confirmed working and correct for both NCHW and NHWC cases (java tests added here: SkymindIO#174)

However, backprop is throwing an exception for both NCHW and NHWC cases. Reproducible with the following test case:

    @Test
    public void testLayerNorm4d() {
        int mb = 3;
        int ch = 4;
        for(boolean nchw : new boolean[]{true, false}) {
            INDArray x = Nd4j.rand(DataType.FLOAT, nchw ? new long[]{mb, ch, 8, 8} : new long[]{mb, 8, 8, ch});
            INDArray gain = Nd4j.rand(DataType.FLOAT, ch);
            INDArray bias = Nd4j.rand(DataType.FLOAT, ch);
            INDArray gradAtOut = x.like();

            INDArray xGrad = x.like();
            INDArray gainGrad = gain.like();
            INDArray bGrad = bias.like();

            Nd4j.exec(DynamicCustomOp.builder("layer_norm_bp")
                    .addInputs(x, gain, bias, gradAtOut)
                    .addOutputs(xGrad, gainGrad, bGrad)
                    .addIntegerArguments(1, 2, 3)
                    .addBooleanArguments(nchw)
                    .build());
        }
    }
o.n.l.c.n.o.NativeOpExecutioner - Failed to execute op layer_norm_bp. Attempted to execute with 4 inputs, 3 outputs, 0 targs,1 bargs and 3 iargs. Inputs: [(FLOAT,[3,4,8,8],c), (FLOAT,[4],c), (FLOAT,[4],c), (FLOAT,[3,4,8,8],c)]. Outputs: [(FLOAT,[3,4,8,8],c), (FLOAT,[4],c), (FLOAT,[4],c)]. tArgs: -. iArgs: [1, 2, 3]. bArgs: [true]. Input var names: [input, gain, bias, layernorm-grad]. Output var names: [input-grad, gain-grad, bias-grad] - Please see above message (printed out from c++) for a possible cause of error.
java.lang.RuntimeException: Op [layer_norm_bp] execution failed
	at org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.exec(NativeOpExecutioner.java:1710)

@AlexDBlack

This comment has been minimized.

Copy link
Contributor Author

commented Aug 28, 2019

Confirmed fixed, including backprop and gradient checks.

@AlexDBlack AlexDBlack closed this Aug 28, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.