DL4J EmbeddingLayer includes bias but Keras Embedding does not

[zheolong]

How can I fix this when loading keras model with hdf5 file.

org.deeplearning4j.nn.modelimport.keras.UnsupportedKerasConfigurationException: DL4J EmbeddingLayer includes bias but Keras Embedding does not. Please file an issue at http://github.com/deeplearning4j/deeplearning4j/issues.
	at org.deeplearning4j.nn.modelimport.keras.layers.KerasEmbedding.<init>(KerasEmbedding.java:67)
	at org.deeplearning4j.nn.modelimport.keras.KerasLayer.getKerasLayerFromConfig(KerasLayer.java:235)
	at org.deeplearning4j.nn.modelimport.keras.KerasModel.helperPrepareLayers(KerasModel.java:190)
	at org.deeplearning4j.nn.modelimport.keras.KerasSequentialModel.<init>(KerasSequentialModel.java:99)
	at org.deeplearning4j.nn.modelimport.keras.KerasSequentialModel.<init>(KerasSequentialModel.java:56)
	at org.deeplearning4j.nn.modelimport.keras.KerasModel$ModelBuilder.buildSequential(KerasModel.java:581)
	at org.deeplearning4j.nn.modelimport.keras.KerasModelImport.importKerasSequentialModelAndWeights(KerasModelImport.java:156)
	at com.antfin.matrix.h7p.huabei.DL4JTest.loadModel(DL4JTest.java:65)
	at com.antfin.matrix.h7p.huabei.DL4JTest.ArksRequestTest(DL4JTest.java:45)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:78)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:212)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:68)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)

[turambar]

@zheolong Are you loading this model for training or just inference (i.e., no further training, just making predictions)?

If it's the latter, then you simply need to import the model with enforceTrainingConfig=False and it will import and work fine (by setting the DL4J EmbeddingLayer bias to 0).

If you intend to train the model further, let us know: this may require changes to core DL4J, specifically the DL4J EmbeddingLayer.

@eraly @AlexDBlack keep an eye on this as well. We might be able to handle this with FrozenLayer; if not, we need to have a chat about our embedding layer (and why we have a bias, which seems non-standard as far as I can tell).

[zheolong]

The latter situation, it works, thx

[AlexDBlack]

@turambar the idea: our embedding layer is literally/mathematically identical to a dense layer with one-hot input.

One option is to set bias LR to 0 (and if necessary, activation function to identity - not sure) to get something equivalent to Keras' implementation.

[turambar]

@AlexDBlack Editing message -- I didn't read your response properly. Yeah, we can set the bias to zero and the learning rate to zero so that it doesn't change during training. That should work fine.

[maxpumperla]

this has been addressed through #3906

[shengc]

@turambar I actually like your idea better...

new FrozenLayer.Builder(
    new EmbeddingLayer.Builder().nIn(nIn).nOut(nOut).biasInit(0.0).activate(Activation.Identity).build()
).build()

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.