DL4J Keras model import with CUDA backend strange behavior (0.9.2-SNAPSHOT)

[emc5ud]

This issue is a little strange.

So I trained a model in keras with tensorflow as a backend and would like to import it to DL4J. This works when I set my backend to be "nd4j-native", but fails when I set by backend as "nd4j-cuda-8.0".

I create the model in the following way: unet.py (training omitted). And load it into DL4J as so

  def getModel() = {
    KerasModelImport.importKerasModelAndWeights("src/main/resources/unet.h5", false)
  }

When I load the model in DL4J with GPUs enabled, process hangs at the above step. I've waited for about 10 minutes with no luck, and all the while the process uses a steady 6-7 GB of my GPU's memory.

Now the strange part is that when I create the model in Keras with "tf" ordering (aka channels last) the hang up no longer happens and everything is great (unet_tf_ordering.py). When I use CPU it doesn't care what ordering I use.

Relevant python libraries:

• keras==1.2.2
• tensorflow-gpu==1.4

Relevant part of my build.sbt:

libraryDependencies ++= Seq(
  "org.deeplearning4j" % "deeplearning4j-core" % "0.9.2-SNAPSHOT",
  "org.deeplearning4j" % "deeplearning4j-modelimport" % "0.9.2-SNAPSHOT",
  "org.nd4j" % "nd4j-cuda-8.0" % "0.9.2-SNAPSHOT" classifier "" classifier "linux-x86_64",
  "org.bytedeco.javacpp-presets" % "cuda" % "8.0-6.0-1.3" classifier "" classifier "linux-x86_64"
//  "org.nd4j" % "nd4j-native" % "0.9.2-SNAPSHOT" classifier "" classifier "linux-x86_64",
//  "org.bytedeco.javacpp-presets" % "mkl" % "2017.3-1.3" classifier "" classifier "linux-x86_64",
//  "org.bytedeco.javacpp-presets" % "openblas" % "0.2.20-1.3" classifier "" classifier "linux-x86_64"
)

[maxpumperla]

Hey @emc5ud we had an HDF5 fix recently (here: #4870), would you mind checking the import again with CUDA? I have a hunch this will fix it.

[maxpumperla]

@emc5ud could you please give it a try on your end? thanks

[emc5ud]

Hi Max, sorry for the delay. I'll give it a try this weekend.

[maxpumperla]

@emc5ud hey, any feedback for me on this? This seems resolved, but feel free to reopen, OK? Thanks

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.