scio-tensorflow tests crashing when cross building #1137
Assignees
Labels
bug
Something isn't working
Comments
nevillelyh
added a commit
that referenced
this issue
May 1, 2018
jto
pushed a commit
that referenced
this issue
May 2, 2018
|
Comes from https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/lib/monitoring/collection_registry.cc#L75-L79 which suggest sbt is somehow loading the library once for the two versions but somehow is initializing it twice. |
|
Running the same test twice in the same sbt session also crashes: |
|
If you launch sbt with I'm not sure what's going on because Tensorflow actually checks if the library is already loaded (https://github.com/tensorflow/tensorflow/blob/a9485a4b59477c722ed480baec9043d04cc25ea0/tensorflow/java/src/main/java/org/tensorflow/NativeLibrary.java#L48-L57). It seems that the lib is properly unloaded between tests. Could be a bug in Tensorflow that makes it leak resources ? What do you think @nevillelyh ? |
|
I looked a bit at the C++ monitoring code, looks like |
|
So I added the following setting in sbt: testOptions in Test += Tests.Setup { _ =>
def loadedLibs: Seq[String] = {
val libs = classOf[ClassLoader].getDeclaredField("loadedLibraryNames")
libs.setAccessible(true)
import scala.collection.JavaConverters._
libs.get(ClassLoader.getSystemClassLoader())
.asInstanceOf[java.util.Vector[String]]
.asScala
}
loadedLibs.foreach(println)
},Here's what happens: So the library is already loaded before running the 2nd test but somehow, |
|
I found a couple of fishy stuff in tensorflow-java. There's a static bloc in At this point your in the middle of the initialisation of the There's also the fact that
|
|
I had a look at sbt's source code to figure out what it's doing with classloader and I think I know what happens. The native library il loaded by
The system The Java classes (including So when you run a test, SBT calls You therefore end up in a situation where the JNI libs are still loaded because they sit in the long lived system Classloader, while all the java classes have been "unloaded" which let's Tensorflow in a weird half loaded state. |
|
Fixed in #1168 |
4lejandrito
added a commit
to 4lejandrito/liferay-portal
that referenced
this issue
Jul 23, 2018
4lejandrito
added a commit
to 4lejandrito/liferay-portal
that referenced
this issue
Jul 23, 2018
This is because tensorflow must be in the root class loader or it will crash every time the module is redeplouyed. See spotify/scio#1137 (comment)
4lejandrito
added a commit
to 4lejandrito/liferay-portal
that referenced
this issue
Jul 24, 2018
This is because tensorflow must be in the root class loader or it will crash every time the module is redeplouyed. See spotify/scio#1137 (comment)
shuyangzhou
pushed a commit
to shuyangzhou/liferay-portal
that referenced
this issue
Jul 25, 2018
This is because tensorflow must be in the root class loader or it will crash every time the module is redeplouyed. See spotify/scio#1137 (comment)
shuyangzhou
pushed a commit
to shuyangzhou/liferay-portal
that referenced
this issue
Jul 25, 2018
This is because tensorflow must be in the root class loader or it will crash every time the module is redeplouyed. See spotify/scio#1137 (comment)
4lejandrito
added a commit
to 4lejandrito/liferay-portal
that referenced
this issue
Jul 25, 2018
This is because tensorflow must be in the root class loader or it will crash every time the module is redeplouyed. See spotify/scio#1137 (comment)
brianchandotcom
pushed a commit
to brianchandotcom/liferay-portal
that referenced
this issue
Jul 26, 2018
This is because tensorflow must be in the root class loader or it will crash every time the module is redeplouyed. See spotify/scio#1137 (comment)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
sbt +scio-tensorflow/testcrashes with something like:#1003 (comment)
Adding
Test / fork := truetoscioTensorflowproject fixes the issue locally but breaks Circle CI builds since now the containers are OOM due to the extra JVM.The text was updated successfully, but these errors were encountered: