New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Loading nd4j multiple times in the same JVM results in UnsatisfiedLinkError #6166
Comments
|
If you're not launching multiple instances of the same app, you could create native libraries with unique names for each app... |
|
BTW, this is something that needs to be fixed in the JDK. If you have any "ammunition" (I really don't like this language, but this is what it is, it's a "fight" against people who do not wish for things to change) we can use to convince the OpenJDK community (think Oracle...) that this is something we need urgently, please do let us know your arguments. This is an issue that comes up with any other libraries using JNI too, such as TensorFlow: tensorflow/tensorflow#19298 /cc @johanvos |
|
@saudet In our particular case, we're running the same app (lib-wise) multifold, but they're distinguished configuration-wise. We've tried to let javacpp cache into different directories, but that's not possible reliably because the cache directory is only configurable globally (system property |
|
The only "ammunition" (I don't like that word too) you need is time to do the work, not to fight. Nobody in OpenJDK will stop you from doing work, but you should be prepared to do it yourself, and it's lots of work and commitment. |
|
@CedricReichenbach The issue isn't with the native libraries, it's with the JDK. Basically, the class that calls @johanvos It's more than that, it's always more than just getting the work done, it's always about politics, always, and I'm very glad you're here, because I'm unable to deal with it. In this case the issue is with a built-in limitation of JNI. If you read https://docs.oracle.com/javase/7/docs/technotes/guides/jni/jni-12.html very carefully you'll notice this paragraph buried in there:
In other words, this is a feature, not a bug. And what they plan to do about this is to have everyone switch to what they are working on in Panama, and because they promise that their replacement for JNI is going to fix all this, they are not considering fixing this for JNI: But they have been promising to come up with this for over 4 years now, with no concrete plans or roadmap about what is going to become available when, and even if they do eventually come out with something, we'll still need to use JNI for Android anyway, but we're talking about OpenJDK here. They don't care about what we need for Android, so here we are: politics! |
|
@CedricReichenbach It might be possible to hack around this if we disable the SecurityManager. Would this be something that you could live with? |
|
@saudet As a workaround, we now make sure to only use dl4j/nd4j (and thus the whole native shebang) in one of the webapps, and avoid in the others. That works for us because using neural networks is only crucial in one of them. Our product is a CMS we're shipping to customers who might run them in any kind of way, so we cannot give a canonical answer on whether the system classloader workaround is acceptable. But it could certainly be helpful at least for dev and testing setups if it's easily configurable. |
|
@CedricReichenbach |
|
@saudet As said above, we don't control our customers' setup, it might be anything. But if JNI is already tied to either disabled or customized security, then I guess it's sensible that users enabling it would also have to configure exceptions for the system class loader. If this is not an option for them, we could give them the option to opt-out of features requiring dl4j. |
|
@CedricReichenbach Yes, it sounds like they would need to enable permissions for JNI anyway, so I think we can assume that we can access the system class loader. However, we still can't have it load arbitrary classes, its class path is set by the container, so I'm still not sure how we could hack this yet... |
|
I think I was able to work around this successfully in commit bytedeco/javacpp@dd57c2c. I've tested a bit on Tomcat and it appears to work well. There might be few things to adjust in ND4J, so more testing welcome! Please give it a try with JavaCPP 1.4.3-SNAPSHOT: http://bytedeco.org/builds/ |
|
@CedricReichenbach I'll close this for now, but please let me know if you still encounter issues with JavaCPP 1.4.3-SNAPSHOT. Thanks for the help! |
|
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
We're running into a similar problem as others before, e.g. this one: https://github.com/deeplearning4j/nd4j/issues/1507
In our case, we're running a tomcat containing two webapps, each of which containing nd4j-native jars (brought in by a maven dependency to
nd4j-native-platform).When calling
build()on a network configuration (likely in the second webapp), the following exception is thrown:When running just one of the two webapps alone, everything works flawlessly.
The text was updated successfully, but these errors were encountered: