New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error while running the mxnet spark examples and test cases #13853

Open
thomelane opened this Issue Jan 11, 2019 · 3 comments

Comments

Projects
None yet
3 participants
@thomelane
Copy link
Contributor

thomelane commented Jan 11, 2019

Opening issue on behalf of @adwivedi on the discussion forum (https://discuss.mxnet.io/t/error-while-running-the-mxnet-spark-examples-and-test-cases/2720).

Quotes from the thread...

I am trying to run the mxnet in distributed mode using spark as implemented here : https://github.com/apache/incubator-mxnet/tree/master/scala-package/spark

but I am not able to run the examples and/or tests.

The commented tests in the file : https://github.com/apache/incubator-mxnet/blob/master/scala-package/spark/src/test/scala/org/apache/mxnet/spark/MXNetGeneralSuite.scala keep running into the following error. I get the same error when I try to run the examples in the repo.

Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: scala/collection/Seq
	at java.lang.Class.getDeclaredMethods0(Native Method)
	at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
	at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
	at java.lang.Class.getMethod0(Class.java:3018)
	at java.lang.Class.getMethod(Class.java:1784)
	at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
	at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: scala.collection.Seq
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 7 more
Exception in thread "Thread-21" java.lang.IllegalArgumentException: requirement failed: Failed to start ps scheduler process with exit code 1
	at scala.Predef$.require(Predef.scala:224)
	at org.apache.mxnet.spark.MXNet.org$apache$mxnet$spark$MXNet$$startPSSchedulerInner$1(MXNet.scala:159)
	at org.apache.mxnet.spark.MXNet$$anonfun$startPSScheduler$1.apply(MXNet.scala:162)
	at org.apache.mxnet.spark.MXNet$$anonfun$startPSScheduler$1.apply(MXNet.scala:162)
	at org.apache.mxnet.spark.MXNet$MXNetControllingThread.run(MXNet.scala:38)
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: scala/collection/Seq
	at java.lang.Class.getDeclaredMethods0(Native Method)
	at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
	at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
	at java.lang.Class.getMethod0(Class.java:3018)
	at java.lang.Class.getMethod(Class.java:1784)
	at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
	at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: scala.collection.Seq
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 7 more

I have seen this error generally when there’s a mismatch between the scala versions in the api, but in this case I am using the using the pom file that’s in the project and not including any external libraries.
I have also looked at the pom file but I’ve not found any lib that might have a mismatch in this case, all the libraries in pom are 2.11 version

I am building it from source, running it with this vm parameter : -

Djava.library.path=/path_to_mxnet_source/incubator-mxnet/scala-package/native/osx-x86_64-cpu/target

to let it find the native library.

@thomelane

This comment has been minimized.

Copy link
Contributor

thomelane commented Jan 11, 2019

@thomelane

This comment has been minimized.

Copy link
Contributor

thomelane commented Jan 11, 2019

@mxnet-label-bot add [Scala]

@marcoabreu marcoabreu added the Scala label Jan 11, 2019

@lanking520 lanking520 self-assigned this Jan 11, 2019

@lanking520

This comment has been minimized.

Copy link
Contributor

lanking520 commented Jan 11, 2019

Thanks for @thomelane raising it here. Add a bunch of Scala guru here:
@piyushghai @zachgk @andrewfayres @CodingCat
Please take a look at this issue. I also think it is a good time now to fix the flaky test on the disabled tests and allow Spark run in the CI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment