Exception in thread "main" java.lang.NoClassDefFoundError:org/apache/spark/mllib/feature/HashingTF

[manishdalmiya]

I am trying to run Collect.scala from Twitter Streaming Language Classifier and keep getting below error. I think i followed all the insructions but, i am still stuck with this error. I would really appreciate if someone can help.

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/mllib/feature/HashingTF
at com.databricks.apps.twitter_classifier.Utils$.(Utils.scala:12)
at com.databricks.apps.twitter_classifier.Utils$.(Utils.scala)
at com.databricks.apps.twitter_classifier.Collect$.main(Collect.scala:26)
at com.databricks.apps.twitter_classifier.Collect.main(Collect.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:292)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.mllib.feature.HashingTF
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 11 more

[vidaha]

Hi Manish,

For this to run, it's assumed you have Spark 1.1 installed, including having the spark mllib jar version 1.1 in your classpath.

As a quick workaround, you could comment out the "provided" statements, but please don't submit a pull request for that and leave that just in your local branch. For more details about what provided does - see this link: https://github.com/databricks/spark-knowledgebase/blob/master/troubleshooting/missing_dependencies_in_jar_files.md

-Vida