-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Spark jobserver giving error while running linearRegressionwithSGD #341
Comments
It looks like that you either miss Breeze as a dependency or have a version mismatch. Do you need to pack spark-core and spark-mllib into the fat jar? I would suggest to make them "provided" |
I see you compiled your project as 2.11. Did you compile & deploy job server as 2.11? Does your version of Spark also match at 2.11? I would also mark Spark dependency as provided as Ji Yan suggested.
|
@velvia & @yanji84 Error still persists after changes, weird thing is that its not consistent. Now I have compiled my project as well as jobserver on scala 2.10 but still I get same error but with some other dependency.
Logs:
I have tested this error for various dependencies which are already avalable in spark.The error is always present with LabeledPoint class. Thanks |
What is your Spark distribution and how are you running job server?
|
@velvia my spark distribution is 1.5.2
|
Naresh, You need to do a local deploy. If you run reStart, then SJS is using job server’s own dependencies, which don’t include mllib at the moment. Or, you can modify package/Dependencies.scala and add the mllib dependency there. -Evan
|
@velvia Thanks a ton!!! Its working. We can close this thread now. |
Hi,
First I will give contents of my project files.
basisStatistics/project/plugins.sbt
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.13.0")
basisStatistics/src/main/scala/linearRegression.scala
basisStatistics/build.sbt
I created a fat JAR by running
sbt assembly
Logs while creating fat jar are
https://gist.github.com/nareshbab/03cee05520e28b0140e6
Logs for uploading jar to jobserver are as below:
But I am always getting an error while running the model
Can someone help me figure out the issue here.
Thanks in Advance
The text was updated successfully, but these errors were encountered: