Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

import session.implicits._ doesn't work #169

Closed
raviteja-ms opened this issue Mar 16, 2017 · 2 comments
Closed

import session.implicits._ doesn't work #169

raviteja-ms opened this issue Mar 16, 2017 · 2 comments

Comments

@raviteja-ms
Copy link

raviteja-ms commented Mar 16, 2017

Spark Version: 2.0.2
Scala Version: 2.11.8

When I tried a simple example to import spark.implicits._ I get the following error while sbt build:

My code:

import io.hydrosphere.mist.lib.{MistJob, SQLSupport}
import org.apache.spark.sql._

object SimpleContext extends MistJob with SQLSupport{
  /** Contains implementation of spark job with ordinary [[org.apache.spark.SparkContext]]
    * Abstract method must be overridden
    *
    * based on https://github.com/Hydrospheredata/mist/blob/master/examples/src/main/scala/SimpleContext.scala
    *
    * @param digits user parameter digits
    * @return result of the job
    */
 import session.implicits._
 def execute(digits:Seq[Int]): Map[String, Any] = {
        val mydf = digits.toSeq.toDF("number")
        Map("result" -> mydf.map(x => x.getInt(0) * 3).collect())
  }
}

Error:

[info] Loading project definition from /home/user/mist/apache-spark-restAPI-example/project
[info] Set current project to sparkMist (in build file:/home/user/mist/apache-spark-restAPI-example/)
[info] Compiling 1 Scala source to /home/user/mist/apache-spark-restAPI-example/target/scala-2.11/classes...
[error] /home/user/mist/apache-spark-restAPI-example/src/main/scala/SimpleContext.scala:13: stable identifier required, but SimpleContext.this.session.implicits found.
[error]  import session.implicits._
[error]                 ^
[error] one error found
[error] (compile:compileIncremental) Compilation failed

Please help me how I can import spark.implicits._

Thanks..!!

@dos65
Copy link
Contributor

dos65 commented Mar 16, 2017

Currently inside MistJob session is defined as a function, but should be a value. We will fix it later.
As a workaround I can suggest to assign session to some variable and then call implicits import from it.

import io.hydrosphere.mist.lib.{MistJob, SQLSupport}
import org.apache.spark.sql._

object SimpleContext extends MistJob with SQLSupport {

   val s = session
   import s.implicits._  

  def execute(digits:Seq[Int]): Map[String, Any] = {
      val mydf = digits.toSeq.toDF("number")
      Map("result" -> mydf.map(x => x.getInt(0) * 3).collect())
  }
}

@raviteja-ms
Copy link
Author

Thanks, this worked...!!
But, please mention this documentation or in any example somewhere. I was confused and other new users will be too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants