SW REPL cannot be used in combination with Spark Dataset

[exalate-issue-sync]

{format}
case class Person(id: Long)
val ds = Seq(Person(0), Person(1)).toDS
val count = ds.count
{format}

executed from Flow REPL will throw an exception:

{noformat}
defined class Person
org.apache.spark.sql.AnalysisException: Unable to generate an encoder for inner class Person without access to the scope that this class was defined in.
Try moving this class out of its parent class.;
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveNewInstance$$anonfun$apply$33$$anonfun$applyOrElse$13.applyOrElse(Analyzer.scala:2104)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveNewInstance$$anonfun$apply$33$$anonfun$applyOrElse$13.applyOrElse(Analyzer.scala:2100)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:288)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:288)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:287)
at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressionDown$1(QueryPlan.scala:248)
at org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1(QueryPlan.scala:258)
at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$6.apply(QueryPlan.scala:267)
at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:188)
at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressionsDown(QueryPlan.scala:267)
at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressions(QueryPlan.scala:236)
{noformat}

The problem is that Spark OuterScopes has hardcoded structure of generated REPL classes:
{noformat}
private[this] val REPLClass = """^($line(?:\d+).$read)(?:$$iw)+$""".r
{noformat}

More readings:

• https://issues.scala-lang.org/browse/SI-9734
• https://issues.apache.org/jira/browse/SPARK-13456
• https://issues.apache.org/jira/browse/SPARK-14146
  We need to explicitly register OuterScope for Spark SQL encoders
  See also here: http://stackoverflow.com/questions/34964565/how-to-create-a-dataset-from-custom-class-person
  and here apache/spark@43ebf7a

[DinukaH2O]

JIRA Issue Migration Info

Jira Issue: SW-386
Assignee: Michal Malohlava
Reporter: Michal Malohlava
State: Resolved
Fix Version: 2.1.3
Attachments: N/A
Development PRs: Available

Linked PRs from JIRA

#243

[hasithjp]

JIRA Issue Migration Info Cont'd

Jira Issue Created Date: 2017-04-04T18:43:53.450-0700