Example Code Throws Null Pointer Exception with 7.3 LTS (includes Apache Spark 3.0.1, Scala 2.12) #285

wangmiao1981 · 2020-12-08T01:38:05Z

wangmiao1981
Dec 8, 2020

Describe the issue

I try to run some example code in a Spark notebook with 7.3 LTS (includes Apache Spark 3.0.1, Scala 2.12).

To Reproduce

Create a notebook and copy & paste the following code:

`
import org.apache.spark.sql.SparkSession
import com.microsoft.hyperspace._
import com.microsoft.hyperspace.Hyperspace
import com.microsoft.hyperspace.index.IndexConfig

val spark = SparkSession
.builder()
.appName("Hyperspace example")
.config("spark.some.config.option", "some-value")
.getOrCreate()

val deptLocation = "dbfs:/departments/"
departments
.toDF("deptId", "deptName", "location")
.write
.mode("overwrite")
.parquet(deptLocation)

val empLocation = "dbfs:/employees/"
employees
  .toDF("empId", "empName", "deptId")
  .write
  .mode("overwrite")
  .parquet(empLocation)

val departments = Seq(
(10, "Accounting", "New York"),
(20, "Research", "Dallas"),
(30, "Sales", "Chicago"),
(40, "Operations", "Boston"))

// Sample employee records.
val employees = Seq(
  (7369, "SMITH", 20),
  (7499, "ALLEN", 30),
  (7521, "WARD", 30),
  (7566, "JONES", 20),
  (7698, "BLAKE", 30),
  (7782, "CLARK", 10),
  (7788, "SCOTT", 20),
  (7839, "KING", 10),
  (7844, "TURNER", 30),
  (7876, "ADAMS", 20),
  (7900, "JAMES", 30),
  (7934, "MILLER", 10),
  (7902, "FORD", 20),
  (7654, "MARTIN", 30))

// Save example data records as Parquet.
import spark.implicits._

val deptLocation = "dbfs:/departments/"
departments
  .toDF("deptId", "deptName", "location")
  .write
  .mode("overwrite")
  .parquet(deptLocation)

val empLocation = "dbfs:/employees/"
employees
.toDF("empId", "empName", "deptId")
.write
.mode("overwrite")
.parquet(empLocation)

val hyperspace = new Hyperspace(spark)

val deptDF = spark.read.parquet(deptLocation)
val empDF = spark.read.parquet(empLocation)
val deptIndexConfig = IndexConfig("deptIndex", Seq("deptId"), Seq("deptName"))
val empIndexConfig = IndexConfig("empIndex", Seq("empId"), Seq("empName"))
Exceptions:java.lang.NullPointerException
at com.microsoft.hyperspace.index.sources.FileBasedSourceProviderManager.run(FileBasedSourceProviderManager.scala:96)
at com.microsoft.hyperspace.index.sources.FileBasedSourceProviderManager.signature(FileBasedSourceProviderManager.scala:80)
at com.microsoft.hyperspace.index.FileBasedSignatureProvider.$anonfun$fingerprintVisitor$1(FileBasedSignatureProvider.scala:53)
at com.microsoft.hyperspace.index.FileBasedSignatureProvider.$anonfun$fingerprintVisitor$1$adapted(FileBasedSignatureProvider.scala:51)
at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:190)
at com.microsoft.hyperspace.index.FileBasedSignatureProvider.fingerprintVisitor(FileBasedSignatureProvider.scala:51)
at com.microsoft.hyperspace.index.FileBasedSignatureProvider.signature(FileBasedSignatureProvider.scala:40)
at com.microsoft.hyperspace.index.IndexSignatureProvider.signature(IndexSignatureProvider.scala:45)
at com.microsoft.hyperspace.actions.CreateActionBase.getIndexLogEntry(CreateActionBase.scala:64)
at com.microsoft.hyperspace.actions.CreateAction.logEntry(CreateAction.scala:38)
at com.microsoft.hyperspace.actions.Action.begin(Action.scala:50)
at com.microsoft.hyperspace.actions.Action.run(Action.scala:90)
at com.microsoft.hyperspace.actions.Action.run$(Action.scala:83)
at com.microsoft.hyperspace.actions.CreateAction.run(CreateAction.scala:30)
at com.microsoft.hyperspace.index.IndexCollectionManager.create(IndexCollectionManager.scala:43)
at com.microsoft.hyperspace.index.CachingIndexCollectionManager.create(CachingIndexCollectionManager.scala:77)
at com.microsoft.hyperspace.Hyperspace.createIndex(Hyperspace.scala:43)
at line04b8f3115af7439a97e3ee6d9355b81c80.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.(command-1356655024485756:1)
at line04b8f3115af7439a97e3ee6d9355b81c80.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.(command-1356655024485756:70)
at line04b8f3115af7439a97e3ee6d9355b81c80.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.(command-1356655024485756:72)
at line04b8f3115af7439a97e3ee6d9355b81c80.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.(command-1356655024485756:74)`

Expected behavior

Environment

sezruby · 2020-12-08T08:47:20Z

sezruby
Dec 8, 2020

Hyperspace currently supports only Spark 2.4 and support for Spark 3.0 is on the way: #85.

0 replies

wangmiao1981 · 2020-12-08T18:42:00Z

wangmiao1981
Dec 8, 2020
Author

@sezruby Thanks! I also suspect the runtime version issue. Will create a new cluster with 2.4.

0 replies

wangmiao1981 · 2020-12-08T18:44:20Z

wangmiao1981
Dec 8, 2020
Author

One more comment: @sezruby Spark 2.4 is scale-2.11. By default, Hyperspace seems compiled with Scala-2.12.

0 replies

sezruby · 2020-12-09T12:38:36Z

sezruby
Dec 9, 2020

Yes it's 2.12 by default, so you need to change the version as 2.11 in build.sbt.
It's being delayed to support Spark 3.0 for now. Thanks!

0 replies

imback82 · 2020-12-09T16:18:57Z

imback82
Dec 9, 2020

Or if you are using sbt CLI to build, you can do the following:

> ++ 2.11.12
[info] Setting version to 2.11.12
> compile

, or do the following to build both:

> +compile

0 replies

wangmiao1981 · 2020-12-10T02:54:53Z

wangmiao1981
Dec 10, 2020
Author

I switched to Databricks runtime 6.4 (includes Apache Spark 2.4.5, Scala 2.11) with ADLS Gen2 as my storage (also tried DBFS).

It throws new exception if I try to create index.

java.lang.ClassCastException: org.apache.spark.sql.execution.datasources.SerializableFileStatus cannot be cast to org.apache.hadoop.fs.FileStatus at com.microsoft.hyperspace.index.sources.default.DefaultFileBasedSource$$anonfun$3.apply(DefaultFileBasedSource.scala:134) at scala.math.Ordering$$anon$5.compare(Ordering.scala:122) at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355) at java.util.TimSort.sort(TimSort.java:220) at java.util.Arrays.sort(Arrays.java:1438) at scala.collection.SeqLike$class.sorted(SeqLike.scala:648) at scala.collection.AbstractSeq.sorted(Seq.scala:41) at scala.collection.SeqLike$class.sortBy(SeqLike.scala:623) at scala.collection.AbstractSeq.sortBy(Seq.scala:41) at com.microsoft.hyperspace.index.sources.default.DefaultFileBasedSource.signature(DefaultFileBasedSource.scala:134) at com.microsoft.hyperspace.index.sources.FileBasedSourceProviderManager$$anonfun$signature$1.apply(FileBasedSourceProviderManager.scala:80) at com.microsoft.hyperspace.index.sources.FileBasedSourceProviderManager$$anonfun$signature$1.apply(FileBasedSourceProviderManager.scala:80) at com.microsoft.hyperspace.index.sources.FileBasedSourceProviderManager$$anonfun$run$1.apply(FileBasedSourceProviderManager.scala:97) at com.microsoft.hyperspace.index.sources.FileBasedSourceProviderManager$$anonfun$run$1.apply(FileBasedSourceProviderManager.scala:96) at scala.collection.IndexedSeqOptimized$class.foldl(IndexedSeqOptimized.scala:57) at scala.collection.IndexedSeqOptimized$class.foldLeft(IndexedSeqOptimized.scala:66) at scala.collection.mutable.ArraySeq.foldLeft(ArraySeq.scala:46) at com.microsoft.hyperspace.index.sources.FileBasedSourceProviderManager.run(FileBasedSourceProviderManager.scala:96) at com.microsoft.hyperspace.index.sources.FileBasedSourceProviderManager.signature(FileBasedSourceProviderManager.scala:80) at com.microsoft.hyperspace.index.FileBasedSignatureProvider$$anonfun$fingerprintVisitor$1.apply(FileBasedSignatureProvider.scala:53) at com.microsoft.hyperspace.index.FileBasedSignatureProvider$$anonfun$fingerprintVisitor$1.apply(FileBasedSignatureProvider.scala:51)

0 replies

wangmiao1981 · 2020-12-10T02:55:24Z

wangmiao1981
Dec 10, 2020
Author

@sezruby @imback82 any thoughts on the above exception?

0 replies

wangmiao1981 · 2020-12-10T02:56:40Z

wangmiao1981
Dec 10, 2020
Author

@sezruby I will first make it work for Spark 2.4 and rebuild runtime for Spark 3.0 later.

0 replies

imback82 · 2020-12-10T03:39:03Z

imback82
Dec 10, 2020

I believe SerializableFileStatus is a known issue in the Databricks (they seem to have a different implementation from the OSS Spark). We are looking into a workaround. cc @rapoth

0 replies

wangmiao1981 · 2020-12-10T18:23:28Z

wangmiao1981
Dec 10, 2020
Author

Glad to know it is not my problem. I can spend some time to see if we can walk-around or get some insights from DB engineering. I am closely working with them on other projects.

0 replies

rapoth · 2020-12-10T22:51:53Z

rapoth
Dec 10, 2020

Thank you for your interest @wangmiao1981!

We are sorry you are hitting into this issue. We tested Hyperspace against open-source Apache Spark so there maybe a few minor modifications to make it work with Databricks. If you manage to find any workarounds or the changes needed to make it work after your discussion with DB engineering, please feel free to open a PR! Thanks again!

0 replies

wangmiao1981 · 2020-12-16T23:32:17Z

wangmiao1981
Dec 16, 2020
Author

Response from Databricks:
yea, we'll have to manually convert it
val statuses = serializableStatuses.map { f =>
new FileStatus(f.length, f.isDir, f.blockReplication, f.blockSize, f.modificationTime, new Path(f.path)), blockLocations)
}

I am following up with them how to import it in the source code, because serializableStatuses is a Private class afaik.

I may be able to submit a PR once I get more details from Databricks.

0 replies

wangmiao1981 · 2020-12-19T17:59:36Z

wangmiao1981
Dec 19, 2020
Author

@rapoth our team member Andrei made Hyperspace working with 5.5 LTS, Spark 2.4.3 last night. He will submit a PR soon.

0 replies

rapoth · 2020-12-19T18:08:03Z

rapoth
Dec 19, 2020

This is great news! Thank you for the update and look forward to the PR!

1 reply

andrei-ionescu Jan 6, 2021

This has been fixed with #303.

andrei-ionescu · 2021-01-06T15:00:46Z

andrei-ionescu
Jan 6, 2021

Fixed by #303.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Example Code Throws Null Pointer Exception with 7.3 LTS (includes Apache Spark 3.0.1, Scala 2.12) #285

{{title}}

Replies: 15 comments 1 reply

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Example Code Throws Null Pointer Exception with 7.3 LTS (includes Apache Spark 3.0.1, Scala 2.12) #285

Describe the issue

To Reproduce

Expected behavior

Environment

Replies: 15 comments · 1 reply

wangmiao1981 Dec 8, 2020 Author

wangmiao1981 Dec 8, 2020 Author

wangmiao1981 Dec 10, 2020 Author

wangmiao1981 Dec 10, 2020 Author

wangmiao1981 Dec 10, 2020 Author

wangmiao1981 Dec 10, 2020 Author

wangmiao1981 Dec 16, 2020 Author

wangmiao1981 Dec 19, 2020 Author

Replies: 15 comments 1 reply

wangmiao1981
Dec 8, 2020
Author

wangmiao1981
Dec 8, 2020
Author

wangmiao1981
Dec 10, 2020
Author

wangmiao1981
Dec 10, 2020
Author

wangmiao1981
Dec 10, 2020
Author

wangmiao1981
Dec 10, 2020
Author

wangmiao1981
Dec 16, 2020
Author

wangmiao1981
Dec 19, 2020
Author