Skip to content

Create SparkSession in SparkFunSuite#122

Closed
fnothaft wants to merge 2 commits intobigdatagenomics:masterfrom
fnothaft:issues/120-spark-session
Closed

Create SparkSession in SparkFunSuite#122
fnothaft wants to merge 2 commits intobigdatagenomics:masterfrom
fnothaft:issues/120-spark-session

Conversation

@fnothaft
Copy link
Copy Markdown
Member

@fnothaft fnothaft commented May 1, 2018

Resolves #120. Necessitates bumping project to Spark 2. (I bumped to Scala 2.11 by default as well). This is inline with ADAM downstream, so I figured it was OK!

Copy link
Copy Markdown
Member

@heuermh heuermh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@heuermh
Copy link
Copy Markdown
Member

heuermh commented May 1, 2018

This fails to build for me locally

$ mvn clean install
...
[INFO] Compiling 4 source files to /Users/mheuer2/working/utils/utils-misc/target/2.11.8/test-classes at 1525194272851
[ERROR] /Users/mheuer2/working/utils/utils-misc/src/test/scala/org/bdgenomics/utils/misc/MathUtilsSuite.scala:25:
error: can't expand macros compiled by previous versions of Scala
[ERROR]     assert(MathUtils.fpEquals(0.0, 0.0))
[ERROR]                              ^
[ERROR] /Users/mheuer2/working/utils/utils-misc/src/test/scala/org/bdgenomics/utils/misc/MathUtilsSuite.scala:26:
error: can't expand macros compiled by previous versions of Scala
[ERROR]     assert(MathUtils.fpEquals(1e-6, 1e-7))
[ERROR]                              ^
...
[ERROR] 42 errors found
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] utils 0.2.14-SNAPSHOT .............................. SUCCESS [  2.389 s]
[INFO] utils-misc-spark2_2.11: Miscellaneous Spark helper code FAILURE [ 14.677 s]
[INFO] utils-minhash-spark2_2.11: MinHash-based Jaccard similarity estimator SKIPPED
[INFO] utils-metrics-spark2_2.11: tools for collecting metrics on top of RDDs SKIPPED
[INFO] utils-io-spark2_2.11: I/O utils .................... SKIPPED
[INFO] utils-statistics-spark2_2.11: Spark code for fitting statistical models SKIPPED
[INFO] utils-cli-spark2_2.11: utilities for building command line applications SKIPPED
[INFO] utils-serialization-spark2_2.11: tools for serializing data SKIPPED
[INFO] utils-intervalrdd-spark2_2.11: IntervalRDD based on interval tree 0.2.14-SNAPSHOT SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

Some dependency missed going 2.10 → 2.11 perhaps?

@heuermh
Copy link
Copy Markdown
Member

heuermh commented May 1, 2018

Created pull request fnothaft#1 that fixes the build issues.

@heuermh
Copy link
Copy Markdown
Member

heuermh commented Dec 3, 2018

Closing in favor of #125

@heuermh heuermh closed this Dec 3, 2018
@heuermh heuermh added this to the 0.2.14 milestone Dec 21, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants