Make Spark SQL APIs supported across all types #1921

fnothaft · 2018-02-21T22:41:11Z

Resolves #1580, #1867, WIP towards #1728. Merges the GenomicRDD and GenomicDataset traits together, and makes the VariantContextRDD and GenericGenomicRDD implementations support GenomicDataset. Not done yet, but I wanted to get this out for review, as it's a big one. Remaining TODO:

Clear merge conflicts
Plumb join and GenericGenomicRDD APIs out to Python
Plumb join and GenericGenomicRDD APIs out to R
Resolve a serialization issue that occurs when using GenericGenomicRDD from the dataset API
Update user docs

One general stylistic question: this PR goes halfway to renaming everything "GenomicDataset" instead of "GenomicRDD". Is this a good change, or should we hold on to the GenomicRDD name for the sake of consistency?

coveralls · 2018-02-21T22:56:06Z

Coverage decreased (-1.8%) to 80.786% when pulling b75295e on fnothaft:issues/1867-1580-vc-sql into 26f0608 on bigdatagenomics:master.

coveralls · 2018-02-21T22:56:06Z

Coverage decreased (-1.8%) to 80.786% when pulling b75295e on fnothaft:issues/1867-1580-vc-sql into 26f0608 on bigdatagenomics:master.

coveralls · 2018-02-21T22:56:06Z

Coverage decreased (-2.8%) to 79.74% when pulling a26047a on fnothaft:issues/1867-1580-vc-sql into 9ea870c on bigdatagenomics:master.

AmplabJenkins · 2018-02-21T23:00:32Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2678/

Build result: FAILURE

[...truncated 7 lines...] > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse b75295e^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains b75295e # timeout=10Checking out Revision b75295e (origin/pr/1921/head) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f b75295e6d1fb816321a5f4bf5206b9541ca7cb8bFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.6.2,2.10,2.2.1,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.1,centosADAM-prb ? 2.6.2,2.10,2.2.1,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.11,2.2.1,centos completed with result FAILUREADAM-prb ? 2.7.3,2.10,2.2.1,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.11,2.2.1,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

heuermh · 2018-02-22T02:36:06Z

One general stylistic question: this PR goes halfway to renaming everything "GenomicDataset" instead of "GenomicRDD".

+1

heuermh

a few random review comments/questions, carry on!

heuermh · 2018-02-22T04:03:18Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMSaveAnyArgs.scala

+  /**
+   * If true and saving as FASTQ, we will sort by read name.
+   */
+  var sortFastqOutput: Boolean


is now a good time to rid the world of this?

Don't think so.

heuermh · 2018-02-22T04:03:18Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala

+   *
+   * @param pathName The path name to load genotypes from.
+   *   Globs/directories are supported.
+   * @return Returns a GenotypeRDD.


doc doesn't match

heuermh · 2018-02-22T04:03:18Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala

+    import sqlContext.implicits._
+    val ds = sqlContext.read.parquet(pathName).as[VariantContextProduct]
+
+    new DatasetBoundVariantContextRDD(ds, sd, samples, headers)


we're intentionally bound to dataset here, is the idea to move other loadParquetXxx methods in the same direction?

Nah, we don't have an Avro impl of VariantContext, so we can't use the normal loadParquet method.

heuermh · 2018-02-22T04:03:18Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/GenomicRDD.scala

+  val uTag: TypeTag[U]
+
+  /**
+   * This data as a Spark SQL Dataset.


These data?

heuermh · 2018-02-22T04:03:18Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/GenomicRDD.scala

+   *
+   * @param tFn A function that transforms the underlying RDD as a DataFrame.
+   * @return A new RDD where the RDD of genomic data has been replaced, but the
+   *   metadata (sequence dictionary, and etc) is copied without modification.


metadata ... are?

heuermh · 2018-02-22T04:03:18Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/GenomicRDD.scala

-    genomicRdd: GenomicRDD[X, Y],
-    flankSize: Long)(
+  def rightOuterBroadcastRegionJoin[X, Y <: Product, Z <: GenomicDataset[X, Y, Z]](
+    genomicRdd: GenomicDataset[X, Y, Z])(


genomicRdd → genomicDataset

How much do you want me to go through and patch these? For now, I've left them as is (genomicRdd). I'd rather make a cleanup pass through later.

ok, please create an issue to track consistency in names and doc for RDD vs Dataset

heuermh · 2018-02-22T04:03:18Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/GenomicRDD.scala

+  }
+
+  /**
+   * Performs a broadcast inner join between this RDD and another RDD.


RDD → Dataset

heuermh · 2018-02-22T04:03:18Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/GenomicRDD.scala

-      regionFn)
+  @transient val uTag: TypeTag[U] = typeTag[U]
+
+  def saveAsParquet(filePath: String,


does this one need an override?

heuermh · 2018-02-22T04:03:18Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/GenomicRDD.scala

+    pageSize: Int = 1 * 1024 * 1024,
+    compressCodec: CompressionCodecName = CompressionCodecName.GZIP,
+    disableDictionaryEncoding: Boolean = false,
+    optSchema: Option[Schema] = None): Unit = SaveAsADAM.time {


would be nice to have separate SaveRddAsParquet and SaveDatasetAsParquet timers

We're not really going to get meaningful timing info out of the SaveDatasetAsParquet timer.

heuermh · 2018-02-22T04:03:18Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/variant/GenotypeRDD.scala

@@ -210,7 +210,10 @@ case class RDDBoundGenotypeRDD private[rdd] (
  }
 }

-sealed abstract class GenotypeRDD extends MultisampleAvroGenomicRDD[Genotype, GenotypeProduct, GenotypeRDD] {
+sealed abstract class GenotypeRDD extends MultisampleAvroGenomicDataset[Genotype, GenotypeProduct, GenotypeRDD] {


will you be proposing to move this to GenotypeDataset?

AmplabJenkins · 2018-03-07T08:46:33Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2702/

Build result: FAILURE

[...truncated 7 lines...] > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1921/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains c8dd89d # timeout=10Checking out Revision c8dd89d (origin/pr/1921/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f c8dd89d25204e4dd7fa729ef22d01b3a1aa9d280First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.2,2.10,2.2.1,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.1,centosADAM-prb ? 2.6.2,2.10,2.2.1,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.11,2.2.1,centos completed with result FAILUREADAM-prb ? 2.7.3,2.10,2.2.1,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.11,2.2.1,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

fnothaft · 2018-03-07T09:20:29Z

Jenkins, retest this please.

AmplabJenkins · 2018-03-07T09:50:59Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2704/
Test PASSed.

fnothaft · 2018-03-07T10:22:05Z

@heuermh I think everything in Scala is going to be stable from here on out; can you make a review pass? I'm going to finish up the R bit in the morning.

AmplabJenkins · 2018-03-07T10:51:42Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2705/
Test PASSed.

AmplabJenkins · 2018-03-07T18:51:30Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2706/
Test PASSed.

AmplabJenkins · 2018-03-07T21:35:37Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2710/
Test PASSed.

AmplabJenkins · 2018-03-07T22:04:05Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2711/
Test PASSed.

heuermh · 2018-03-08T21:36:34Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/GenomicRDD.scala

-                                    tManifest: ClassTag[T],
-                                    xManifest: ClassTag[X]): Y = {
+  def pipe[X, Y <: Product, Z <: GenomicDataset[X, Y, Z], W <: InFormatter[T, U, V, W]](cmd: Seq[String],
+                                                                                        files: Seq[String] = Seq.empty,


nit: reformat as above for line length. Same with other pipe methods

heuermh · 2018-03-08T21:36:34Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/GenomicRDD.scala

+    implicit val txTag = ClassTag.AnyRef.asInstanceOf[ClassTag[(Option[T], X)]]
+    implicit val u1Tag: TypeTag[U] = uTag
+    implicit val u2Tag: TypeTag[Y] = genomicRdd.uTag
+    implicit val uyTag = typeTag[(Option[U], Y)]


hope I never need to touch this method signature ;)

heuermh · 2018-03-08T21:36:34Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/GenomicRDD.scala

-    genomicRdd: GenomicRDD[X, Y],
-    flankSize: Long)(
+  def rightOuterBroadcastRegionJoin[X, Y <: Product, Z <: GenomicDataset[X, Y, Z]](
+    genomicRdd: GenomicDataset[X, Y, Z])(


ok, please create an issue to track consistency in names and doc for RDD vs Dataset

heuermh · 2018-03-08T21:36:34Z

scripts/jenkins-test

 	    mvn -U \
 		-P python,r \
 		test \
+		-Dsuites=select.no.suites\* \


is this to ignore the scala tests while still running the Python and R ones?

antonkulaga · 2018-03-08T23:39:01Z

Is this a good change, or should we hold on to the GenomicRDD name for the sake of consistency?

Do not know, for my ADAM-based code/libs renaming is not a problem.. What I noticed with the previous version was that in many cases using genomic RDDs was faster than trying to use spark sql.

Resolves #1867.

Resolves #1580. Does not compile due to abstract classes not implemented.

Resolves #1728.

AmplabJenkins · 2018-03-15T21:38:30Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2729/
Test PASSed.

heuermh · 2018-03-20T23:58:14Z

Thank you, @fnothaft

fnothaft requested review from heuermh and akmorrow13 Feb 21, 2018

heuermh requested changes Feb 22, 2018

View changes

fnothaft added this to the 0.24.0 milestone Mar 5, 2018

fnothaft force-pushed the fnothaft:issues/1867-1580-vc-sql branch from b75295e to 6d363e7 Mar 7, 2018

fnothaft force-pushed the fnothaft:issues/1867-1580-vc-sql branch 2 times, most recently from 6184711 to a26047a Mar 7, 2018

fnothaft mentioned this pull request Mar 7, 2018

Surface VariantContextRDD loaders in Python and R #1945

Open

heuermh requested changes Mar 8, 2018

View changes

Frank Austin Nothaft added 4 commits Jan 24, 2018

[ADAM-1867] Add support for processing VariantContexts from SQL.

a95e811

Resolves #1867.

[ADAM-1580] Eliminate ADAMRDDFunctions classes.

1c7f10e

Resolves #1580. Does not compile due to abstract classes not implemented.

Collapse GenomicRDD trait into GenomicDataset.

845175f

[ADAM-1728] Expose region join API through Python and R.

Loading status checks…

40043e5

Resolves #1728.

fnothaft force-pushed the fnothaft:issues/1867-1580-vc-sql branch from a26047a to 40043e5 Mar 15, 2018

fnothaft mentioned this pull request Mar 15, 2018

Clarify GenomicRDD vs. GenomicDataset name #1954

Closed

heuermh approved these changes Mar 15, 2018

View changes

heuermh merged commit a28da44 into bigdatagenomics:master Mar 20, 2018
1 of 2 checks passed

1 of 2 checks passed

Codacy/PR Quality Review Not so good... This pull request quality could be better.
Details

default Merged build finished.
Details

heuermh mentioned this pull request Mar 21, 2018

Inconsistent counts for shuffleRegionJoin without Sequence Dictionaries #1956

Open

bigdatagenomics / adam

Make Spark SQL APIs supported across all types #1921

Make Spark SQL APIs supported across all types #1921

Conversation

fnothaft commented Feb 21, 2018 • edited

coveralls commented Feb 21, 2018

coveralls commented Feb 21, 2018

coveralls commented Feb 21, 2018 • edited

AmplabJenkins commented Feb 21, 2018

Build result: FAILURE

heuermh commented Feb 22, 2018

heuermh left a comment

This comment has been minimized.

heuermh Feb 22, 2018 Member

This comment has been minimized.

fnothaft Mar 7, 2018 Author Member

This comment has been minimized.

heuermh Mar 8, 2018 Member

This comment has been minimized.

heuermh Feb 22, 2018 Member

This comment has been minimized.

heuermh Feb 22, 2018 Member

This comment has been minimized.

fnothaft Mar 7, 2018 Author Member

This comment has been minimized.

heuermh Mar 8, 2018 Member

This comment has been minimized.

heuermh Feb 22, 2018 Member

This comment has been minimized.

heuermh Feb 22, 2018 Member

This comment has been minimized.

heuermh Feb 22, 2018 Member

This comment has been minimized.

fnothaft Mar 7, 2018 Author Member

This comment has been minimized.

heuermh Mar 8, 2018 Member

This comment has been minimized.

heuermh Feb 22, 2018 Member

This comment has been minimized.

heuermh Feb 22, 2018 Member

This comment has been minimized.

heuermh Feb 22, 2018 Member

This comment has been minimized.

fnothaft Mar 7, 2018 Author Member

This comment has been minimized.

heuermh Mar 8, 2018 Member

This comment has been minimized.

heuermh Feb 22, 2018 Member

AmplabJenkins commented Mar 7, 2018

Build result: FAILURE

fnothaft commented Mar 7, 2018

AmplabJenkins commented Mar 7, 2018

fnothaft commented Mar 7, 2018

AmplabJenkins commented Mar 7, 2018

AmplabJenkins commented Mar 7, 2018

AmplabJenkins commented Mar 7, 2018

AmplabJenkins commented Mar 7, 2018

This comment has been minimized.

heuermh Mar 8, 2018 Member

This comment has been minimized.

heuermh Mar 8, 2018 Member

This comment has been minimized.

heuermh Mar 8, 2018 Member

This comment has been minimized.

heuermh Mar 8, 2018 Member

antonkulaga commented Mar 8, 2018

AmplabJenkins commented Mar 15, 2018

heuermh commented Mar 20, 2018

fnothaft commented Feb 21, 2018 •

edited

coveralls commented Feb 21, 2018 •

edited

heuermh Feb 22, 2018
Member

fnothaft Mar 7, 2018
Author Member

heuermh Mar 8, 2018
Member

heuermh Feb 22, 2018
Member

heuermh Feb 22, 2018
Member

fnothaft Mar 7, 2018
Author Member

heuermh Mar 8, 2018
Member

heuermh Feb 22, 2018
Member

heuermh Feb 22, 2018
Member

heuermh Feb 22, 2018
Member

fnothaft Mar 7, 2018
Author Member

heuermh Mar 8, 2018
Member

heuermh Feb 22, 2018
Member

heuermh Feb 22, 2018
Member

heuermh Feb 22, 2018
Member

fnothaft Mar 7, 2018
Author Member

heuermh Mar 8, 2018
Member

heuermh Feb 22, 2018
Member

heuermh Mar 8, 2018
Member

heuermh Mar 8, 2018
Member

heuermh Mar 8, 2018
Member

heuermh Mar 8, 2018
Member