Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ADAM-651] Implement Hive-style partitioning by genomic range of Parquet backed datasets #1948

Merged
merged 1 commit into from Mar 14, 2018

Conversation

Projects
None yet
6 participants
@fnothaft
Copy link
Member

fnothaft commented Mar 7, 2018

Fixes #651.

Continuation of #1922.

@fnothaft fnothaft added this to the 0.24.0 milestone Mar 7, 2018

@fnothaft fnothaft requested a review from heuermh Mar 7, 2018

@AmplabJenkins

This comment has been minimized.

Copy link

AmplabJenkins commented Mar 8, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2714/
Test PASSed.

@heuermh
Copy link
Member

heuermh left a comment

Minor typos/nits. Note the Github UI doesn't seem to be showing all the comments here correctly due to the new Load more... function. They all look to be there in the Files changed tab.

@@ -131,6 +131,10 @@ class TransformAlignmentsArgs extends Args4jBase with ADAMSaveAnyArgs with Parqu
var storageLevel: String = "MEMORY_ONLY"
@Args4jOption(required = false, name = "-disable_pg", usage = "Disable writing a new @PG line.")
var disableProcessingStep = false
@Args4jOption(required = false, name = "-partition_by_start_pos", usage = "Save the data partitioned by genomic range bins based on start pos using Hive-style partitioning.")
var partitionByStartPos: Boolean = false
@Args4jOption(required = false, name = "-partition_bin_size", usage = "Partition bin sized used in Hive-style partitioning.")

This comment has been minimized.

Copy link
@heuermh

heuermh Mar 8, 2018

Member

add Defaults to 1Mbp (1,000,000 base pairs). to usage

This comment has been minimized.

Copy link
@jpdna

jpdna Mar 9, 2018

Member

fixed

@Args4jOption(required = false, name = "-partition_by_start_pos", usage = "Save the data partitioned by genomic range bins based on start pos using Hive-style partitioning.")
var partitionByStartPos: Boolean = false

@Args4jOption(required = false, name = "-partition_bin_size", usage = "Partition bin sized used in Hive-style partitioning.")

This comment has been minimized.

Copy link
@heuermh

heuermh Mar 8, 2018

Member

add Defaults to 1Mbp (1,000,000 base pairs). to usage

This comment has been minimized.

Copy link
@jpdna

jpdna Mar 9, 2018

Member

fixed

@@ -3052,4 +3192,44 @@ class ADAMContext(@transient val sc: SparkContext) extends Serializable with Log
loadParquetFragments(pathName, optPredicate = optPredicate, optProjection = optProjection)
}
}

/**
* Return integer size of partitions if the specified path of Parquet + Avro files is partitioned.

This comment has been minimized.

Copy link
@heuermh

heuermh Mar 8, 2018

Member

Didn't you have a better suggestion for this doc previously? Return length of partitions in base pairs... perhaps?

This comment has been minimized.

Copy link
@jpdna

jpdna Mar 9, 2018

Member

fixed

*
* @param pathName Path in which to look for partitioned flag.
* @return Return true if the specified path of Parquet + Avro files is partitioned.
* Behavior is undefined if some paths in glob are contain paritioned flag and some do not.

This comment has been minimized.

Copy link
@heuermh

heuermh Mar 8, 2018

Member

paritioned → partitioned

This comment has been minimized.

Copy link
@jpdna

jpdna Mar 9, 2018

Member

fixed

val optPartitionBinSize: Option[Int]
val optLookbackPartitions: Option[Int]

private def referenceRegionsToDatasetQueryString(regions: Iterable[ReferenceRegion], partitionSize: Int = 1000000, partitionedLookBackNum: Int = 1): String = {

This comment has been minimized.

Copy link
@heuermh

heuermh Mar 8, 2018

Member

shouldn't this method use the vals optPartitionBinSize and optLookbackPartitions above?

This comment has been minimized.

Copy link
@jpdna

jpdna Mar 9, 2018

Member

eeek, yes you are correct.

This comment has been minimized.

Copy link
@jpdna

jpdna Mar 9, 2018

Member

Well actually, it was working correctly before because the optPartitionedBinSize was what was passed as a parameter when called fro filterByoverlappingRegions but given that both functions are in the same trait, no no need for the parameter.

This comment has been minimized.

Copy link
@jpdna

jpdna Mar 9, 2018

Member

fixed

The existence of the file ``_partitionedByStartPos`` can be tested with the public
function ``ADAMContext.isPartitioned(path)`` and can be used to determine explicitly
if an ADAM Parquet dataset is partitioned using this scheme. The parition size used
when the dataset was written to disk is stored in ``_paritionedByStartPos`` and is

This comment has been minimized.

Copy link
@heuermh

heuermh Mar 8, 2018

Member

paritionedByStartPos → partitionedByStartPos

This comment has been minimized.

Copy link
@jpdna

jpdna Mar 9, 2018

Member

fixed


The Spark dataset API recognizes that the field ``positionBin`` is defined implicitly
by the Parquet files' paritioning scheme, and makes ``positionBin`` available as a field
that can be queried through the Spark SQL API. ``positionBin`` is used interally by

This comment has been minimized.

Copy link
@heuermh

heuermh Mar 8, 2018

Member

interally → internally

This comment has been minimized.

Copy link
@jpdna

jpdna Mar 9, 2018

Member

fixed

The Spark dataset API recognizes that the field ``positionBin`` is defined implicitly
by the Parquet files' paritioning scheme, and makes ``positionBin`` available as a field
that can be queried through the Spark SQL API. ``positionBin`` is used interally by
the public function ``GenomicRDD.filterByOverlappinRegions``. User code in ADAM-shell or

This comment has been minimized.

Copy link
@heuermh

heuermh Mar 8, 2018

Member

filterByOverlappinRegions → filterByOverlappingRegions

This comment has been minimized.

Copy link
@jpdna

jpdna Mar 9, 2018

Member

fixed

by the Parquet files' paritioning scheme, and makes ``positionBin`` available as a field
that can be queried through the Spark SQL API. ``positionBin`` is used interally by
the public function ``GenomicRDD.filterByOverlappinRegions``. User code in ADAM-shell or
a user applcations could similarly utilize the ``positionBin`` field when creating SPARK

This comment has been minimized.

Copy link
@heuermh

heuermh Mar 8, 2018

Member

a user applcations → user applications
SPARK → Spark

This comment has been minimized.

Copy link
@jpdna

jpdna Mar 9, 2018

Member

fixed

val mydata = loadPartitionedParquetAlignments("alignmets.adam")
val filteredCount1 = mydata.filterByOverlappingRegions(regions1).dataset.count
val filteredCount2 = mydata.filterByOverlappingRegions(regions2).dataset.count

This comment has been minimized.

Copy link
@heuermh

heuermh Mar 8, 2018

Member

remove blank lines

This comment has been minimized.

Copy link
@jpdna

jpdna Mar 9, 2018

Member

fixed

@coveralls

This comment has been minimized.

Copy link

coveralls commented Mar 8, 2018

Coverage Status

Coverage decreased (-0.07%) to 82.465% when pulling f745bca on jpdna-hive_partitioned_v11 into 9ea870c on master.

@AmplabJenkins

This comment has been minimized.

Copy link

AmplabJenkins commented Mar 8, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2716/

Build result: ABORTED

[...truncated 7 lines...] > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1948/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains f1d6be1 # timeout=10Checking out Revision f1d6be1 (origin/pr/1948/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f f1d6be122ce302eda8ea2c4d88fc1644e1377ef5First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.2,2.10,2.2.1,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.1,centosADAM-prb ? 2.6.2,2.10,2.2.1,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.11,2.2.1,centos completed with result ABORTEDADAM-prb ? 2.7.3,2.10,2.2.1,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.11,2.2.1,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@fnothaft

This comment has been minimized.

Copy link
Member Author

fnothaft commented Mar 8, 2018

Jenkins, retest this please.

@AmplabJenkins

This comment has been minimized.

Copy link

AmplabJenkins commented Mar 8, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2717/
Test PASSed.

The existence of the file ``_partitionedByStartPos`` can be tested with the public
function ``ADAMContext.isPartitioned(path)`` and can be used to determine explicitly
if an ADAM Parquet dataset is partitioned using this scheme. The parition size used
when the dataset was written to disk is stored in ``_parititionedByStartPos`` and is

This comment has been minimized.

Copy link
@heuermh

heuermh Mar 8, 2018

Member

parititionedByStartPos → partitionedByStartPos

This comment has been minimized.

Copy link
@jpdna

jpdna Mar 9, 2018

Member

fixed -thanks, I honestly need to increase my font size so I can see typos.

@AmplabJenkins

This comment has been minimized.

Copy link

AmplabJenkins commented Mar 9, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2718/
Test PASSed.

@AmplabJenkins

This comment has been minimized.

Copy link

AmplabJenkins commented Mar 9, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2719/
Test PASSed.

@AmplabJenkins

This comment has been minimized.

Copy link

AmplabJenkins commented Mar 9, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2720/
Test PASSed.

@AmplabJenkins

This comment has been minimized.

Copy link

AmplabJenkins commented Mar 9, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2721/
Test PASSed.

@AmplabJenkins

This comment has been minimized.

Copy link

AmplabJenkins commented Mar 9, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2722/
Test PASSed.

@AmplabJenkins

This comment has been minimized.

Copy link

AmplabJenkins commented Mar 9, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2723/

Build result: FAILURE

[...truncated 7 lines...] > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1948/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains b589bfc # timeout=10Checking out Revision b589bfc (origin/pr/1948/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f b589bfc > /home/jenkins/git2/bin/git rev-list 0367f44 # timeout=10Triggering ADAM-prb ? 2.6.2,2.10,2.2.1,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.1,centosADAM-prb ? 2.6.2,2.10,2.2.1,centos completed with result FAILUREADAM-prb ? 2.6.2,2.11,2.2.1,centos completed with result FAILUREADAM-prb ? 2.7.3,2.10,2.2.1,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,2.2.1,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@jpdna

This comment has been minimized.

Copy link
Member

jpdna commented Mar 9, 2018

I'm getting an error when I'm testing this downstream in Mango, I'll be looking into it later this evening.

@jpdna

This comment has been minimized.

Copy link
Member

jpdna commented Mar 10, 2018

Resolved problem I had with using in Mango.

This is ready for another review pass.

@jpdna jpdna referenced this pull request Mar 10, 2018

Closed

Use Hive-style partitioning #370

@AmplabJenkins

This comment has been minimized.

Copy link

AmplabJenkins commented Mar 10, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2724/

Build result: FAILURE

[...truncated 7 lines...] > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1948/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 14d75a9 # timeout=10Checking out Revision 14d75a9 (origin/pr/1948/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 14d75a9 > /home/jenkins/git2/bin/git rev-list b589bfc # timeout=10Triggering ADAM-prb ? 2.6.2,2.10,2.2.1,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.1,centosADAM-prb ? 2.6.2,2.10,2.2.1,centos completed with result FAILUREADAM-prb ? 2.6.2,2.11,2.2.1,centos completed with result FAILUREADAM-prb ? 2.7.3,2.10,2.2.1,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,2.2.1,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@jpdna

This comment has been minimized.

Copy link
Member

jpdna commented Mar 14, 2018

Ping for review here.


require(partitionSizes.nonEmpty, "Input Parquet files (%s) are not partitioned.".format(pathName))
require(partitionSizes.size == 1, "Found multiple partition sizes (%s).".format(partitionSizes.mkString(", ")))
partitionSizes.head

This comment has been minimized.

Copy link
@akmorrow13

akmorrow13 Mar 14, 2018

Contributor

If partitionSizes is empty this will throw java.util.NoSuchElementException: next on empty iterator

This comment has been minimized.

Copy link
@jpdna

jpdna Mar 14, 2018

Member

But on like 3219 doesn't the require check for partitionSizes being empty and produce the intended message when the require condition is not met?

This comment has been minimized.

Copy link
@fnothaft

fnothaft Mar 14, 2018

Author Member

But on like 3219 doesn't the require check for partitionSizes being empty and produce the intended message when the require condition is not met?

Correct, you're fine here.

@heuermh heuermh changed the title Implement Hive-style partitioning by genomic range of Parquet backed datasets [ADAM-651] Implement Hive-style partitioning by genomic range of Parquet backed datasets Mar 14, 2018

@heuermh

This comment has been minimized.

Copy link
Member

heuermh commented Mar 14, 2018

Jenkins, retest this please.

@AmplabJenkins

This comment has been minimized.

Copy link

AmplabJenkins commented Mar 14, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2725/

Build result: FAILURE

[...truncated 7 lines...] > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1948/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 14d75a9 # timeout=10Checking out Revision 14d75a9 (origin/pr/1948/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 14d75a9 > /home/jenkins/git2/bin/git rev-list 14d75a9 # timeout=10Triggering ADAM-prb ? 2.6.2,2.10,2.2.1,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.1,centosADAM-prb ? 2.6.2,2.10,2.2.1,centos completed with result FAILUREADAM-prb ? 2.6.2,2.11,2.2.1,centos completed with result FAILUREADAM-prb ? 2.7.3,2.10,2.2.1,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,2.2.1,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@heuermh

This comment has been minimized.

Copy link
Member

heuermh commented Mar 14, 2018

Looks like there is a code style issue in GenomicRDD.scala

++ git status --porcelain
+ test -n ' M adam-core/src/main/scala/org/bdgenomics/adam/rdd/GenomicRDD.scala'
@jpdna

This comment has been minimized.

Copy link
Member

jpdna commented Mar 14, 2018

@heuermh - you online? I'm not sure how to get details about and how to resolve this code style issue.

@heuermh

This comment has been minimized.

Copy link
Member

heuermh commented Mar 14, 2018

If you run the maven build from the command line, git will tell you if a file changed due to the formatter that is run as part of the build

@fnothaft

This comment has been minimized.

Copy link
Member Author

fnothaft commented Mar 14, 2018

If you run the maven build from the command line, git will tell you if a file changed due to the formatter that is run as part of the build

If this doesn't show any changes, run ./scripts/format-sources.

partitionSizes is checked for nonEmpty before head is called.

@fnothaft

This comment has been minimized.

Copy link
Member Author

fnothaft commented Mar 14, 2018

@jpdna don't worry about it; I'll clean the source formatting issue.

@fnothaft fnothaft force-pushed the jpdna-hive_partitioned_v11 branch from f745bca to 452d760 Mar 14, 2018

@AmplabJenkins

This comment has been minimized.

Copy link

AmplabJenkins commented Mar 14, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2726/

Build result: FAILURE

[...truncated 7 lines...] > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1948/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains f461abf # timeout=10Checking out Revision f461abf (origin/pr/1948/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f f461abf > /home/jenkins/git2/bin/git rev-list 14d75a9 # timeout=10Triggering ADAM-prb ? 2.6.2,2.10,2.2.1,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.1,centosADAM-prb ? 2.6.2,2.10,2.2.1,centos completed with result FAILUREADAM-prb ? 2.6.2,2.11,2.2.1,centos completed with result FAILUREADAM-prb ? 2.7.3,2.10,2.2.1,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,2.2.1,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@fnothaft

This comment has been minimized.

Copy link
Member Author

fnothaft commented Mar 14, 2018

Jenkins, test this please.

@AmplabJenkins

This comment has been minimized.

Copy link

AmplabJenkins commented Mar 14, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2727/

Build result: ABORTED

[...truncated 7 lines...] > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1948/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains f461abf # timeout=10Checking out Revision f461abf (origin/pr/1948/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f f461abf > /home/jenkins/git2/bin/git rev-list f461abf # timeout=10Triggering ADAM-prb ? 2.6.2,2.10,2.2.1,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.1,centosADAM-prb ? 2.6.2,2.10,2.2.1,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.11,2.2.1,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,2.2.1,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.11,2.2.1,centos completed with result ABORTEDNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@fnothaft

This comment has been minimized.

Copy link
Member Author

fnothaft commented Mar 14, 2018

Jenkins, test this please.

@AmplabJenkins

This comment has been minimized.

Copy link

AmplabJenkins commented Mar 14, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2728/
Test PASSed.

@heuermh heuermh merged commit c54f1a1 into master Mar 14, 2018

1 of 2 checks passed

Codacy/PR Quality Review Not so good... This pull request quality could be better.
Details
default Merged build finished.
Details
@heuermh

This comment has been minimized.

Copy link
Member

heuermh commented Mar 14, 2018

Thank you, @jpdna @fnothaft!

@fnothaft fnothaft deleted the jpdna-hive_partitioned_v11 branch Mar 14, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.