New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ADAM-1909] Refactor pipe cmd parameter from String to Seq[String]. #1920

Merged
merged 2 commits into from Feb 19, 2018

Conversation

Projects
None yet
4 participants
@heuermh
Member

heuermh commented Feb 16, 2018

Fixes #1909.

@heuermh heuermh added this to the 0.24.0 milestone Feb 16, 2018

@coveralls

This comment has been minimized.

coveralls commented Feb 16, 2018

Coverage Status

Coverage decreased (-0.09%) to 82.581% when pulling db2136e on heuermh:issue-1909 into 67890b8 on bigdatagenomics:master.

@AmplabJenkins

This comment has been minimized.

AmplabJenkins commented Feb 16, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2670/

Build result: FAILURE

[...truncated 7 lines...] > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1920/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains e66609b # timeout=10Checking out Revision e66609b (origin/pr/1920/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f e66609b597563ecd44e67131129afa6a24f3e79cFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.6.2,2.10,2.2.1,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.1,centosADAM-prb ? 2.6.2,2.10,2.2.1,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.11,2.2.1,centos completed with result FAILUREADAM-prb ? 2.7.3,2.10,2.2.1,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.11,2.2.1,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@heuermh

This comment has been minimized.

Member

heuermh commented Feb 16, 2018

@fnothaft @pbashyal-nmdp Any help for an R novice?

Jenkins test failure above is due to:

R version 3.4.3 (2017-11-30) -- "Kite-Eating Tree"
Copyright (C) 2017 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin16.7.0 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> #
> # Licensed to Big Data Genomics (BDG) under one
> # or more contributor license agreements.  See the NOTICE file
> # distributed with this work for additional information
> # regarding copyright ownership.  The BDG licenses this file
> # to you under the Apache License, Version 2.0 (the
> # "License"); you may not use this file except in compliance
> # with the License.  You may obtain a copy of the License at
> #
> #     http://www.apache.org/licenses/LICENSE-2.0
> #
> # Unless required by applicable law or agreed to in writing, software
> # distributed under the License is distributed on an "AS IS" BASIS,
> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> # See the License for the specific language governing permissions and
> # limitations under the License.
> #
> library(testthat)
> library(bdgenomics.adam)

Attaching package: 'bdgenomics.adam'

The following objects are masked from 'package:base':

    pipe, save, sort, transform

> 
> test_check("bdgenomics.adam")

Attaching package: 'SparkR'

The following objects are masked from 'package:bdgenomics.adam':

    cache, persist, transform, unpersist

The following object is masked from 'package:testthat':

    describe

The following objects are masked from 'package:stats':

    cov, filter, lag, na.omit, predict, sd, var, window

The following objects are masked from 'package:base':

    as.data.frame, colnames, colnames<-, drop, endsWith, intersect,
    rank, rbind, sample, startsWith, subset, summary, transform, union

Launching java with spark-submit command spark-submit   --jars /Users/heuermh/working/adam/adam-assembly/target/adam-assembly-spark2_2.11-0.24.0-SNAPSHOT.jar --driver-class-path /Users/heuermh/working/adam/adam-assembly/target/adam-assembly-spark2_2.11-0.24.0-SNAPSHOT.jar sparkr-shell /var/folders/m6/4yqn_4q129lbth_dq3qzj_8h0000gn/T//RtmpLFXtCX/backend_port8fc26e2285b7 
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
2018-02-16 17:41:44 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-02-16 17:41:50 WARN  ObjectStore:6666 - Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
2018-02-16 17:41:50 WARN  ObjectStore:568 - Failed to get database default, returning NoSuchObjectException
2018-02-16 17:41:51 WARN  ObjectStore:568 - Failed to get database global_temp, returning NoSuchObjectException
2018-02-16 17:41:54 WARN  Utils:66 - Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.debug.maxToStringFields' in SparkEnv.conf.

[Stage 32:==============================================>       (171 + 4) / 200]
                                                                                
2018-02-16 17:42:02 WARN  RBackendHandler:66 - cannot find matching method class org.bdgenomics.adam.rdd.read.RDDBoundAlignmentRecordRDD.pipe. Candidates are:
2018-02-16 17:42:02 WARN  RBackendHandler:66 - pipe(interface scala.collection.Seq,interface scala.collection.Seq,interface scala.collection.immutable.Map,int,class scala.Option,interface org.bdgenomics.adam.rdd.InFormatterCompanion,interface org.bdgenomics.adam.rdd.OutFormatter,interface scala.Function2,interface scala.reflect.ClassTag,interface scala.reflect.ClassTag)
2018-02-16 17:42:02 WARN  RBackendHandler:66 - pipe(interface java.util.List,interface scala.collection.Seq,interface java.util.Map,class java.lang.Double,class java.lang.Class,interface org.bdgenomics.adam.rdd.OutFormatter,interface org.apache.spark.api.java.function.Function2)
2018-02-16 17:42:02 WARN  RBackendHandler:66 - pipe(interface java.util.List,interface java.util.List,interface java.util.Map,class java.lang.Integer,class java.lang.Class,interface org.bdgenomics.adam.rdd.OutFormatter,interface org.apache.spark.api.java.function.Function2)
2018-02-16 17:42:02 ERROR RBackendHandler:91 - pipe on 55 failed
java.lang.Exception: No matched method found for class org.bdgenomics.adam.rdd.read.RDDBoundAlignmentRecordRDD.pipe
	at org.apache.spark.api.r.RBackendHandler.handleMethodCall(RBackendHandler.scala:164)
	at org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:108)
	at org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:40)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:643)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:566)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:480)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
	at java.lang.Thread.run(Thread.java:745)
1. Error: pipe as sam (@test_alignmentRecordRdd.R#76) --------------------------

1: pipe(reads, cmd, "org.bdgenomics.adam.rdd.read.SAMInFormatter", "org.bdgenomics.adam.rdd.read.AnySAMOutFormatter", 
       "org.bdgenomics.adam.api.java.AlignmentRecordsToAlignmentRecordsConverter") at testthat/test_alignmentRecordRdd.R:76
2: pipe(reads, cmd, "org.bdgenomics.adam.rdd.read.SAMInFormatter", "org.bdgenomics.adam.rdd.read.AnySAMOutFormatter", 
       "org.bdgenomics.adam.api.java.AlignmentRecordsToAlignmentRecordsConverter")
3: .local(ardd, cmd, tFormatter, xFormatter, convFn, ...)
4: sparkR.callJMethod(ardd@jrdd, "pipe", cmd, files, environment, flankSize, tFormatterClass, 
       xFormatterInst, convFnInst)
5: callJMethod(x, methodName, ...)
6: invokeJava(isStatic = FALSE, objId$id, methodName, ...)
7: handleErrors(returnStatus, conn)
8: stop(readString(conn))

testthat results ================================================================
OK: 24 SKIPPED: 0 FAILED: 1
1. Error: pipe as sam (@test_alignmentRecordRdd.R#76) 

Error: testthat unit tests failed
Execution halted
@fnothaft

LGTM! Dropped a comment about the SparkR issues above. I'll merge once that's resolved.

@@ -646,7 +645,7 @@ trait GenomicRDD[T, U <: GenomicRDD[T, U]] extends Logging {
* @tparam V The InFormatter to use for formatting the data being piped to the
* command.
*/
def pipe[X, Y <: GenomicRDD[X, Y], V <: InFormatter[T, U, V]](cmd: String,
def pipe[X, Y <: GenomicRDD[X, Y], V <: InFormatter[T, U, V]](cmd: java.util.List[String],

This comment has been minimized.

@fnothaft

fnothaft Feb 18, 2018

Member

@heuermh to your R woes, SparkR uses homegrown R to JVM binding code which uses Scala types. If you change this from java.util.List[String] to Seq[Any] and then cast cmd to Seq[String] (as we do with the files parameter), you should be good.

This comment has been minimized.

@heuermh

heuermh Feb 19, 2018

Member

Thanks! Looks like this particular comment is on the wrong line; I made the fix you recommended and tests pass for me locally. Not sure why Jenkins is still having trouble.

@AmplabJenkins

This comment has been minimized.

AmplabJenkins commented Feb 19, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2675/

Build result: FAILURE

[...truncated 7 lines...] > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1920/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 70e2261 # timeout=10Checking out Revision 70e2261 (origin/pr/1920/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 70e2261f0e3ab4c97f84ffb2f182fa1f165c44d2First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.2,2.10,2.2.1,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.1,centosADAM-prb ? 2.6.2,2.10,2.2.1,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.11,2.2.1,centos completed with result FAILUREADAM-prb ? 2.7.3,2.10,2.2.1,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.11,2.2.1,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@AmplabJenkins

This comment has been minimized.

AmplabJenkins commented Feb 19, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2676/

Build result: FAILURE

[...truncated 7 lines...] > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1920/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 4bbfcc9 # timeout=10Checking out Revision 4bbfcc9 (origin/pr/1920/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 4bbfcc9c5f8fd60a81521d534ddccdd038962a47First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.2,2.10,2.2.1,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.1,centosADAM-prb ? 2.6.2,2.10,2.2.1,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.11,2.2.1,centos completed with result FAILUREADAM-prb ? 2.7.3,2.10,2.2.1,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.11,2.2.1,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@heuermh

This comment has been minimized.

Member

heuermh commented Feb 19, 2018

Jenkins, retest this please.

@AmplabJenkins

This comment has been minimized.

AmplabJenkins commented Feb 19, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2677/
Test PASSed.

@fnothaft fnothaft merged commit 26f0608 into bigdatagenomics:master Feb 19, 2018

1 of 2 checks passed

Codacy/PR Quality Review Not so good... This pull request quality could be better.
Details
default Merged build finished.
Details

@heuermh heuermh deleted the heuermh:issue-1909 branch Mar 1, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment