Add -print_metrics option to Jenkins build #947

Merged
merged 1 commit into from Apr 7, 2016

Conversation

Projects
None yet
4 participants
@heuermh
Member

heuermh commented Feb 16, 2016

I expect this may fail.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Feb 16, 2016

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1084/

Build result: FAILURE

[...truncated 32 lines...]Triggering ADAM-prb ? 2.3.0,2.10,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 1.0.4,2.11,1.2.1,centosTriggering ADAM-prb ? 2.3.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.3.0,2.11,1.3.1,centosADAM-prb ? 2.3.0,2.10,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.3.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.6.0,centos completed with result SUCCESSADAM-prb ? 1.0.4,2.10,1.2.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.6.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.5.2,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.2.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.6.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,1.2.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.6.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.5.2,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.5.2,centos completed with result SUCCESSADAM-prb ? 1.0.4,2.11,1.2.1,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,1.5.2,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,1.3.1,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1084/

Build result: FAILURE

[...truncated 32 lines...]Triggering ADAM-prb ? 2.3.0,2.10,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 1.0.4,2.11,1.2.1,centosTriggering ADAM-prb ? 2.3.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.3.0,2.11,1.3.1,centosADAM-prb ? 2.3.0,2.10,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.3.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.6.0,centos completed with result SUCCESSADAM-prb ? 1.0.4,2.10,1.2.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.6.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.5.2,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.2.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.6.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,1.2.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.6.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.5.2,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.5.2,centos completed with result SUCCESSADAM-prb ? 1.0.4,2.11,1.2.1,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,1.5.2,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,1.3.1,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Feb 17, 2016

Member

Not exactly what I would've expected.

Looks like there is an unrelated Spark 1.5.2 build --> 1.2.1 runtime incompatibility

java.lang.NoClassDefFoundError: org/apache/spark/rdd/RDD$
  at org.apache.spark.rdd.InstrumentedOrderedRDDFunctions
    .sortByKey(InstrumentedOrderedRDDFunctions.scala:35)
  at org.bdgenomics.adam.rdd.read.AlignmentRecordRDDFunctions$$anonfun
    $adamSortReadsByReferencePosition$1.apply(AlignmentRecordRDDFunctions.scala:662)

and inconsistent failures elsewhere due to execution problems

java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext.
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1, localhost): java.io.IOException: org.apache.spark.SparkException: Failed to get broadcast_0_piece0 of broadcast_0
    at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1178)
    at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165)

Perhaps calling ADAMMain.main(...) isn't compatible with our test suite, although it works for me locally.

Member

heuermh commented Feb 17, 2016

Not exactly what I would've expected.

Looks like there is an unrelated Spark 1.5.2 build --> 1.2.1 runtime incompatibility

java.lang.NoClassDefFoundError: org/apache/spark/rdd/RDD$
  at org.apache.spark.rdd.InstrumentedOrderedRDDFunctions
    .sortByKey(InstrumentedOrderedRDDFunctions.scala:35)
  at org.bdgenomics.adam.rdd.read.AlignmentRecordRDDFunctions$$anonfun
    $adamSortReadsByReferencePosition$1.apply(AlignmentRecordRDDFunctions.scala:662)

and inconsistent failures elsewhere due to execution problems

java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext.
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1, localhost): java.io.IOException: org.apache.spark.SparkException: Failed to get broadcast_0_piece0 of broadcast_0
    at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1178)
    at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165)

Perhaps calling ADAMMain.main(...) isn't compatible with our test suite, although it works for me locally.

@heuermh heuermh modified the milestone: 0.20.0 Feb 24, 2016

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Mar 22, 2016

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1109/

Build result: FAILURE

[...truncated 24 lines...]Triggering ADAM-prb ? 2.6.0,2.10,1.3.1,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.0,centosTriggering ADAM-prb ? 2.6.0,2.10,1.4.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.3.1,centosTriggering ADAM-prb ? 2.3.0,2.10,1.4.1,centosTriggering ADAM-prb ? 2.3.0,2.10,1.3.1,centosTriggering ADAM-prb ? 2.3.0,2.11,1.4.1,centosTriggering ADAM-prb ? 2.3.0,2.11,1.3.1,centosTriggering ADAM-prb ? 2.3.0,2.10,1.5.2,centosADAM-prb ? 2.6.0,2.11,1.6.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.5.2,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.6.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.6.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.3.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.6.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.3.1,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.5.2,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1109/

Build result: FAILURE

[...truncated 24 lines...]Triggering ADAM-prb ? 2.6.0,2.10,1.3.1,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.0,centosTriggering ADAM-prb ? 2.6.0,2.10,1.4.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.3.1,centosTriggering ADAM-prb ? 2.3.0,2.10,1.4.1,centosTriggering ADAM-prb ? 2.3.0,2.10,1.3.1,centosTriggering ADAM-prb ? 2.3.0,2.11,1.4.1,centosTriggering ADAM-prb ? 2.3.0,2.11,1.3.1,centosTriggering ADAM-prb ? 2.3.0,2.10,1.5.2,centosADAM-prb ? 2.6.0,2.11,1.6.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.5.2,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.6.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.6.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.3.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.6.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.3.1,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.5.2,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Mar 22, 2016

Member

@fnothaft @massie I can't tell what's going wrong here.

I am not able to reproduce these SparkException: Failed to get broadcast_0_piece0 of broadcast_0 exceptions when running scripts/jenkins-test locally.

Member

heuermh commented Mar 22, 2016

@fnothaft @massie I can't tell what's going wrong here.

I am not able to reproduce these SparkException: Failed to get broadcast_0_piece0 of broadcast_0 exceptions when running scripts/jenkins-test locally.

@massie

This comment has been minimized.

Show comment
Hide comment
@massie

massie Mar 22, 2016

Member

The underlying issue seems to be having multiple Spark contexts running in a single JVM. It's possible that Jenkins is running the tests differently than when you run them locally (e.g. in a single JVM vs. threaded).

Member

massie commented Mar 22, 2016

The underlying issue seems to be having multiple Spark contexts running in a single JVM. It's possible that Jenkins is running the tests differently than when you run them locally (e.g. in a single JVM vs. threaded).

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Mar 22, 2016

Member

It must be problem with the unit tests then.
I'll have to find a way to exercise -print_metrics without going through ADAMMain.main(...).

Member

heuermh commented Mar 22, 2016

It must be problem with the unit tests then.
I'll have to find a way to exercise -print_metrics without going through ADAMMain.main(...).

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Apr 6, 2016

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1140/

Build result: FAILURE

GitHub pull request #947 of commit 47dfa4d automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/947/merge^{commit} # timeout=10 > git branch -a --contains 6c7b2f01821bb586bd916fc3369d37246f31c4f5 # timeout=10 > git rev-parse remotes/origin/pr/947/merge^{commit} # timeout=10Checking out Revision 6c7b2f01821bb586bd916fc3369d37246f31c4f5 (origin/pr/947/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f 6c7b2f01821bb586bd916fc3369d37246f31c4f5First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1140/

Build result: FAILURE

GitHub pull request #947 of commit 47dfa4d automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/947/merge^{commit} # timeout=10 > git branch -a --contains 6c7b2f01821bb586bd916fc3369d37246f31c4f5 # timeout=10 > git rev-parse remotes/origin/pr/947/merge^{commit} # timeout=10Checking out Revision 6c7b2f01821bb586bd916fc3369d37246f31c4f5 (origin/pr/947/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f 6c7b2f01821bb586bd916fc3369d37246f31c4f5First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Apr 7, 2016

Member

Jenkins, retest this please.

Member

heuermh commented Apr 7, 2016

Jenkins, retest this please.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Apr 7, 2016

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1143/

Build result: FAILURE

GitHub pull request #947 of commit 47dfa4d automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/947/merge^{commit} # timeout=10 > git branch -a --contains 6c7b2f01821bb586bd916fc3369d37246f31c4f5 # timeout=10 > git rev-parse remotes/origin/pr/947/merge^{commit} # timeout=10Checking out Revision 6c7b2f01821bb586bd916fc3369d37246f31c4f5 (origin/pr/947/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f 6c7b2f01821bb586bd916fc3369d37246f31c4f5First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1143/

Build result: FAILURE

GitHub pull request #947 of commit 47dfa4d automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/947/merge^{commit} # timeout=10 > git branch -a --contains 6c7b2f01821bb586bd916fc3369d37246f31c4f5 # timeout=10 > git rev-parse remotes/origin/pr/947/merge^{commit} # timeout=10Checking out Revision 6c7b2f01821bb586bd916fc3369d37246f31c4f5 (origin/pr/947/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f 6c7b2f01821bb586bd916fc3369d37246f31c4f5First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Apr 7, 2016

Member

Jenkins, retest this please.

Member

heuermh commented Apr 7, 2016

Jenkins, retest this please.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Apr 7, 2016

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1145/
Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1145/
Test PASSed.

@heuermh heuermh changed the title from Investigating -print_metrics via Jenkins build matrix, do not merge to Add -print_metrics option to Jenkins build Apr 7, 2016

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Apr 7, 2016

Member

Ready to merge, -print-metrics works for me locally across our current build matrix.

Fixes #902, even though the code change happened in merged pull request #961.

Member

heuermh commented Apr 7, 2016

Ready to merge, -print-metrics works for me locally across our current build matrix.

Fixes #902, even though the code change happened in merged pull request #961.

@fnothaft fnothaft merged commit 47dfa4d into bigdatagenomics:master Apr 7, 2016

1 check passed

default Merged build finished.
Details
@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Apr 7, 2016

Member

Thanks @heuermh! Merged.

Member

fnothaft commented Apr 7, 2016

Thanks @heuermh! Merged.

@heuermh heuermh deleted the heuermh:print-metrics branch Apr 7, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment