Gracefully handle missing Spark- and Hadoop-versions in jenkins-test; document how to set them. #827

Closed
sbvarre opened this Issue Sep 23, 2015 · 5 comments

Comments

Projects
None yet
4 participants
@sbvarre

sbvarre commented Sep 23, 2015

I tried running the Jenkins test (./scripts/jenkins-test) and I get the following message which I am not sure how to interpret.

jenkins test

The reactor summary shows that the jars are built fine.But it still doesn't recognize the Hadoop version and the Spark version.

Any help/suggestions are greatly appreciated.

Thanks.
Bharath

@ryan-williams

This comment has been minimized.

Show comment
Hide comment
@ryan-williams

ryan-williams Sep 23, 2015

Member

it is expecting the $HADOOP_VERSION env var to be set to one of those three
values.
https://github.com/bigdatagenomics/adam/blob/master/scripts/jenkins-test#L29-L38

Presumably something in the jenkins config sets those things when it's
doing its testing matrix over all supported {Spark, Hadoop} versions.

I never run jenkins-test locally, fwiw. I just run mvn test, and let
jenkins do the heavy lifting on PRs; it's ok to submit PRs where jenkins
fails many times before it ultimately passes. People will often hold off on
looking at them until the tests pass.

On Wed, Sep 23, 2015 at 4:43 PM Bharath Varre notifications@github.com
wrote:

I tried running the Jenkins test and I get the following message which I
am not sure how to interpret.

[image: jenkins test]
https://cloud.githubusercontent.com/assets/8506259/10058173/f298bb7c-61f8-11e5-9794-40b9bf145c10.JPG

The reactor summary shows that the jars are built fine.But it still
doesn't recognize the Hadoop version and the Spark version.

Any help/suggestions are greatly appreciated.

Thanks.
Bharath


Reply to this email directly or view it on GitHub
#827.

Member

ryan-williams commented Sep 23, 2015

it is expecting the $HADOOP_VERSION env var to be set to one of those three
values.
https://github.com/bigdatagenomics/adam/blob/master/scripts/jenkins-test#L29-L38

Presumably something in the jenkins config sets those things when it's
doing its testing matrix over all supported {Spark, Hadoop} versions.

I never run jenkins-test locally, fwiw. I just run mvn test, and let
jenkins do the heavy lifting on PRs; it's ok to submit PRs where jenkins
fails many times before it ultimately passes. People will often hold off on
looking at them until the tests pass.

On Wed, Sep 23, 2015 at 4:43 PM Bharath Varre notifications@github.com
wrote:

I tried running the Jenkins test and I get the following message which I
am not sure how to interpret.

[image: jenkins test]
https://cloud.githubusercontent.com/assets/8506259/10058173/f298bb7c-61f8-11e5-9794-40b9bf145c10.JPG

The reactor summary shows that the jars are built fine.But it still
doesn't recognize the Hadoop version and the Spark version.

Any help/suggestions are greatly appreciated.

Thanks.
Bharath


Reply to this email directly or view it on GitHub
#827.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Sep 23, 2015

Member

Presumably something in the jenkins config sets those things when it's doing its testing matrix over all supported {Spark, Hadoop} versions.

Correct!

Member

fnothaft commented Sep 23, 2015

Presumably something in the jenkins config sets those things when it's doing its testing matrix over all supported {Spark, Hadoop} versions.

Correct!

@sbvarre

This comment has been minimized.

Show comment
Hide comment
@sbvarre

sbvarre Sep 23, 2015

Thank you Ryan and Frank. I was just trying to follow the instructions to see if It would fetch the mouse chromosome, encode it to ADAM reads and pileups, run flagstat, etc.

sbvarre commented Sep 23, 2015

Thank you Ryan and Frank. I was just trying to follow the instructions to see if It would fetch the mouse chromosome, encode it to ADAM reads and pileups, run flagstat, etc.

@ryan-williams

This comment has been minimized.

Show comment
Hide comment
@ryan-williams

ryan-williams Sep 23, 2015

Member

We should probably:

  • have a better error message in jenkins-test when versions are not set
  • clarify how/what they should be set to in the README / other docs
Member

ryan-williams commented Sep 23, 2015

We should probably:

  • have a better error message in jenkins-test when versions are not set
  • clarify how/what they should be set to in the README / other docs

@ryan-williams ryan-williams changed the title from Issues with Jenkins Test to Gracefully handle missing Spark- and Hadoop-versions in jenkins-test; document how to set them. Sep 23, 2015

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Nov 5, 2015

Member

I started converting jenkins-test into a nextflow pipeline as a demonstration and found that it calls

"$ADAM" reads2ref "$READS" "$PILEUPS"

which doesn't exist as an ADAM command. Was reads2ref removed or renamed, or perhaps turned into a transform option?

Member

heuermh commented Nov 5, 2015

I started converting jenkins-test into a nextflow pipeline as a demonstration and found that it calls

"$ADAM" reads2ref "$READS" "$PILEUPS"

which doesn't exist as an ADAM command. Was reads2ref removed or renamed, or perhaps turned into a transform option?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment