New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add test sample(verify number of reads in loadAlignments function) and ADAM SNAPSHOT document #1433

Closed
wants to merge 2 commits into
from

Conversation

Projects
None yet
5 participants
@xubo245

xubo245 commented Mar 10, 2017

add test sample : sc.loadAlignments should not fail on single-end and paired-end fastq reads

there are some problem in adam-0.21.0, link: #1424

It was fixed in Adam-0.21.1-SNAPSHOT, link 0e57357; So I only add a test sample for it!

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Mar 10, 2017

Can one of the admins verify this patch?

Can one of the admins verify this patch?

@xubo245

This comment has been minimized.

Show comment
Hide comment
@xubo245

xubo245 Mar 10, 2017

I test the native data in adam-0.21.0. the number of line in bqsr1-r1.fq is 1952, it should be 488 reads.
bqsr1-r2.fq too.

It should be total 976 reads after loadAlignments, but there are 892 reads by running my code.

code:

  sparkTest("load fastq") {
    val fastq1Path = testFile("bqsr1-r1.fq")
    val fastq2Path = testFile("bqsr1-r2.fq")
    var align1=sc.loadAlignments(fastq1Path,filePath2Opt = Option(fastq2Path))
    println(align1.rdd.count())
    println(align1.rdd.count()*2)
  }

I add the code into org.bdgenomics.adam.cli.Adam2FastqSuite
the data in adam-adam-parent-spark2_2.10-0.21.0\adam-cli\src\test\resources

result:

2017-03-08 11:21:46 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-03-08 11:21:47 WARN  MetricsSystem:71 - Using default name DAGScheduler for source because spark.app.id is not set.
892
1784

=>
892!=976

xubo245 commented Mar 10, 2017

I test the native data in adam-0.21.0. the number of line in bqsr1-r1.fq is 1952, it should be 488 reads.
bqsr1-r2.fq too.

It should be total 976 reads after loadAlignments, but there are 892 reads by running my code.

code:

  sparkTest("load fastq") {
    val fastq1Path = testFile("bqsr1-r1.fq")
    val fastq2Path = testFile("bqsr1-r2.fq")
    var align1=sc.loadAlignments(fastq1Path,filePath2Opt = Option(fastq2Path))
    println(align1.rdd.count())
    println(align1.rdd.count()*2)
  }

I add the code into org.bdgenomics.adam.cli.Adam2FastqSuite
the data in adam-adam-parent-spark2_2.10-0.21.0\adam-cli\src\test\resources

result:

2017-03-08 11:21:46 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-03-08 11:21:47 WARN  MetricsSystem:71 - Using default name DAGScheduler for source because spark.app.id is not set.
892
1784

=>
892!=976

xubo245

@xubo245 xubo245 changed the title from add test: sc.loadAlignments should not fail on single-end and paired-… to Add test sample(verify loadAlignments number of reads) and ADAM SNAPSHOT document Mar 10, 2017

@xubo245 xubo245 changed the title from Add test sample(verify loadAlignments number of reads) and ADAM SNAPSHOT document to Add test sample(verify number of reads in loadAlignments function) and ADAM SNAPSHOT document Mar 10, 2017

@xubo245

This comment has been minimized.

Show comment
Hide comment
@xubo245

xubo245 Mar 10, 2017

some related issue:
#1428

#1432

xubo245 commented Mar 10, 2017

some related issue:
#1428

#1432

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Mar 10, 2017

Member

Jenkins, test this please

Member

heuermh commented Mar 10, 2017

Jenkins, test this please

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls Mar 10, 2017

Coverage Status

Coverage increased (+0.1%) to 76.54% when pulling 446ae81 on xubo245:master into 07c1982 on bigdatagenomics:master.

coveralls commented Mar 10, 2017

Coverage Status

Coverage increased (+0.1%) to 76.54% when pulling 446ae81 on xubo245:master into 07c1982 on bigdatagenomics:master.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Mar 10, 2017

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1850/

Build result: FAILURE

[...truncated 16 lines...] > /home/jenkins/git2/bin/git rev-parse origin/pr/1433/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 39eeb84 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1433/merge^{commit} # timeout=10Checking out Revision 39eeb84 (origin/pr/1433/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 39eeb84f282d952038b0c984047cdb49adf6214bFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1850/

Build result: FAILURE

[...truncated 16 lines...] > /home/jenkins/git2/bin/git rev-parse origin/pr/1433/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 39eeb84 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1433/merge^{commit} # timeout=10Checking out Revision 39eeb84 (origin/pr/1433/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 39eeb84f282d952038b0c984047cdb49adf6214bFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

@xubo245

This comment has been minimized.

Show comment
Hide comment
@xubo245

xubo245 Mar 11, 2017

I can not find any error from my code in https://amplab.cs.berkeley.edu/jenkins/job/ADAM-prb/HADOOP_VERSION=2.3.0,SCALAVER=2.11,SPARK_VERSION=1.6.1,label=centos/1850/consoleText, the errors are other functions'...

Can you tell me where error in my PR code? Please.

xubo245 commented Mar 11, 2017

I can not find any error from my code in https://amplab.cs.berkeley.edu/jenkins/job/ADAM-prb/HADOOP_VERSION=2.3.0,SCALAVER=2.11,SPARK_VERSION=1.6.1,label=centos/1850/consoleText, the errors are other functions'...

Can you tell me where error in my PR code? Please.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Mar 14, 2017

Member

Hi @xubo245! Apologies for the slow reply, been busy the last few days tracking down various bugs and etc. None of the tests failed, but our source formatting check failed. Can you run ./scripts/format-source from the root of your local checkout, commit the changes, and push to your PR branch? This should fix the formatting issues. Let me know if you have any issues otherwise! Otherwise, this test looks good to me, thanks for adding it!

Member

fnothaft commented Mar 14, 2017

Hi @xubo245! Apologies for the slow reply, been busy the last few days tracking down various bugs and etc. None of the tests failed, but our source formatting check failed. Can you run ./scripts/format-source from the root of your local checkout, commit the changes, and push to your PR branch? This should fix the formatting issues. Let me know if you have any issues otherwise! Otherwise, this test looks good to me, thanks for adding it!

@xubo245

This comment has been minimized.

Show comment
Hide comment
@xubo245

xubo245 Mar 14, 2017

Don‘t worry.
What are various bugs in your track? Can you tell me?please.
Sorry, I don't run format.
I run in another branch: #1434
It is the same as this.

Thanks you very much!

xubo245 commented Mar 14, 2017

Don‘t worry.
What are various bugs in your track? Can you tell me?please.
Sorry, I don't run format.
I run in another branch: #1434
It is the same as this.

Thanks you very much!

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Mar 14, 2017

Member

What are various bugs in your track? Can you tell me?please.

Definitely! I'm most focused on finishing up #1423 and #1412 right now.

Member

fnothaft commented Mar 14, 2017

What are various bugs in your track? Can you tell me?please.

Definitely! I'm most focused on finishing up #1423 and #1412 right now.

@xubo245

This comment has been minimized.

Show comment
Hide comment
@xubo245

xubo245 Mar 14, 2017

Thanks, I also need to invoke BQSR 、MarkDuplicates and SNP/INDEL discovery function to process adam data later. May I ask you some questions if I have problem about it later?

xubo245 commented Mar 14, 2017

Thanks, I also need to invoke BQSR 、MarkDuplicates and SNP/INDEL discovery function to process adam data later. May I ask you some questions if I have problem about it later?

@xubo245 xubo245 closed this Mar 17, 2017

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Mar 21, 2017

Member

Succeeded by #1448.

Member

heuermh commented Mar 21, 2017

Succeeded by #1448.

heuermh added a commit to heuermh/adam that referenced this pull request Mar 22, 2017

@xubo245

This comment has been minimized.

Show comment
Hide comment
@xubo245

xubo245 Jan 10, 2018

I raise a PR, but why don't you review this PR and then merge it ? but raise https://github.com/bigdatagenomics/adam/pull/1448/files,.............

xubo245 commented Jan 10, 2018

I raise a PR, but why don't you review this PR and then merge it ? but raise https://github.com/bigdatagenomics/adam/pull/1448/files,.............

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment