Scaffold failed with return code -11

[KDSR]

I have tried running scaffold a few times on a sample, and I keep getting an error. The details are below.

During scaffold, the following command failed with return code -11:

/opt/metAMOS/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/OrientContigs -minRedundancy 5 -all -redundancy 10 -b /master/kreeves/metAMOS/20160812_microbiome_test_c/Scaffold/in/proba.bnk -repeats /master/kreeves/metAMOS/20160812_microbiome_test_c/Scaffold/in/proba.reps

_DETAILS_**********
Last 10 commands run before the error (/master/kreeves/metAMOS/20160812_microbiome_test_c/Logs/COMMANDS.log)
|2016-08-15 10:23:19|# [SCAFFOLD]
|2016-08-15 10:23:19| rm -rf /master/kreeves/metAMOS/20160812_microbiome_test_c/Scaffold/in/proba.bnk
|2016-08-15 10:48:26| /opt/metAMOS/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/toAmos_new -Q /master/kreeves/metAMOS/20160812_microbiome_test_c/Preprocess/out/lib1.seq -i --min 200 --max 800 --libname lib1 -b /master/kreeves/metAMOS/20160812_microbiome_test_c/Scaffold/in/proba.bnk
|2016-08-15 10:55:30| /opt/metAMOS/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/toAmos_new -c /master/kreeves/metAMOS/20160812_microbiome_test_c/Assemble/out/proba.asm.tigr -b /master/kreeves/metAMOS/20160812_microbiome_test_c/Scaffold/in/proba.bnk
|2016-08-15 11:10:05| /opt/metAMOS/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/asmQC -b /master/kreeves/metAMOS/20160812_microbiome_test_c/Scaffold/in/proba.bnk -scaff -recompute -update -numsd 2
|2016-08-15 11:10:05| perl /opt/metAMOS/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/bank-unlock /master/kreeves/metAMOS/20160812_microbiome_test_c/Scaffold/in/proba.bnk
|2016-08-15 11:18:35| /opt/metAMOS/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/clk -b /master/kreeves/metAMOS/20160812_microbiome_test_c/Scaffold/in/proba.bnk
|2016-08-15 11:23:23| /opt/metAMOS/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/Bundler -b /master/kreeves/metAMOS/20160812_microbiome_test_c/Scaffold/in/proba.bnk
|2016-08-15 21:18:15| /opt/metAMOS/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/MarkRepeats -redundancy 50 -b /master/kreeves/metAMOS/20160812_microbiome_test_c/Scaffold/in/proba.bnk > /master/kreeves/metAMOS/20160812_microbiome_test_c/Scaffold/in/proba.reps
|2016-08-16 00:21:22| /opt/metAMOS/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/OrientContigs -minRedundancy 5 -all -redundancy 10 -b /master/kreeves/metAMOS/20160812_microbiome_test_c/Scaffold/in/proba.bnk -repeats /master/kreeves/metAMOS/20160812_microbiome_test_c/Scaffold/in/proba.reps

Last 10 lines of output (/master/kreeves/metAMOS/20160812_microbiome_test_c/Logs/SCAFFOLD.log)
FOR SKIPPED EDGE 1061300 SET EDGE STATUS TO BE 5
FOR SKIPPED EDGE 1108067 SET EDGE STATUS TO BE 6
FOR SKIPPED EDGE 1222376 SET EDGE STATUS TO BE 2
FOR SKIPPED EDGE 1237593 SET EDGE STATUS TO BE 5
FOR SKIPPED EDGE 1272663 SET EDGE STATUS TO BE 5
FOR SKIPPED EDGE 1348062 SET EDGE STATUS TO BE 5
FOR SKIPPED EDGE 1364684 SET EDGE STATUS TO BE 5
FOR SKIPPED EDGE 1461303 SET EDGE STATUS TO BE 5
FOR SKIPPED EDGE 1471533 SET EDGE STATUS TO BE 6
FOR SKIPPED EDGE 1500178 SET EDGE STATUS TO BE 5

Please veryify input data and restart MetAMOS. If the problem persists please contact the MetAMOS development team.

Any help or guidance will be appreciated!
Thanks,
Kim

[KDSR]

I ran the MetAMOS pipeline again, trying to skip the scaffold step, but I got another error during the postprocess step. The details are below, and I would appreciate any help or feedback to get the pipeline working.
Kim

Warning: untrusted X11 forwarding setup failed: xauth key data not generated
Last login: Mon Aug 15 10:22:33 2016 from mac206.txbiomedgenetics.org
Rocks 6.2 (SideWinder)
Profile built 20:54 25-Jan-2016

Kickstarted 16:22 25-Jan-2016
stevens.txbiomedgenetics.org% cd metAMOS
/master/kreeves/metAMOS
stevens.txbiomedgenetics.org% runPipeline -n Scaffold -d 20160812_microbiome_test_c

Starting metAMOS pipeline
Warning: BLASR is not found, some functionality will not be available
Warning: Newbler is not found, some functionality will not be available
Warning: MetaGeneMark is not found, some functionality will not be available
Warning: SignalP+ is not found, some functionality will not be available
Warning: PHmmer is not found, some functionality will not be available
Warning: REAPR is not found, some functionality will not be available
Warning: FreeBayes is not found, some functionality will not be available
Warning: MPI is not available, some functionality may not be available
[Available RAM: 130 GB]
*ok
[Available CPUs: 48]
*ok

---

Tasks which will be run:

Task = assemble.SplitAssemblers
Task = assemble.Assemble
Task = assemble.CheckAsmResults
Task = assemble.SplitMappers
Task = mapreads.MapReads
Task = mapreads.CheckMapResults
Task = mapreads.SplitForORFs
Task = findorfs.FindORFS
Task = validate.Validate
Task = findreps.FindRepeats
Task = annotate.Annotate
Task = fannotate.FunctionalAnnotation
Task = scaffold.Scaffold
Task = findscforfs.FindScaffoldORFS
Task = abundance.Abundance
Task = propagate.Propagate
Task = classify.Classify
Task = postprocess.Postprocess

---

metAMOS configuration summary:
metAMOS Version: v1.5rc3 "Praline Brownie" workflows: core,optional,imetamos
Time and Date: 2016-09-02
Working directory: /master/kreeves/metAMOS/20160812_microbiome_test_c
Prefix: proba
K-Mer: 31
Threads: 47
Taxonomic level: class
Verbose: False
Steps to skip: MultiAlign, FindRepeats, Scaffold, FunctionalAnnotation, Propagate, FindScaffoldORFS
Steps to force:

[citation]
MetAMOS
Treangen, TJ ⇔ Koren, S, Sommer, DD, Liu, B, Astrovskaya, I, Ondov, B, Darling AE, Phillippy AM, Pop, M. MetAMOS: a modular and open source metagenomic assembly and analysis pipeline. Genome biology, 14(1), R2, 2013.

Step-specific configuration:
[abundance]
MetaPhyler
/opt/metAMOS/metAMOS-1.5rc3/Utilities/cpp/Linux-x86_64
Liu B, Gibbons T, Ghodsi M, Treangen T, Pop M. Accurate and fast estimation of taxonomic profiles from metagenomic shotgun sequences. BMC Genomics. 2011;12 Suppl 2:S4. Epub 2011 Jul 27.

[multialign]
M-GCAT
/opt/metAMOS/metAMOS-1.5rc3/Utilities/cpp/Linux-x86_64
Treangen TJ, Messeguer X. M-GCAT: interactively and efficiently constructing large-scale multiple genome comparison frameworks in closely related species. BMC Bioinformatics, 2006.

[fannotate]
BLAST
/opt/metAMOS/metAMOS-1.5rc3/Utilities/cpp/Linux-x86_64
Altschul SF, Gish W, Miller W, Myers EW, Lipman DJ. Basic local alignment search tool. J Mol Biol. 1990 Oct 5;215(3):403-10.

[scaffold]
Bambus 2
/opt/metAMOS/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin
Koren S, Treangen TJ, Pop M. Bambus 2: scaffolding metagenomes. Bioinformatics 27(21): 2964-2971 2011.

[findorfs]
FragGeneScan
/opt/metAMOS/metAMOS-1.5rc3/Utilities/cpp/Linux-x86_64
Rho M, Tang H, Ye Y: FragGeneScan: predicting genes in short and error-prone reads. Nucleic Acids Research 2010, 38:e191-e191.

[annotate]
Kraken
/opt/metAMOS/metAMOS-1.5rc3/Utilities/cpp/Linux-x86_64/kraken/bin
Wood DE, Salzberg SL: Kraken: ultrafast metagenomic sequence classification using exact alignments. Genome Biology 2014, 15:R46.

[assemble]
SOAPdenovo
/opt/metAMOS/metAMOS-1.5rc3/Utilities/cpp/Linux-x86_64
Li Y, Hu Y, Bolund L, Wang J: State of the art de novo assembly of human genomes from massively parallel sequencing data.Human genomics 2010, 4:271-277.

[mapreads]
Bowtie
/opt/metAMOS/metAMOS-1.5rc3/Utilities/cpp/Linux-x86_64
Langmead B, Trapnell C, Pop M, Salzberg SL. Ultrafast and memory-efficient alignment of short DNA sequences to the human genome. Genome Biol. 2009;10(3):R25. Epub 2009 Mar 4.

[preprocess]
metAMOS built-in filtering
N/A

[validate]
LAP
/opt/metAMOS/metAMOS-1.5rc3/LAP
Ghodsi M, Hill CM, Astrovskaya I, Lin H, Sommer DD, Koren S, Pop M. De novo likelihood-based measures for comparing genome assemblies. BMC research notes 6:334, 2013.

[other]
Krona
/opt/metAMOS/metAMOS-1.5rc3/KronaTools/bin
Ondov BD, Bergman NH, Phillippy AM. Interactive metagenomic visualization in a Web browser. BMC Bioinformatics. 2011 Sep 30;12:385.

Job  = [preprocess.success -> *.run] completed

Completed Task = assemble.SplitAssemblers
Uptodate Task = assemble.Assemble
Uptodate Task = assemble.CheckAsmResults
Uptodate Task = assemble.SplitMappers
Uptodate Task = mapreads.MapReads
Uptodate Task = mapreads.CheckMapResults
Uptodate Task = mapreads.SplitForORFs
Uptodate Task = findorfs.FindORFS
Uptodate Task = validate.Validate
Starting Task = findrepeats.FINDREPEATS
Job = [proba.fna -> proba.repeats] completed
Completed Task = findreps.FindRepeats
Uptodate Task = annotate.Annotate
Starting Task = functionalannotation.FUNCTIONALANNOTATION
Job = [proba.faa -> [blast.out, krona.ec.input]] completed
Completed Task = fannotate.FunctionalAnnotation
Starting Task = scaffold.SCAFFOLD
Job = [[proba.asm.contig] -> scaffold.ok] completed
Completed Task = scaffold.Scaffold

Starting Task = findscaffoldorfs.FINDSCAFFOLDORFS
Job = [proba.linearize.scaffolds.final -> proba.fna] completed
Completed Task = findscforfs.FindScaffoldORFS
Starting Task = abundance.ABUNDANCE
Job = [proba.asm.contig -> proba.taxprof.pct.txt] completed
Completed Task = abundance.Abundance
Starting Task = propagate.PROPAGATE
Job = [proba.annots -> propagate.ok] completed
Completed Task = propagate.Propagate
Starting Task = classify.CLASSIFY
Job = [proba.clusters -> classify.ok] completed
Completed Task = classify.Classify
Starting Task = postprocess.POSTPROCESS

---

_ERROR_**********
During postprocess, the following command failed with return code 2:

perl -I /opt/metAMOS/metAMOS-1.5rc3/AMOS/Linux-x86_64/lib /opt/metAMOS/metAMOS-1.5rc3/Utilities/perl/statistics.pl /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/proba.scf.fa > /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/html/asmstats.out

_DETAILS_**********
Last 10 commands run before the error (/master/kreeves/metAMOS/20160812_microbiome_test_c/Logs/COMMANDS.log)
|2016-09-02 16:36:01| ln /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/abundance.krona.html /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/html/Abundance.html
|2016-09-02 16:36:01| touch /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/ref.name
|2016-09-02 16:36:01| mv /master/kreeves/metAMOS/20160812_microbiome_test_c/Preprocess/out/*.fastqc /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/html
|2016-09-02 16:36:01| unlink /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/html/propagate.in.clusters
|2016-09-02 16:36:01| ln /master/kreeves/metAMOS/20160812_microbiome_test_c/Propagate/in/proba.clusters /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/html/propagate.in.clusters
|2016-09-02 16:36:01| unlink /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/html/propagate.out.clusters
|2016-09-02 16:36:01| ln /master/kreeves/metAMOS/20160812_microbiome_test_c/Propagate/out/proba.clusters /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/html/propagate.out.clusters
|2016-09-02 16:36:01| unlink /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/html/FunctionalAnnotation.html
|2016-09-02 16:36:01| ln /master/kreeves/metAMOS/20160812_microbiome_test_c/FunctionalAnnotation/out/ec.krona.html /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/html/FunctionalAnnotation.html
|2016-09-02 16:36:01| perl -I /opt/metAMOS/metAMOS-1.5rc3/AMOS/Linux-x86_64/lib /opt/metAMOS/metAMOS-1.5rc3/Utilities/perl/statistics.pl /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/proba.scf.fa > /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/html/asmstats.out

Last 10 lines of output (/master/kreeves/metAMOS/20160812_microbiome_test_c/Logs/POSTPROCESS.log)
unlink: cannot unlink /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/html/class.classified': No such file or directory unlink: cannot unlink/master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/html/Annotate.html': No such file or directory
unlink: cannot unlink /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/html/Abundance.html': No such file or directory mv: cannot stat/master/kreeves/metAMOS/20160812_microbiome_test_c/Preprocess/out/*.fastqc': No such file or directory
unlink: cannot unlink /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/html/propagate.in.clusters': No such file or directory unlink: cannot unlink/master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/html/propagate.out.clusters': No such file or directory
unlink: cannot unlink /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/html/FunctionalAnnotation.html': No such file or directory ln: accessing/master/kreeves/metAMOS/20160812_microbiome_test_c/FunctionalAnnotation/out/ec.krona.html': No such file or directory
Can't locate Statistics/Descriptive.pm in @inc (@inc contains: /opt/metAMOS/metAMOS-1.5rc3/AMOS/Linux-x86_64/lib /opt/metAMOS/metAMOS-1.5rc3/src/phylosift/lib/ /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /opt/metAMOS/metAMOS-1.5rc3/Utilities/perl/statistics.pl line 7.
BEGIN failed--compilation aborted at /opt/metAMOS/metAMOS-1.5rc3/Utilities/perl/statistics.pl line 7.

Please veryify input data and restart MetAMOS. If the problem persists please contact the MetAMOS development team.
_ERROR_**********

---

rm: cannot remove `/master/kreeves/metAMOS/20160812_microbiome_test_c/Logs/postprocess.ok': No such file or directory
Oops, MetAMOS finished with errors! see text in red above for details.

[skoren]

You are missing the Statistics/Descriptive.pm perl library. It should be installed by metAMOS but it looks like it wasn't in your case. If you download/install it it should fix the error.

[KDSR]

Hi Sergey,

Thanks so much for getting back to me! It looks like the Statistics/Descriptive library is already installed (see below). Is there a special perl that needs to be running, or is this a special perl belonging to metAMOS that needs to be in the path? How do you install the library using metAMOS?

Thanks again!!
Kim

[root@stevens ~]# cpanm Statistics::Descriptive
Statistics::Descriptive is up to date. (3.0612)
[root@stevens ~]#

Here is a list of the perl modules installed on Stevens…

[root@stevens ~]# instmodsh
Available commands are:
l - List all installed modules
m - Select a module
q - Quit the program
cmd? l
Installed modules are:
Algorithm::Diff
App::cpanminus
Archive::Tar
Archive::Zip
B::Hooks::EndOfScope
Bio
Bio::FASTASequence
Bio::FASTASequence::File
Bio::Phylo
Bio::Run
CPAN
Capture::Tiny
Class::Accessor
Class::Data::Inheritable
Compress::Raw::Bzip2
Compress::Raw::Zlib
Convert::Binary::C
Cwd
Data::Dumper
Devel::StackTrace
Digest::SHA
Exception::Class
Exporter::Tiny
ExtUtils::CBuilder
ExtUtils::MakeMaker
File::Copy::Link
File::HomeDir
File::Sort
File::Temp
File::Which
HTML-TableExtract
IO::Compress
IPC::System::Simple
Module::Build
Module::Implementation
Module::Runtime
Net
Package::Stash
Parse::CPAN::Meta
Perl
SUPER
Socket
Sort::Naturally
Statistics::Descriptive
Sub::Exporter::Progressive
Sub::Identify
Term::ReadKey
Term::ReadLine
Test::Differences
Test::Exception
Test::Harness
Test::MockModule
Test::Most
Test::Simple
Text::Diff
Time::Piece
Variable::Magic
XML::SAX::Base
XML::SAX::Expat
XML::Simple
namespace::clean

On Sep 22, 2016, at 2:44 PM, Sergey Koren <notifications@github.commailto:notifications@github.com> wrote:

You are missing the Statistics/Descriptive.pm perl library. It should be installed by metAMOS but it looks like it wasn't in your case. If you download/install it it should fix the error.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com//issues/248#issuecomment-249007780, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AUr5Vf4Md7BYzneIl_Ti2GFqmCHDIzSlks5qstqWgaJpZM4Jy6Wa.

CONFIDENTIALITY NOTICE: This e-mail and any files and/or attachments transmitted, may contain privileged and confidential information and is intended solely for the exclusive use of the individual or entity to whom it is addressed. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or copying of this e-mail and/or attachments is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender stating that this transmission was misdirected; return the e-mail to sender; destroy all paper copies and delete all electronic copies from your system without disclosing its contents.

[skoren]

The paths perl is checking is listed in the error message:

Can't locate Statistics/Descriptive.pm in @INC (@INC contains: /opt/metAMOS/metAMOS-1.5rc3/AMOS/Linux-x86_64/lib /opt/metAMOS/metAMOS-1.5rc3/src/phylosift/lib/ /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .)

It looks like it includes your system perl5 libraries, is Statistics::Descriptive available in the paths listed above? If not, you'd need to add it to your PERL5LIB path.

INSTALL.py installs it along with AMOS so if you remove the AMOS folder in metAMOS and re-run INSTALL.py it will check for Statistics::Descriptive and install if it is not found, assuming your environment where you install metAMOS is the same as where you run.

[shizhaojingszj]

Exactly the same error, using HMP sample: SRS077730, two files: fq1=SRS077730.denovo_duplicates_marked.trimmed.1.fastq
fq2=SRS077730.denovo_duplicates_marked.trimmed.2.fastq

First time run:
$Script/initPipeline -q -1 $fq1 -2 $fq2 -d $outdir -i 160:260
$Script/runPipeline -a soap -c fcp -g fraggenescan -p 15 -d $outdir -k 55 -f Assemble,MapReads,FindORFS,Annotate,FunctionalAnnotation,Propagate,Classify,Abundance,FindScaffoldORFS

failed at Mapreads without any error:

Second Time Run:
$Script/runPipeline -a soap -c fcp -g fraggenescan -p 15 -d $outdir -k 55 -f FindORFS,Annotate,FunctionalAnnotation,Propagate,Classify,Abundance,FindScaffoldORFS

failed at Scaffold step, error message is quite the same, error code -11

dmesg result:
[1923489.287583] OrientContigs[114077]: segfault at 10 ip 000000000041f63a sp 00007ffc350e3cd0 error 4 in OrientContigs[400000+119000]

Hope this can be fixed.