Consensus Failed #1325

Lcornet · 2019-04-11T21:04:49Z

Dear Canu Team,

I run CANU (v1.8 +90 changes) by defaullt on nanopore data, including some very long reads.
My consensus jobs failed :

...
-- Using slow alignment for consensus (iteration '2').
-- Configured 57 contig and 40 unitig consensus jobs.
--
-- Consensus jobs failed, tried 2 times, giving up.
--   job ctgcns/0001.cns FAILED.
--   job ctgcns/0006.cns FAILED.
...
...

I tried to relaunch with adding this : cnsErrorRate=0.25, but I still have the same error.

canu -assemble -nanopore-corrected ONT/ONT.trimmedReads.fasta.gz genomeSize=0.750g \
maxMemory=455 maxThreads=59 cnsErrorRate=0.25 -p ONT -d ONT \
useGrid=false

Could you help me ? Do I have to increase the cnsErrorRate ?

Thanks,
I have the complete log if needed.

The text was updated successfully, but these errors were encountered:

skoren · 2019-04-11T21:57:23Z

Can you post the error from the failed jobs (unitigging/consensus/*1 or 6.out)? Can you also post the contents of consensus.sh?

Lcornet · 2019-04-11T23:15:41Z

#unitigging/5-consensus/*1.out

     79     66752      53        49   10.19x        0    0.00x         4    2.23x
     82     27914      14        10    2.87x        0    0.00x         4    2.59x
     88   1861446    2815      2631   21.54x        0    0.00x       184    3.90x
     89     14705      19        17    7.03x        0    0.00x         2    1.94x
     93    321934     262       222    9.03x        0    0.00x        40    3.32x
     96     82837      55        51    7.90x        0    0.00x         4    2.00x
     97     56804      44        39    8.86x        0    0.00x         5    3.generateTemplateStitch()-- FAILED to align - no more template to remove!  Fail!
generateTemplateStitch()-- FAILED to align - no more template to remove!  Fail!
utgcns: utgcns/unitigConsensus.C:989: void unitigConsensus::findCoordinates(): Assertion `bgn < end' failed.
40x
     99    178578     108        97    7.42x        0    0.00x        11    2.51x
    100     29384      14        12    3.22x        0    0.00x         2    1.45x
    104     75649      44        41    7.63x        0    0.00x         3    1.95x
    105     42107      21        19    5.33x        0    0.00x         2    1.57x
    106     46465      32        29    8.69x        0    0.00x         3    2.10x
    107     84520      63        56    8.43x        0    0.00x         7    2.43x
./consensus.sh: line 100: 53395 Aborted                 (core dumped) $bin/utgcns -S ../ONT.${tag}Store/partitionedReads.seqStore -T ../ONT.${tag}Store 1 $jobid -O ./${tag}cns/$jobid.cns.WORKING -maxcoverage 40 -e 0.25 -pbdagcon -edlib -threads 8

#unitigging/5-consensus/*6.out

Running job 6 based on command line options.
-- Opening seqStore '../ONT.ctgStore/partitionedReads.seqStore' partition 6.
-- Opening tigStore '../ONT.ctgStore' version 1.
-- Opening output results file './ctgcns/0006.cns.WORKING'.
--
-- Computing consensus for b=0 to e=324849 with errorRate 0.2500 (max 0.4000) and minimum overlap 40
--
generateTemplateStitch()-- FAILED to align - no more template to remove!  Fail!
generateTemplateStitch()-- FAILED to align - no more template to remove!  Fail!
utgcns: utgcns/unitigConsensus.C:989: void unitigConsensus::findCoordinates(): Assertion `bgn < end' failed.
                           ----------CONTAINED READS----------  -DOVETAIL  READS-
  tigID    length   reads      used coverage  ignored coverage      used coverage
------- --------- -------  -------- -------- -------- --------  -------- --------
   1600     53484      23        19    3.73x        0    0.00x         4    1.87x
   1605     60646      24        14    2.28x        0    0.00x        10    2.65x
   1608     25793       7         5    1.60x        0    0.00x         2    1.69x
   1612     59031      21        15    2.43x        0    0.00x         6    2.19x
   1613     40144      26        21    4.33x        0    0.00x         5    2.61x
   1615     56528      45        40    8.36x        0    0.00x         5    2.57x
   1618    104212      87        78   10.86x        0    0.00x         9    3.20x
   1625    344891     343       311   12.60x        0    0.00x        32    3.02x
   1627     21028      12         8    2.03x        0    0.00x         4    2.93x
   1628    106856      66        58    7.68x        0    0.00x         8    2.12x
   1629    212100     151       122    6.16x        0    0.00x        29    3.08x
   1630     39075      43        39   10.86x        0    0.00x         4    2.54x
   1631     20224      13         8    1.53x        0    0.00x         5    2.79x
   1636     18169       4         1    0.18x        0    0.00x         3    2.01x
   1641    210320     170       146    9.24x        0    0.00x        24    3.41x
   1643     47610      18        14    2.53x        0    0.00x         4    1.65x
   1644     12335       2         0    0.00x        0    0.00x         2    1.73x
   1645    472628     589       517   13.83x        0    0.00x        72    3.70x
   1649     34771      20        16    5.23x        0    0.00x         4    2.55x
   1650    263534     285       261   10.60x        0    0.00x        24    2.87x
./consensus.sh: line 100: 53402 Aborted                 (core dumped) $bin/utgcns -S ../ONT.${tag}Store/partitionedReads.seqStore -T ../ONT.${tag}Store 1 $jobid -O ./${tag}cns/$jobid.cns.WORKING -maxcoverage 40 -e 0.25 -pbdagcon -edlib -threads 8

#unitigging/5-consensus/consensus.sh

#!/bin/sh


#  Path to Canu.

syst=`uname -s`
arch=`uname -m | sed s/x86_64/amd64/`

bin="/media/vol2/scratch/lcornet/cassava/CANU-1.8/canu/$syst-$arch/bin"

if [ ! -d "$bin" ] ; then
  bin="/media/vol2/scratch/lcornet/cassava/CANU-1.8/canu"
fi

#  Report paths.

echo ""
echo "Found perl:"
echo "  " `which perl`
echo "  " `perl --version | grep version`
echo ""
echo "Found java:"
echo "  " `which java`
echo "  " `java -showversion 2>&1 | head -n 1`
echo ""
echo "Found canu:"
echo "  " $bin/canu
echo "  " `$bin/canu -version`
echo ""


#  Environment for any object storage.

export CANU_OBJECT_STORE_CLIENT=
export CANU_OBJECT_STORE_NAMESPACE=ONT
export CANU_OBJECT_STORE_PROJECT=




#  Discover the job ID to run, from either a grid environment variable and a
#  command line offset, or directly from the command line.
#
if [ x$CANU_LOCAL_JOB_ID = x -o x$CANU_LOCAL_JOB_ID = xundefined -o x$CANU_LOCAL_JOB_ID = x0 ]; then
  baseid=$1
  offset=0
else
  baseid=$CANU_LOCAL_JOB_ID
  offset=$1
fi
if [ x$offset = x ]; then
  offset=0
fi
if [ x$baseid = x ]; then
  echo Error: I need CANU_LOCAL_JOB_ID set, or a job index on the command line.
  exit
fi
jobid=`expr $baseid + $offset`
if [ x$CANU_LOCAL_JOB_ID = x ]; then
  echo Running job $jobid based on command line options.
else
  echo Running job $jobid based on CANU_LOCAL_JOB_ID=$CANU_LOCAL_JOB_ID and offset=$offset.
fi

if [ $jobid -gt 97 ]; then
  echo Error: Only 97 partitions, you asked for $jobid.
  exit 1
fi

if [ $jobid -le 57 ] ; then
  tag="ctg"
else
  tag="utg"
  jobid=`expr $jobid - 57`
fi

jobid=`printf %04d $jobid`

if [ ! -d ./${tag}cns ] ; then
  mkdir -p ./${tag}cns
fi

if [ -e ./${tag}cns/$jobid.cns ] ; then
  exit 0
fi



$bin/utgcns \
  -S ../ONT.${tag}Store/partitionedReads.seqStore \
  -T ../ONT.${tag}Store 1 $jobid \
  -O ./${tag}cns/$jobid.cns.WORKING \
  -maxcoverage 40 \
  -e 0.25 \
  -pbdagcon \
  -edlib    \
  -threads 8 \
&& \
mv ./${tag}cns/$jobid.cns.WORKING ./${tag}cns/$jobid.cns \


exit 0

brianwalenz · 2019-04-12T00:13:41Z

I think the problem is already fixed, but I don't want you updating to the latest code due to potential changes in data files.

Instead, disable the failing feature by changing -pbdagcon at the bottom of consensus.sh (its an option to utgcns) to -norealign, then rerun these two jobs by hand, without restarting canu.

sh ./consensus.sh 1
sh ./consensus.sh 6

When those finish (successfully), restart canu.

Lcornet · 2019-04-12T14:09:21Z

I turn -pbdagcon to -norealign in consensus.sh :

$bin/utgcns \
  -S ../ONT.${tag}Store/partitionedReads.seqStore \
  -T ../ONT.${tag}Store 1 $jobid \
  -O ./${tag}cns/$jobid.cns.WORKING \
  -maxcoverage 40 \
  -e 0.25 \
  -norealign \
  -edlib    \
  -threads 8 \
&& \
mv ./${tag}cns/$jobid.cns.WORKING ./${tag}cns/$jobid.cns \

But I still have errors in the unitigging/5-consensus/*1.out and unitigging/5-consensus/*6.out :
#1.out

Running job 1 based on command line options.
-- Opening seqStore '../ONT.ctgStore/partitionedReads.seqStore' partition 1.
-- Opening tigStore '../ONT.ctgStore' version 1.
-- Opening output results file './ctgcns/0001.cns.WORKING'.
--
-- Computing consensus for b=0 to e=324849 with errorRate 0.2500 (max 0.4000) and minimum overlap 40
--
generateTemplateStitch()-- FAILED to align - no more template to remove!  Fail!

But I got the message succesfully completed at the end of both.
I restart Canu like this (it skip consensus.sh 1, 3 and 6 to begin at 7) :
#canu
/media/vol2/scratch/lcornet/cassava/CANU-1.8/canu/Linux-amd64/bin/canu -assemble -nanopore-corrected ONT/ONT.trimmedReads.fasta.gz genomeSize=0.750g
maxMemory=455 maxThreads=59 stopOnLowCoverage=5 cnsErrorRate=0.25 -p ONT -d ONT
useGrid=false

The consensus strill crash :

-- Using slow alignment for consensus (iteration '2').
-- Configured 57 contig and 40 unitig consensus jobs.
--
-- Consensus jobs failed, tried 2 times, giving up.
--   job ctgcns/0007.cns FAILED.
...

brianwalenz · 2019-04-12T14:32:30Z

The generateTemplateStitch() message can be ignored. Those two jobs are OK.

Your original comment says (quote)

-- Consensus jobs failed, tried 2 times, giving up.
--   job ctgcns/0001.cns FAILED.
--   job ctgcns/0006.cns FAILED.
...
...

were there only two failed jobs, or more? You're probably having the same issue on all jobs. The error you're looking for is

utgcns: utgcns/unitigConsensus.C:989: void unitigConsensus::findCoordinates(): Assertion `bgn < end' failed.

Lcornet · 2019-04-12T14:40:51Z

Yes I have the same error for all cns jobs :
-- Consensus jobs failed, tried 2 times, giving up.
-- job ctgcns/0007.cns FAILED.
-- job ctgcns/0011.cns FAILED.
-- job ctgcns/0012.cns FAILED.
-- job ctgcns/0013.cns FAILED.
-- job ctgcns/0014.cns FAILED.
-- job ctgcns/0017.cns FAILED.
-- job ctgcns/0018.cns FAILED.
-- job ctgcns/0019.cns FAILED.
-- job ctgcns/0022.cns FAILED.
-- job ctgcns/0023.cns FAILED.
-- job ctgcns/0024.cns FAILED.
-- job ctgcns/0025.cns FAILED.
-- job ctgcns/0027.cns FAILED.
-- job ctgcns/0028.cns FAILED.
-- job ctgcns/0029.cns FAILED.
-- job ctgcns/0031.cns FAILED.
-- job ctgcns/0032.cns FAILED.
-- job ctgcns/0033.cns FAILED.
-- job ctgcns/0034.cns FAILED.
-- job ctgcns/0035.cns FAILED.
-- job ctgcns/0036.cns FAILED.
-- job ctgcns/0037.cns FAILED.
-- job ctgcns/0055.cns FAILED.
-- job ctgcns/0057.cns FAILED.
-- job utgcns/0002.cns FAILED.
-- job utgcns/0003.cns FAILED.
-- job utgcns/0004.cns FAILED.
-- job utgcns/0005.cns FAILED.
-- job utgcns/0006.cns FAILED.
-- job utgcns/0008.cns FAILED.
-- job utgcns/0010.cns FAILED.
-- job utgcns/0011.cns FAILED.
-- job utgcns/0013.cns FAILED.
-- job utgcns/0014.cns FAILED.
-- job utgcns/0015.cns FAILED.
-- job utgcns/0018.cns FAILED.
-- job utgcns/0020.cns FAILED.
-- job utgcns/0022.cns FAILED.
-- job utgcns/0023.cns FAILED.
-- job utgcns/0024.cns FAILED.
-- job utgcns/0025.cns FAILED.
-- job utgcns/0026.cns FAILED.
-- job utgcns/0027.cns FAILED.
-- job utgcns/0028.cns FAILED.
-- job utgcns/0029.cns FAILED.
-- job utgcns/0030.cns FAILED.
-- job utgcns/0031.cns FAILED.
-- job utgcns/0032.cns FAILED.
-- job utgcns/0033.cns FAILED.
-- job utgcns/0034.cns FAILED.
-- job utgcns/0038.cns FAILED.

ABORT:
ABORT: Canu snapshot v1.8 +90 changes (r9300 27ebc98)
ABORT: Don't panic, but a mostly harmless error occurred and Canu stopped.
ABORT: Try restarting. If that doesn't work, ask for help.
ABORT:

I found the utgcns error in 53 files (out of 97)
$ grep unitigConsensus.C:989 *.out | cut -f1 -d":" | sort -f | uniq | wc -l
53

brianwalenz · 2019-04-12T15:11:05Z

Higher than expected.

Three solutions to choose from:

run consensus.sh manually for every job. it won't redo jobs that are already finished, and you can just rerun for all 1 through 97.
modify src/canu/pipelines/Consensus.pm to change pbdagcon to norealign, rebuild canu, and restart.
chance a git pull to grab the latest bits. the risk here is that binary file formats changed and it'll just fail; or that intermediate file names changed and it'll want to recompute something. i don't recall either of these happening since your version, but i'm not 100% sure.

Option 2 is the simplest.

Lcornet · 2019-04-12T22:23:03Z

Finally, I installed the latest code from the git and restart CANU, it solved my problems.
Thanks for the help

rodriguez-migs10 · 2021-08-23T01:36:34Z

I think the problem is already fixed, but I don't want you updating to the latest code due to potential changes in data files.

Instead, disable the failing feature by changing -pbdagcon at the bottom of consensus.sh (its an option to utgcns) to -norealign, then rerun these two jobs by hand, without restarting canu.
sh ./consensus.sh 1
sh ./consensus.sh 6
When those finish (successfully), restart canu.

I used this and it worked great for me. However, how exactly is changing -pbdagcon to -norealign? I am still very new at this so pardon if the answer is obvious.

Lcornet closed this as completed Apr 12, 2019

skoren mentioned this issue May 16, 2019

abort during consensus jobs #1364

Closed

joannarifkin mentioned this issue Aug 6, 2019

No grid engine detected, grid disabled. #1435

Closed

rozaimirazali mentioned this issue Sep 16, 2019

Error during Unitig Consensus - " Assertion `bgn < end' failed " #1483

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Consensus Failed #1325

Consensus Failed #1325

Lcornet commented Apr 11, 2019 •

edited by skoren

Loading

skoren commented Apr 11, 2019

Lcornet commented Apr 11, 2019 •

edited by brianwalenz

Loading

brianwalenz commented Apr 12, 2019

Lcornet commented Apr 12, 2019 •

edited by brianwalenz

Loading

brianwalenz commented Apr 12, 2019

Lcornet commented Apr 12, 2019

brianwalenz commented Apr 12, 2019

Lcornet commented Apr 12, 2019

rodriguez-migs10 commented Aug 23, 2021

Consensus Failed #1325

Consensus Failed #1325

Comments

Lcornet commented Apr 11, 2019 • edited by skoren Loading

skoren commented Apr 11, 2019

Lcornet commented Apr 11, 2019 • edited by brianwalenz Loading

brianwalenz commented Apr 12, 2019

Lcornet commented Apr 12, 2019 • edited by brianwalenz Loading

brianwalenz commented Apr 12, 2019

Lcornet commented Apr 12, 2019

brianwalenz commented Apr 12, 2019

Lcornet commented Apr 12, 2019

rodriguez-migs10 commented Aug 23, 2021

Lcornet commented Apr 11, 2019 •

edited by skoren

Loading

Lcornet commented Apr 11, 2019 •

edited by brianwalenz

Loading

Lcornet commented Apr 12, 2019 •

edited by brianwalenz

Loading