very slow overlapInCore

[mschatz]

Following up to #1202, the error correction finished, but now is stuck in trimming. Symptoms are similar to before where most batches finish right away, but some have taken weeks with overlapInCore. Is there a way to monitor which reads it is processing and exclude any that take suspiciously long time? I also launched a separate job to trim and assemble with the corrected reads using -fast mode as a backup but would be nice to push this forward too

Thanks!
Mike

[brianwalenz]

I was about to say there's no useful logging, and was reading the code to figure out how to add some, when I discovered the never-recently-if-ever-used -l (-ell) option. This, default UINT64_MAX, claims to control how many overlaps to output per read end. When there are more than this many potential overlaps, comments claim overlaps will be processed positive diagonal longest to shortest, then negative diagonal longest to shortest, until that many overlaps are found (per end, not in total).

WARNING! Unused code path! It's not a big code path (at the bottom of overlapInCore-Process_String_Overlaps.C; -l is G.Frag_Olap_Limit) but almost certainly never used since I refactored this code.

Some experiments with minimum overlap length in the fall is convincing me that long read assembly doesn't suffer (at all) from losing overlaps to repeats at the ends of reads. Setting -l to 2x, 3x, 5x, 10x (1x?) coverage is probably safe (assuming -l actually it works). If you want to wait a bit, we can probably run a parameter sweep on dmel or arabidopsis and see what happens.

[KarinaMartins]

Dear Brian and Sergey,
I'm having the same issue. My assembly is also in trimming and I'm wondering if it is normal overlapInCore task taking so long to finish. It's the first time I'm running Canu.
I'm assembling a ~940mpb genome sequenced with PacBio using canu 1.8. Raw reads coverage was 119x. I ran correction module using default parameters (but correctedErrorRate=0.035). Correction took 7 days to finish and resulted in a coverage of 38.4x of corrected reads for trimming module.

Canu is in overlapInCore task for 10 days and it is currently processing batches 6 to 9 out of 200.
I'm running in a node 40 vCPUs, 370GB RAM, 700GB disk space. Is it normal this slow speed? Or it may be due to limited computacional resources?

Thank you
Karina

---

Starting 'obtovl' concurrent execution on Mon Jan 21 14:47:13 2019 with 554.65 GB free disk space (200 processes; 4 concurrently)

cd trimming/1-overlapper
./overlap.sh 1 > ./overlap.000001.out 2>&1
./overlap.sh 2 > ./overlap.000002.out 2>&1
./overlap.sh 3 > ./overlap.000003.out 2>&1
./overlap.sh 4 > ./overlap.000004.out 2>&1
./overlap.sh 5 > ./overlap.000005.out 2>&1
./overlap.sh 6 > ./overlap.000006.out 2>&1
./overlap.sh 7 > ./overlap.000007.out 2>&1
./overlap.sh 8 > ./overlap.000008.out 2>&1
./overlap.sh 9 > ./overlap.000009.out 2>&1

/canu_out/trimming/1-overlapper/001$ ls -ilthr
256222 -rw-rw-r-- 1 ubuntu ubuntu 1.7M Jan 22 04:33 000001.ovb
256228 -rw-rw-r-- 1 ubuntu ubuntu 42M Jan 22 04:33 000001.oc
256229 -rw-rw-r-- 1 ubuntu ubuntu 246 Jan 22 04:33 000001.stats
256225 -rw-rw-r-- 1 ubuntu ubuntu 4.8M Jan 24 07:41 000002.ovb
256236 -rw-rw-r-- 1 ubuntu ubuntu 42M Jan 24 07:41 000002.oc
256409 -rw-rw-r-- 1 ubuntu ubuntu 249 Jan 24 07:42 000002.stats
256224 -rw-rw-r-- 1 ubuntu ubuntu 7.0M Jan 25 04:16 000003.ovb
256234 -rw-rw-r-- 1 ubuntu ubuntu 42M Jan 25 04:17 000003.oc
256418 -rw-rw-r-- 1 ubuntu ubuntu 249 Jan 25 04:18 000003.stats
256226 -rw-rw-r-- 1 ubuntu ubuntu 11M Jan 26 22:43 000004.ovb
256417 -rw-rw-r-- 1 ubuntu ubuntu 42M Jan 26 22:44 000004.oc
256425 -rw-rw-r-- 1 ubuntu ubuntu 249 Jan 26 22:45 000004.stats
256233 -rw-rw-r-- 1 ubuntu ubuntu 13M Jan 27 14:35 000005.ovb
256423 -rw-rw-r-- 1 ubuntu ubuntu 42M Jan 27 14:36 000005.oc
256431 -rw-rw-r-- 1 ubuntu ubuntu 249 Jan 27 14:36 000005.stats
256429 -rw-rw-r-- 1 ubuntu ubuntu 3.7M Jan 31 07:41 000008.ovb.WORKING
256435 -rw-rw-r-- 1 ubuntu ubuntu 4.6M Jan 31 12:46 000009.ovb.WORKING
256422 -rw-rw-r-- 1 ubuntu ubuntu 5.5M Jan 31 15:09 000007.ovb.WORKING
256415 -rw-rw-r-- 1 ubuntu ubuntu 3.7M Jan 31 15:55 000006.ovb.WORKING

[skoren]

@KarinaMartins, you've got only 40 cpus for a 1gb genome, we recommend using a cluster when possible for that assembly. The issue is each of the overlap jobs wants to use 8cpus and so you're only running 4 at a time.

Are you running the 1.8 release (what does canu --version say)? You can try re-starting from trimming and adding ovlMerDistinct=0.975 (see also FAQ and issue #1232) though you don't need to worry about correctedErrRate, you're already using one lower than the default. If anything, you'd probably want to increase from 0.035 if this is Sequel data but this won't make it faster.

[KarinaMartins]

Thank you, Sergey. I'll do as you recommended.
Yes, I'm running version 1.8.
Best,
Karina

[mchaisso]

I'm also seeing some very slow overlapInCore runtimes for a cluster job. At first I thought this was due to my reads being sorted by position in chromosome, so I modified my pipeline to randomly shuffle reads before running. This is for CLR human data. When I attach to a running overlapInCore in gdb, it is in a step aligning long-ish (30-40kb) centromeric reads in both the shuffled and ordered runs. Depending on the overlapInCore format, I can swap in some pretty quick overlapping code I wrote last summer.

[skoren]

I didn't think read order should affect runtime, as I said in #1202. I think you can adjust the threshold to filter more repetitive k-mers in these reads to speed up the computation (and potentially adjust the error rate down). It's easy to substitute another program, canu already has this option via the -fast parameter but the question is how it will affect the assembly continuity.

[mchaisso]

Does -fast create options to speed up the falconsense step? After getting through the overlapInCore step with the options you provided, this is also taking quite some time.

[skoren]

No, fast is independent of falcon_sense unfortunately. Falcon_sense usually can be sped up by letting it run on scratch disk (via staging in canu or just copying data) as it does a lot of random access.

[mchaisso]

Thanks! I'll give that a try - most of the jobs run at 100% cpu over a nfs, but it won't hurt to have them work on scratch.

…

On Wed, Feb 19, 2020 at 11:36 AM Sergey Koren ***@***.***> wrote: No, fast is independent of falcon_sense unfortunately. Falcon_sense usually can be sped up by letting it run on scratch disk (via staging in canu or just copying data) as it does a lot of random access. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1222?email_source=notifications&email_token=AAKPGPRGAAI7NP7RTSDAYB3RDWC23A5CNFSM4GRSP7Y2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMJGM2Y#issuecomment-588408427>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKPGPTR6ZJ2LFYKF56KN33RDWC23ANCNFSM4GRSP7YQ> .

[cfc424]

Hi,
nohup ~/bin/canu-2.2/bin/canu -d 01.canu_all_in_one -p Arabidopsis executiveThreads=24 executiveMemory=80G genomeSize=138m -nanopore /home/debian/data/08.arabidopsis_t2t_genome/CRR302667/CRR302667.fastq.gz 2> canu_all_in_one.log &

54Gb nanopore fastq data
canu -version
canu 2.2

My sever is 28 cpus and 500G RAM， and the process has run 5 days....

By now, canu is running overlapInCore step.

I wonder if the phenomenon is normal, and I tested other methods never demanding this long time...

Thanks in adcance,
From Chao

[skoren]

Yes, this step can take a while with ONT data. There is the -fast option which should be much quicker but you might get a less continuous assembly.

[cfc424]

Yes, this step can take a while with ONT data. There is the -fast option which should be much quicker but you might get a less continuous assembly.

Ok, thanks for your reply.