Samtools sort out-of-memory

[RenscoHogers]

Describe the bug
While running NextDenovo on SLURM, I run into a samtools sort error, seemingly due to too little reserved memory.

Error message
Main task:
[1700406 INFO] 2023-05-25 08:28:29 Submitted jobID:[42186471] jobCmd:[/home/WUR/hoger006/lustre_dir/Data/Dutch_Belted/NextDenovo_out/03.ctg_graph/02.ctg_align.sh.work/ctg_align16/nextDenovo.sh] in the slurm_cycle.
[1700406 ERROR] 2023-05-25 08:55:12 ESC[35mctg_align failed: please check the following logs:ESC[0m
[1700406 ERROR] 2023-05-25 08:55:12 ESC[35m/home/WUR/hoger006/lustre_dir/Data/Dutch_Belted/NextDenovo_out/03.ctg_graph/02.ctg_align.sh.work/ctg_align01/nextDenovo.sh.eESC[0m

Subtask:
hostname

• hostname
  cd /home/WUR/hoger006/lustre_dir/Data/Dutch_Belted/NextDenovo_out/03.ctg_graph/02.ctg_align.sh.work/ctg_align01
• cd /home/WUR/hoger006/lustre_dir/Data/Dutch_Belted/NextDenovo_out/03.ctg_graph/02.ctg_align.sh.work/ctg_align01
  time /lustre/nobackup/WUR/ABGC/hoger006/Tools/mambaforge/envs/nextdenovo/lib/python3.10/site-packages/nextdenovo/bin/minimap2-nd --step 3 -x map-ont -a -t 32 /home/WUR/hoger006/lustre_dir/Data/Dutch_Belted/NextDenovo_out/03.ctg_graph/01.ctg_graph.sh.work/ctg_graph1/nd.asm.p.fasta /home/WUR/hoger006/lustre_dir/Data/Dutch_Belted/NextDenovo_out/01.raw_align/input.seed.003.2bit|/lustre/nobackup/WUR/ABGC/hoger006/Tools/mambaforge/envs/nextdenovo/lib/python3.10/site-packages/nextdenovo/bin/bam_sort -i -@ 32 -o input.seed.003.2bit.sort.bam
• • time /lustre/nobackup/WUR/ABGC/hoger006/Tools/mambaforge/envs/nextdenovo/lib/python3.10/site-packages/nextdenovo/bin/minimap2-nd/lustre/nobackup/WUR/ABGC/hoger006/Tools/mambaforge/envs/nextdenovo/lib/python3.10/site-packages/nextdenovo/bin/bam_sort --step -i 3 -@ -x 32 map-ont -o -a input.seed.003.2bit.sort.bam -t
    32 /home/WUR/hoger006/lustre_dir/Data/Dutch_Belted/NextDenovo_out/03.ctg_graph/01.ctg_graph.sh.work/ctg_graph1/nd.asm.p.fasta /home/WUR/hoger006/lustre_dir/Data/Dutch_Belted/NextDenovo_out/01.raw_align/input.seed.003.2bit
    [M::mm_idx_gen::58.8211.66] collected minimizers
    [M::mm_idx_gen::60.6662.35] sorted minimizers
    [M::main::60.6692.35] loaded/built the index for 3033 target sequence(s)
    [M::mm_mapopt_update::62.5772.31] mid_occ = 660
    [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 3033
    [M::mm_idx_stat::63.6932.29] distinct minimizers: 94498812 (41.66% are singletons); average occurrences: 4.866; average spacing: 5.331
    [M::worker_pipeline::158.26617.91] mapped 22594 sequences
    [M::worker_pipeline::239.393*22.00] mapped 22805 sequences
    Command terminated by signal 9
    5927.23user 178.00system 4:27.79elapsed 2279%CPU (0avgtext+0avgdata 33847516maxresident)k
    5931360inputs+24outputs (0major+85737664minor)pagefaults 0swaps
    [W::sam_read1] Parse error at line 113646
    samtools sort: truncated file. Aborting
    slurmstepd: error: Detected 1 oom-kill event(s) in StepId=42185229.batch. Some of your processes may have been killed by the cgroup out-of-memory handler.

Config file
[General]
job_type = slurm
submit = auto
job_prefix = nextDenovo
task = all
rewrite = yes
deltmp = yes
parallel_jobs = 32
input_type = raw
read_type = ont
input_fofn = /home/WUR/hoger006/lustre_dir/Data/Dutch_Belted/NextDenovo_out/input.fofn
workdir = /home/WUR/hoger006/lustre_dir/Data/Dutch_Belted/NextDenovo_out

[correct_option]
read_cutoff = 1K
genome_size = 827922386
sort_options = -m 100g -t 32
minimap2_options_raw = -t 32 -m 200g
pa_correction = 8
correction_options = -p 32

[assemble_option]
minimap2_options_cns = -t 32
nextgraph_options = -a 1

Operating system
Which operating system and version are you using?
Ubuntu 20.04 LTS

GCC
What version of GCC are you using?
11.2.0

Python
What version of Python are you using?
3.10.8

NextDenovo
What version of NextDenovo are you using?
2.5.2

[moold]

see here to set submit. such as sbatch --cpus-per-task={cpu} --mem-per-cpu=2g -o {out} -e {err} {script}

or set submit = sbatch --cpus-per-task={cpu} --mem-per-cpu=2g -o {out} -e {err} {script} in your config file, see details

[DaniPaulo]

Hi @RenscoHogers. I'm still trying to figure out how to run NextDenovo in a HPC environment using SLURM. Would you be able to share your script.slurm.sh with me?