Two bugs for Nextpolish v1.2.4

[cassytha]

Describe the bug
(1) bug1: Make does not go through bwa compilation with my Arch linux workstation, but not with another CentOS computer.

Solution, I replace the source folder with bwa from git clone (git clone https://github.com/lh3/bwa.git), and make is successful now. bwa folder from https://nchc.dl.sourceforge.net/project/bio-bwa/bwa-0.7.17.tar.bz2 does not work.

(2)bug2: test is not successful when I run ./nextPolish test_data/run.cfg. If long nanopore reads are not used, error info disappears and test runs through.

Solution, use nextpolish v1.2.1 instead. Other versions have problems with long-read corrections.

Error message
(1) Bug1

[hui@archlinux NextPolish]$ make -j 72
mkdir /home/hui/software/NextPolish/test5/test4/NextPolish/bin
make -C util;
make[1]: 警告: jobserver 不可用: 正使用 -j1。添加 “+” 到父 make 的规则。
make[1]: 进入目录“/home/hui/software/NextPolish/test5/test4/NextPolish/util”
gcc -Wall -O3 -s -pthread -o seq_split seq_split.c thpool.c -lz
seq_split.c: 在函数‘get_fp_index’中:
seq_split.c:74:6: 警告：overflow in conversion from ‘uint32_t’ {或称 ‘unsigned int’} to ‘int’ changes value from ‘count = 4294967295’ to ‘-1’ [-Woverflow]
74 | k = count = -1;
| ^~~~~
seq_split.c: 在函数‘main’中:
seq_split.c:369:25: 警告：‘sprintf’ may write a terminating nul past the end of the destination [-Wformat-overflow=]
369 | sprintf(opt.out, "%s/%s", opt.outdir, opt.outpre);
| ^
seq_split.c:369:2: 附注：‘sprintf’ output 2 or more bytes (assuming 1025) into a destination of size 1024
369 | sprintf(opt.out, "%s/%s", opt.outdir, opt.outpre);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
seq_split.c:158:19: 警告：‘%03d’ directive writing between 3 and 10 bytes into a region of size between 0 and 1023 [-Wformat-overflow=]
158 | sprintf(fn, "%s.%03d.%s", opt->out, i, suffix);
| ^~~~
seq_split.c:158:15: 附注：directive argument in the range [0, 2147483647]
158 | sprintf(fn, "%s.%03d.%s", opt->out, i, suffix);
| ^~~~~~~~~~~~
seq_split.c:158:3: 附注：‘sprintf’ output between 14 and 1044 bytes into a destination of size 1024
158 | sprintf(fn, "%s.%03d.%s", opt->out, i, suffix);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
gcc -Wall -O3 -s -std=c99 -o seq_count seq_count.c -lz
make[2]: 进入目录“/home/hui/software/NextPolish/test5/test4/NextPolish/util/bwa”
gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS utils.c -o utils.o
gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS kthread.c -o kthread.o
gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS kstring.c -o kstring.o
gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS ksw.c -o ksw.o
gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS bwt.c -o bwt.o
gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS bntseq.c -o bntseq.o
gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS bwa.c -o bwa.o
gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS bwamem.c -o bwamem.o
gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS bwamem_pair.c -o bwamem_pair.o
gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS bwamem_extra.c -o bwamem_extra.o
gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS malloc_wrap.c -o malloc_wrap.o
gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS QSufSort.c -o QSufSort.o
gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS bwt_gen.c -o bwt_gen.o
bwt_gen.c: 在函数‘BWTIncBuildRelativeRank’中:
bwt_gen.c:879:10: 警告：变量‘oldInverseSa0RelativeRank’被设定但未被使用 [-Wunused-but-set-variable]
879 | bgint_t oldInverseSa0RelativeRank = 0;
| ^~~~~~~~~~~~~~~~~~~~~~~~~
bwt_gen.c: 在函数‘BWTIncMergeBwt’中:
bwt_gen.c:953:15: 警告：变量‘bitsInWordMinusBitPerChar’被设定但未被使用 [-Wunused-but-set-variable]
953 | unsigned int bitsInWordMinusBitPerChar;
| ^~~~~~~~~~~~~~~~~~~~~~~~~
gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS rope.c -o rope.o
gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS rle.c -o rle.o
gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS is.c -o is.o
gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS bwtindex.c -o bwtindex.o
ar -csru libbwa.a utils.o kthread.o kstring.o ksw.o bwt.o bntseq.o bwa.o bwamem.o bwamem_pair.o bwamem_extra.o malloc_wrap.o QSufSort.o bwt_gen.o rope.o rle.o is.o bwtindex.o
ar: u' 修饰符被忽略，因为 D' 为默认（参见 U'） gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS bwashm.c -o bwashm.o gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS bwase.c -o bwase.o gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS bwaseqio.c -o bwaseqio.o gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS bwtgap.c -o bwtgap.o gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS bwtaln.c -o bwtaln.o gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS bamlite.c -o bamlite.o gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS bwape.c -o bwape.o gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS kopen.c -o kopen.o gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS pemerge.c -o pemerge.o gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS maxk.c -o maxk.o gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS bwtsw2_core.c -o bwtsw2_core.o gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS bwtsw2_main.c -o bwtsw2_main.o gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS bwtsw2_aux.c -o bwtsw2_aux.o gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS bwt_lite.c -o bwt_lite.o gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS bwtsw2_chain.c -o bwtsw2_chain.o gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS fastmap.c -o fastmap.o gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS bwtsw2_pair.c -o bwtsw2_pair.o gcc -c -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS main.c -o main.o gcc -g -Wall -Wno-unused-function -O2 -DHAVE_PTHREAD -DUSE_MALLOC_WRAPPERS bwashm.o bwase.o bwaseqio.o bwtgap.o bwtaln.o bamlite.o bwape.o kopen.o pemerge.o maxk.o bwtsw2_core.o bwtsw2_main.o bwtsw2_aux.o bwt_lite.o bwtsw2_chain.o fastmap.o bwtsw2_pair.o main.o -o bwa -L. -lbwa -lm -lz -lpthread -lrt /usr/bin/ld: ./libbwa.a(rope.o):/home/hui/software/NextPolish/test5/test4/NextPolish/util/bwa/rle.h:33: multiple definition of rle_auxtab'; ./libbwa.a(bwtindex.o):/home/hui/software/NextPolish/test5/test4/NextPolish/util/bwa/rle.h:33: first defined here
/usr/bin/ld: ./libbwa.a(rle.o):/home/hui/software/NextPolish/test5/test4/NextPolish/util/bwa/rle.h:33: multiple definition of `rle_auxtab'; ./libbwa.a(bwtindex.o):/home/hui/software/NextPolish/test5/test4/NextPolish/util/bwa/rle.h:33: first defined here
collect2: 错误：ld 返回 1
make[2]: *** [Makefile:30：bwa] 错误 1
make[2]: 离开目录“/home/hui/software/NextPolish/test5/test4/NextPolish/util/bwa”
make[1]: *** [Makefile:19：bwa_] 错误 2
make[1]: 离开目录“/home/hui/software/NextPolish/test5/test4/NextPolish/util”
make: *** [Makefile:18：all] 错误 2
[hui@archlinux NextPolish]$

(2) Bug2
[hui@archlinux NextPolish]$ ./nextPolish test_data/run.cfg
[INFO] 2020-07-28 01:30:46,296 start...
[INFO] 2020-07-28 01:30:46,297 logfile: pid1443877.log.info
[WARNING] 2020-07-28 01:30:46,298 Re-write workdir
[INFO] 2020-07-28 01:30:46,303 scheduled tasks:
[5, 1, 2]
[INFO] 2020-07-28 01:30:46,303 options:
[INFO] 2020-07-28 01:30:46,303 {'polish_options': '-p 3', 'rewrite': 1, 'job_prefix': 'nextPolish', 'job_type': 'local', 'cluster_options': '', 'snp_valid': '/home/hui/software/NextPolish/test5/test5/NextPolish/test_data/./01_rundir/%02d.snp_valid', 'kmer_count': '/home/hui/software/NextPolish/test5/test5/NextPolish/test_data/./01_rundir/%02d.kmer_count', 'lgs_fofn': '/home/hui/software/NextPolish/test5/test5/NextPolish/test_data/./lgs.fofn', 'sgs_max_depth': '100', 'align_threads': 3, 'sgs_block_size': 5556450L, 'lgs_max_read_len': '150k', 'parallel_jobs': '2', 'multithread_jobs': '3', 'snp_phase': '/home/hui/software/NextPolish/test5/test5/NextPolish/test_data/./01_rundir/%02d.snp_phase', 'genome': '/home/hui/software/NextPolish/test5/test5/NextPolish/test_data/./raw.genome.fasta', 'genome_size': 111129L, 'workdir': '/home/hui/software/NextPolish/test5/test5/NextPolish/test_data/./01_rundir', 'cleantmp': 0, 'sgs_align_options': 'bwa mem -p -t 3', 'sgs_unpaired': '0', 'sgs_fofn': '/home/hui/software/NextPolish/test5/test5/NextPolish/test_data/./sgs.fofn', 'lgs_polish': '/home/hui/software/NextPolish/test5/test5/NextPolish/test_data/./01_rundir/%02d.lgs_polish', 'sgs_use_duplicate_reads': 0, 'score_chain': '/home/hui/software/NextPolish/test5/test5/NextPolish/test_data/./01_rundir/%02d.score_chain', 'task': [5, 1, 2], 'lgs_max_depth': '60', 'lgs_block_size': 3333870L, 'lgs_minimap2_options': '-x map-ont -t 3', 'rerun': 3, 'lgs_min_read_len': '1k'}
[INFO] 2020-07-28 01:30:46,304 step 0 and task 5 start:
[INFO] 2020-07-28 01:30:46,305 analysis tasks done
[INFO] 2020-07-28 01:30:46,309 total jobs: 4
[INFO] 2020-07-28 01:30:46,311 Throw jobID:[1443910] jobCmd:[/home/hui/software/NextPolish/test5/test5/NextPolish/test_data/01_rundir/00.lgs_polish/01.db_split.sh.work/db_split0/nextPolish.sh] in the local_cycle.
[INFO] 2020-07-28 01:30:46,814 Throw jobID:[1443918] jobCmd:[/home/hui/software/NextPolish/test5/test5/NextPolish/test_data/01_rundir/00.lgs_polish/01.db_split.sh.work/db_split1/nextPolish.sh] in the local_cycle.
[INFO] 2020-07-28 01:30:49,124 Throw jobID:[1443927] jobCmd:[/home/hui/software/NextPolish/test5/test5/NextPolish/test_data/01_rundir/00.lgs_polish/01.db_split.sh.work/db_split2/nextPolish.sh] in the local_cycle.
[INFO] 2020-07-28 01:30:49,676 Throw jobID:[1443940] jobCmd:[/home/hui/software/NextPolish/test5/test5/NextPolish/test_data/01_rundir/00.lgs_polish/01.db_split.sh.work/db_split3/nextPolish.sh] in the local_cycle.
[INFO] 2020-07-28 01:30:50,687 db_split done
[INFO] 2020-07-28 01:30:50,688 analysis tasks done
[INFO] 2020-07-28 01:30:50,691 total jobs: 2
[INFO] 2020-07-28 01:30:50,693 Throw jobID:[1443947] jobCmd:[/home/hui/software/NextPolish/test5/test5/NextPolish/test_data/01_rundir/00.lgs_polish/02.map.ref.sh.work/map_genome0/nextPolish.sh] in the local_cycle.
[INFO] 2020-07-28 01:30:51,196 Throw jobID:[1443959] jobCmd:[/home/hui/software/NextPolish/test5/test5/NextPolish/test_data/01_rundir/00.lgs_polish/02.map.ref.sh.work/map_genome1/nextPolish.sh] in the local_cycle.
[INFO] 2020-07-28 01:30:53,313 align_genome done
[INFO] 2020-07-28 01:30:53,314 analysis tasks done
[INFO] 2020-07-28 01:30:53,317 total jobs: 1
[INFO] 2020-07-28 01:30:53,319 Throw jobID:[1444012] jobCmd:[/home/hui/software/NextPolish/test5/test5/NextPolish/test_data/01_rundir/00.lgs_polish/03.merge.bam.sh.work/merge_bam0/nextPolish.sh] in the local_cycle.
[INFO] 2020-07-28 01:30:54,974 merge_bam done
[INFO] 2020-07-28 01:30:54,975 analysis tasks done
[INFO] 2020-07-28 01:30:54,979 total jobs: 2
[INFO] 2020-07-28 01:30:54,981 Throw jobID:[1444026] jobCmd:[/home/hui/software/NextPolish/test5/test5/NextPolish/test_data/01_rundir/00.lgs_polish/04.polish.ref.sh.work/polish_genome0/nextPolish.sh] in the local_cycle.
[INFO] 2020-07-28 01:30:55,483 Throw jobID:[1444035] jobCmd:[/home/hui/software/NextPolish/test5/test5/NextPolish/test_data/01_rundir/00.lgs_polish/04.polish.ref.sh.work/polish_genome1/nextPolish.sh] in the local_cycle.
[ERROR] 2020-07-28 01:30:56,672 polish_genome failed: please check the following logs:
[ERROR] 2020-07-28 01:30:56,672 /home/hui/software/NextPolish/test5/test5/NextPolish/test_data/01_rundir/00.lgs_polish/04.polish.ref.sh.work/polish_genome0/nextPolish.sh.e
[ERROR] 2020-07-28 01:30:56,672 /home/hui/software/NextPolish/test5/test5/NextPolish/test_data/01_rundir/00.lgs_polish/04.polish.ref.sh.work/polish_genome1/nextPolish.sh.e
[hui@archlinux NextPolish]$less /home/hui/software/NextPolish/test5/test5/NextPolish/test_data/01_rundir/00.lgs_polish/04.polish.ref.sh.work/polish_genome0/nextPolish.sh.e

hostname

• hostname
  cd /home/hui/software/NextPolish/test5/test5/NextPolish/test_data/01_rundir/00.lgs_polish/04.polish.ref.sh.work/polish_genome0
• cd /home/hui/software/NextPolish/test5/test5/NextPolish/test_data/01_rundir/00.lgs_polish/04.polish.ref.sh.work/polish_genome0
  /home/hui/.pyenv/versions/2.7.14/bin/python /home/hui/software/NextPolish/test5/test5/NextPolish/lib/nextpolish2.py -s -p 3 -g /home/hui/software/NextPolish/test5/test5/NextPolish/test_data/./01_rundir/00.lgs_polish/input.genome.fasta -b /home/hui/software/NextPolish/test5/test5/NextPolish/test_data/./01_rundir/00.lgs_polish/input.genome.fasta.blc -i 0 -l /home/hui/software/NextPolish/test5/test5/NextPolish/test_data/./01_rundir/00.lgs_polish/lgs.sort.bam.list -r ont -o genome.nextpolish.part000.fasta
• /home/hui/.pyenv/versions/2.7.14/bin/python /home/hui/software/NextPolish/test5/test5/NextPolish/lib/nextpolish2.py -s -p 3 -g /home/hui/software/NextPolish/test5/test5/NextPolish/test_data/./01_rundir/00.lgs_polish/input.genome.fasta -b /home/hui/software/NextPolish/test5/test5/NextPolish/test_data/./01_rundir/00.lgs_polish/input.genome.fasta.blc -i 0 -l /home/hui/software/NextPolish/test5/test5/NextPolish/test_data/./01_rundir/00.lgs_polish/lgs.sort.bam.list -r ont -o genome.nextpolish.part000.fasta
  [INFO] 2020-07-28 01:30:55,054 Corrected step options:
  [INFO] 2020-07-28 01:30:55,054 Namespace(alignment_identity_ratio=0.8, alignment_score_ratio=0.8, auto=True, bam_list='/home/hui/software/NextPolish/test5/test5/NextPolish/test_data/./01_rundir/00.lgs_polish/lgs.sort.bam.list', block='/home/hui/software/NextPolish/test5/test5/NextPolish/test_data/./01_rundir/00.lgs_polish/input.genome.fasta.blc', block_index='0', genome='/home/hui/software/NextPolish/test5/test5/NextPolish/test_data/./01_rundir/00.lgs_polish/input.genome.fasta', out='genome.nextpolish.part000.fasta', process=3, read_type=1, split=0, window=5000000)
  /home/hui/software/NextPolish/test5/test5/NextPolish/test_data/01_rundir/00.lgs_polish/04.polish.ref.sh.work/polish_genome0/nextPolish.sh：行 5: 1444030 段错误 （核心已转储）/home/hui/.pyenv/versions/2.7.14/bin/python /home/hui/software/NextPolish/test5/test5/NextPolish/lib/nextpolish2.py -s -p 3 -g /home/hui/software/NextPolish/test5/test5/NextPolish/test_data/./01_rundir/00.lgs_polish/input.genome.fasta -b /home/hui/software/NextPolish/test5/test5/NextPolish/test_data/./01_rundir/00.lgs_polish/input.genome.fasta.blc -i 0 -l /home/hui/software/NextPolish/test5/test5/NextPolish/test_data/./01_rundir/00.lgs_polish/lgs.sort.bam.list -r ont -o genome.nextpolish.part000.fasta

Operating system
[hui@archlinux NextPolish]$ uname -a
Linux archlinux 5.7.5-arch1-1 #1 SMP PREEMPT Mon, 22 Jun 2020 08:10:02 +0000 x86_64 GNU/Linux

GCC
[hui@archlinux NextPolish]$ gcc -v
使用内建 specs。
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-pc-linux-gnu/10.1.0/lto-wrapper
目标：x86_64-pc-linux-gnu
配置为：/build/gcc/src/gcc/configure --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=https://bugs.archlinux.org/ --enable-languages=c,c++,ada,fortran,go,lto,objc,obj-c++,d --with-isl --with-linker-hash-style=gnu --with-system-zlib --enable-__cxa_atexit --enable-cet=auto --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-default-ssp --enable-gnu-indirect-function --enable-gnu-unique-object --enable-install-libiberty --enable-linker-build-id --enable-lto --enable-multilib --enable-plugin --enable-shared --enable-threads=posix --disable-libssp --disable-libstdcxx-pch --disable-libunwind-exceptions --disable-werror gdc_include_dir=/usr/include/dlang/gdc
线程模型：posix
Supported LTO compression algorithms: zlib zstd
gcc 版本 10.1.0 (GCC)

Python
I am using pyenv, so I tried both python 2.7.14 and python 3.8.0.

NextPolish
v1.2.4

To Reproduce (Optional)
Steps to reproduce the behavior. Providing a minimal test dataset on which we can reproduce the behavior will generally lead to quicker turnaround time!

Additional context (Optional)
Add any other context about the problem here.