Using NanoCaller with BED files and Indel Calling Gives Empty VCF files

[hkarakurt8742]

Hello everyone,
Thank you for developing NanoCaller. I have multiple questions about the usage.
First, when I give a BED file to Nanocaller using --bed parameter, it takes really long compared to run without BED file. Is it normal?
And when I use "--exclude_bed hg19" I have messages for SNP calling in terminal as:
[W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option?

I downloaded the Docker image of NanoCaller and tried to test it with ONT Case Study you provided.

I ran NanoCaller with this code:

sudo docker run -v ${PWD}:'/input/' genomicslab/nanocaller:3.0.0 NanoCaller --bam /input/nanocall.bam --ref /input/GRCh38.fa --prefix HG002 --preset ont --output /input/calls --cpu 15 --exclude_bed hg38 --wgs_contigs chr1-22XY

After installation I used the code to call SNPs and Indels. SNPs are working okay. But Indel calling have this error message:

`Indel Calling Progress: 0%| | 0/30970 [00:00<?, ?it/s]

[E::hts_open_format] Failed to open file "vcf_files_nanocaller/FAB42395_fast/intermediate_phase_files/chr9.phased.bam" : No such file or directory
Process Process-18:
Traceback (most recent call last):
File "/home/huk/anaconda3/envs/nanocaller_env/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/home/huk/anaconda3/envs/nanocaller_env/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/huk/NanoCaller/nanocaller_src/indelCaller.py", line 200, in caller
indel_run(params, indel_dict, job_Q, counter_Q, indel_files_list)
File "/home/huk/NanoCaller/nanocaller_src/indelCaller.py", line 59, in indel_run
pos, x0_test, x1_test, x2_test, alleles_seq, phase = get_indel_testing_candidates(params, chunk)
File "/home/huk/NanoCaller/nanocaller_src/generate_indel_pileups.py", line 147, in get_indel_testing_candidates
samfile = pysam.Samfile(sam_path, "rb")
File "pysam/libcalignmentfile.pyx", line 751, in pysam.libcalignmentfile.AlignmentFile.cinit
File "pysam/libcalignmentfile.pyx", line 950, in pysam.libcalignmentfile.AlignmentFile._open
FileNotFoundError: [Errno 2] could not open alignment file vcf_files_nanocaller/FAB42395_fast/intermediate_phase_files/chr9.phased.bam: No such file or directory
[E::hts_open_format] Failed to open file "vcf_files_nanocaller/FAB42395_fast/intermediate_phase_files/chr9.phased.bam" : No such file or directory
Process Process-28:
Traceback (most recent call last):
File "/home/huk/anaconda3/envs/nanocaller_env/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/home/huk/anaconda3/envs/nanocaller_env/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/huk/NanoCaller/nanocaller_src/indelCaller.py", line 200, in caller
indel_run(params, indel_dict, job_Q, counter_Q, indel_files_list)
File "/home/huk/NanoCaller/nanocaller_src/indelCaller.py", line 59, in indel_run
pos, x0_test, x1_test, x2_test, alleles_seq, phase = get_indel_testing_candidates(params, chunk)
File "/home/huk/NanoCaller/nanocaller_src/generate_indel_pileups.py", line 147, in get_indel_testing_candidates
samfile = pysam.Samfile(sam_path, "rb")
File "pysam/libcalignmentfile.pyx", line 751, in pysam.libcalignmentfile.AlignmentFile.cinit
File "pysam/libcalignmentfile.pyx", line 950, in pysam.libcalignmentfile.AlignmentFile._open
FileNotFoundError: [Errno 2] could not open alignment file vcf_files_nanocaller/FAB42395_fast/intermediate_phase_files/chr9.phased.bam: No such file or directory

[E::hts_open_format] Failed to open file "vcf_files_nanocaller/FAB42395_fast/intermediate_phase_files/chr9.phased.bam" : No such file or directory
Process Process-26:
Traceback (most recent call last):
File "/home/huk/anaconda3/envs/nanocaller_env/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/home/huk/anaconda3/envs/nanocaller_env/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/huk/NanoCaller/nanocaller_src/indelCaller.py", line 200, in caller
indel_run(params, indel_dict, job_Q, counter_Q, indel_files_list)
File "/home/huk/NanoCaller/nanocaller_src/indelCaller.py", line 59, in indel_run
pos, x0_test, x1_test, x2_test, alleles_seq, phase = get_indel_testing_candidates(params, chunk)
File "/home/huk/NanoCaller/nanocaller_src/generate_indel_pileups.py", line 147, in get_indel_testing_candidates
samfile = pysam.Samfile(sam_path, "rb")
File "pysam/libcalignmentfile.pyx", line 751, in pysam.libcalignmentfile.AlignmentFile.cinit
File "pysam/libcalignmentfile.pyx", line 950, in pysam.libcalignmentfile.AlignmentFile._open
FileNotFoundError: [Errno 2] could not open alignment file vcf_files_nanocaller/FAB42395_fast/intermediate_phase_files/chr9.phased.bam: No such file or directory

[E::hts_open_format] Failed to open file "vcf_files_nanocaller/FAB42395_fast/intermediate_phase_files/chr9.phased.bam" : No such file or directory
Process Process-23:
Traceback (most recent call last):
File "/home/huk/anaconda3/envs/nanocaller_env/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/home/huk/anaconda3/envs/nanocaller_env/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/huk/NanoCaller/nanocaller_src/indelCaller.py", line 200, in caller
indel_run(params, indel_dict, job_Q, counter_Q, indel_files_list)
File "/home/huk/NanoCaller/nanocaller_src/indelCaller.py", line 59, in indel_run
pos, x0_test, x1_test, x2_test, alleles_seq, phase = get_indel_testing_candidates(params, chunk)
File "/home/huk/NanoCaller/nanocaller_src/generate_indel_pileups.py", line 147, in get_indel_testing_candidates
samfile = pysam.Samfile(sam_path, "rb")
File "pysam/libcalignmentfile.pyx", line 751, in pysam.libcalignmentfile.AlignmentFile.cinit
File "pysam/libcalignmentfile.pyx", line 950, in pysam.libcalignmentfile.AlignmentFile._open
FileNotFoundError: [Errno 2] could not open alignment file vcf_files_nanocaller/FAB42395_fast/intermediate_phase_files/chr9.phased.bam: No such file or directory

[E::hts_open_format] Failed to open file "vcf_files_nanocaller/FAB42395_fast/intermediate_phase_files/chr9.phased.bam" : No such file or directory
Process Process-21:
Traceback (most recent call last):
File "/home/huk/anaconda3/envs/nanocaller_env/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/home/huk/anaconda3/envs/nanocaller_env/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/huk/NanoCaller/nanocaller_src/indelCaller.py", line 200, in caller
indel_run(params, indel_dict, job_Q, counter_Q, indel_files_list)
File "/home/huk/NanoCaller/nanocaller_src/indelCaller.py", line 59, in indel_run
pos, x0_test, x1_test, x2_test, alleles_seq, phase = get_indel_testing_candidates(params, chunk)
File "/home/huk/NanoCaller/nanocaller_src/generate_indel_pileups.py", line 147, in get_indel_testing_candidates
samfile = pysam.Samfile(sam_path, "rb")
File "pysam/libcalignmentfile.pyx", line 751, in pysam.libcalignmentfile.AlignmentFile.cinit
File "pysam/libcalignmentfile.pyx", line 950, in pysam.libcalignmentfile.AlignmentFile._open
FileNotFoundError: [Errno 2] could not open alignment file vcf_files_nanocaller/FAB42395_fast/intermediate_phase_files/chr9.phased.bam: No such file or directory

[E::hts_open_format] Failed to open file "vcf_files_nanocaller/FAB42395_fast/intermediate_phase_files/chr9.phased.bam" : No such file or directory
Process Process-19:
Traceback (most recent call last):
File "/home/huk/anaconda3/envs/nanocaller_env/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/home/huk/anaconda3/envs/nanocaller_env/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/huk/NanoCaller/nanocaller_src/indelCaller.py", line 200, in caller
indel_run(params, indel_dict, job_Q, counter_Q, indel_files_list)
File "/home/huk/NanoCaller/nanocaller_src/indelCaller.py", line 59, in indel_run
pos, x0_test, x1_test, x2_test, alleles_seq, phase = get_indel_testing_candidates(params, chunk)
File "/home/huk/NanoCaller/nanocaller_src/generate_indel_pileups.py", line 147, in get_indel_testing_candidates
samfile = pysam.Samfile(sam_path, "rb")
File "pysam/libcalignmentfile.pyx", line 751, in pysam.libcalignmentfile.AlignmentFile.cinit
File "pysam/libcalignmentfile.pyx", line 950, in pysam.libcalignmentfile.AlignmentFile._open
FileNotFoundError: [Errno 2] could not open alignment file vcf_files_nanocaller/FAB42395_fast/intermediate_phase_files/chr9.phased.bam: No such file or directory

[E::hts_open_format] Failed to open file "vcf_files_nanocaller/FAB42395_fast/intermediate_phase_files/chr9.phased.bam" : No such file or directory
Process Process-27:
Traceback (most recent call last):
File "/home/huk/anaconda3/envs/nanocaller_env/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/home/huk/anaconda3/envs/nanocaller_env/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/huk/NanoCaller/nanocaller_src/indelCaller.py", line 200, in caller
indel_run(params, indel_dict, job_Q, counter_Q, indel_files_list)
File "/home/huk/NanoCaller/nanocaller_src/indelCaller.py", line 59, in indel_run
pos, x0_test, x1_test, x2_test, alleles_seq, phase = get_indel_testing_candidates(params, chunk)
File "/home/huk/NanoCaller/nanocaller_src/generate_indel_pileups.py", line 147, in get_indel_testing_candidates
samfile = pysam.Samfile(sam_path, "rb")
File "pysam/libcalignmentfile.pyx", line 751, in pysam.libcalignmentfile.AlignmentFile.cinit
File "pysam/libcalignmentfile.pyx", line 950, in pysam.libcalignmentfile.AlignmentFile._open
FileNotFoundError: [Errno 2] could not open alignment file vcf_files_nanocaller/FAB42395_fast/intermediate_phase_files/chr9.phased.bam: No such file or directory
[E::hts_open_format] Failed to open file "vcf_files_nanocaller/FAB42395_fast/intermediate_phase_files/chr9.phased.bam" : No such file or directory
Process Process-24:
Traceback (most recent call last):
File "/home/huk/anaconda3/envs/nanocaller_env/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/home/huk/anaconda3/envs/nanocaller_env/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/huk/NanoCaller/nanocaller_src/indelCaller.py", line 200, in caller
indel_run(params, indel_dict, job_Q, counter_Q, indel_files_list)
File "/home/huk/NanoCaller/nanocaller_src/indelCaller.py", line 59, in indel_run
pos, x0_test, x1_test, x2_test, alleles_seq, phase = get_indel_testing_candidates(params, chunk)
File "/home/huk/NanoCaller/nanocaller_src/generate_indel_pileups.py", line 147, in get_indel_testing_candidates
samfile = pysam.Samfile(sam_path, "rb")
File "pysam/libcalignmentfile.pyx", line 751, in pysam.libcalignmentfile.AlignmentFile.cinit
File "pysam/libcalignmentfile.pyx", line 950, in pysam.libcalignmentfile.AlignmentFile._open
FileNotFoundError: [Errno 2] could not open alignment file vcf_files_nanocaller/FAB42395_fast/intermediate_phase_files/chr9.phased.bam: No such file or directory
[E::hts_open_format] Failed to open file "vcf_files_nanocaller/FAB42395_fast/intermediate_phase_files/chr9.phased.bam" : No such file or directory
Process Process-20:
Traceback (most recent call last):
File "/home/huk/anaconda3/envs/nanocaller_env/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/home/huk/anaconda3/envs/nanocaller_env/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/huk/NanoCaller/nanocaller_src/indelCaller.py", line 200, in caller
indel_run(params, indel_dict, job_Q, counter_Q, indel_files_list)
File "/home/huk/NanoCaller/nanocaller_src/indelCaller.py", line 59, in indel_run
pos, x0_test, x1_test, x2_test, alleles_seq, phase = get_indel_testing_candidates(params, chunk)
File "/home/huk/NanoCaller/nanocaller_src/generate_indel_pileups.py", line 147, in get_indel_testing_candidates
samfile = pysam.Samfile(sam_path, "rb")
File "pysam/libcalignmentfile.pyx", line 751, in pysam.libcalignmentfile.AlignmentFile.cinit
File "pysam/libcalignmentfile.pyx", line 950, in pysam.libcalignmentfile.AlignmentFile._open
FileNotFoundError: [Errno 2] could not open alignment file vcf_files_nanocaller/FAB42395_fast/intermediate_phase_files/chr9.phased.bam: No such file or directory

[E::hts_open_format] Failed to open file "vcf_files_nanocaller/FAB42395_fast/intermediate_phase_files/chr9.phased.bam" : No such file or directory
Process Process-22:
Traceback (most recent call last):
File "/home/huk/anaconda3/envs/nanocaller_env/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/home/huk/anaconda3/envs/nanocaller_env/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/huk/NanoCaller/nanocaller_src/indelCaller.py", line 200, in caller
indel_run(params, indel_dict, job_Q, counter_Q, indel_files_list)
File "/home/huk/NanoCaller/nanocaller_src/indelCaller.py", line 59, in indel_run
pos, x0_test, x1_test, x2_test, alleles_seq, phase = get_indel_testing_candidates(params, chunk)
File "/home/huk/NanoCaller/nanocaller_src/generate_indel_pileups.py", line 147, in get_indel_testing_candidates
samfile = pysam.Samfile(sam_path, "rb")
File "pysam/libcalignmentfile.pyx", line 751, in pysam.libcalignmentfile.AlignmentFile.cinit
File "pysam/libcalignmentfile.pyx", line 950, in pysam.libcalignmentfile.AlignmentFile._open
FileNotFoundError: [Errno 2] could not open alignment file vcf_files_nanocaller/FAB42395_fast/intermediate_phase_files/chr9.phased.bam: No such file or directory

[E::hts_open_format] Failed to open file "vcf_files_nanocaller/FAB42395_fast/intermediate_phase_files/chr9.phased.bam" : No such file or directory
Process Process-17:
Traceback (most recent call last):
File "/home/huk/anaconda3/envs/nanocaller_env/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/home/huk/anaconda3/envs/nanocaller_env/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/huk/NanoCaller/nanocaller_src/indelCaller.py", line 200, in caller
indel_run(params, indel_dict, job_Q, counter_Q, indel_files_list)
File "/home/huk/NanoCaller/nanocaller_src/indelCaller.py", line 59, in indel_run
pos, x0_test, x1_test, x2_test, alleles_seq, phase = get_indel_testing_candidates(params, chunk)
File "/home/huk/NanoCaller/nanocaller_src/generate_indel_pileups.py", line 147, in get_indel_testing_candidates
samfile = pysam.Samfile(sam_path, "rb")
File "pysam/libcalignmentfile.pyx", line 751, in pysam.libcalignmentfile.AlignmentFile.cinit
File "pysam/libcalignmentfile.pyx", line 950, in pysam.libcalignmentfile.AlignmentFile._open
FileNotFoundError: [Errno 2] could not open alignment file vcf_files_nanocaller/FAB42395_fast/intermediate_phase_files/chr9.phased.bam: No such file or directory

[E::hts_open_format] Failed to open file "vcf_files_nanocaller/FAB42395_fast/intermediate_phase_files/chr9.phased.bam" : No such file or directory
Process Process-25:
Traceback (most recent call last):
File "/home/huk/anaconda3/envs/nanocaller_env/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/home/huk/anaconda3/envs/nanocaller_env/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/huk/NanoCaller/nanocaller_src/indelCaller.py", line 200, in caller
indel_run(params, indel_dict, job_Q, counter_Q, indel_files_list)
File "/home/huk/NanoCaller/nanocaller_src/indelCaller.py", line 59, in indel_run
pos, x0_test, x1_test, x2_test, alleles_seq, phase = get_indel_testing_candidates(params, chunk)
File "/home/huk/NanoCaller/nanocaller_src/generate_indel_pileups.py", line 147, in get_indel_testing_candidates
samfile = pysam.Samfile(sam_path, "rb")
File "pysam/libcalignmentfile.pyx", line 751, in pysam.libcalignmentfile.AlignmentFile.cinit
File "pysam/libcalignmentfile.pyx", line 950, in pysam.libcalignmentfile.AlignmentFile._open
FileNotFoundError: [Errno 2] could not open alignment file vcf_files_nanocaller/FAB42395_fast/intermediate_phase_files/chr9.phased.bam: No such file or directory
Indel Calling Progress: 0%| | 0/30970 [00:04<?, ?it/s]

2022-07-06 10:57:51.977706: Compressing and indexing indel calls.
Writing to /tmp/bcftools.NVRBLx
Lines total/split/realigned/skipped: 0/0/0/0
Merging 0 temporary files
Cleaning
Done

Checking the headers and starting positions of 2 files

2022-07-06 10:57:52.054713: Indel calling completed. Time taken= 17.7431
`

I have no idea what is the problem in my codes. Is something wrong with my codes or data?

Thank you in advance

[tuannguyen8390]

Hi there,

I ran into the same issue with my ONT data. Does this issue resolve yet?

For my case I could see a few indel vcfs in the intermediate_indel_files but the all files are empty.

Tuan

[tuannguyen8390]

I think I find the reason behind this, cross-ref with issue #1 (already closed)

The default NanoCaller workflow, which is the mode 'both', goes like this:

bam file > SNP calling by NanoCaller > SNP phasing by whatshap > Reads phasing by whatshap > Indel calling by NanoCaller on reads phased in previous step > Combine indel and SNP calls

If you choose 'indel' mode, then we do not have any SNPs that we can use to phase the reads in order to call indels. This option is allowed so that if a user prefers to use a different tool than whatshap to carry out SNPs and reads phasing, or if they already have phased reads that they believe are phased using some other reliable method, then they can bypass the whatshap subroutine by NanoCaller and supply their own phased bam file.

Another reason for this option is that if a user runs NanoCaller in 'both' mode, but the program crashes during indel calling stage for some likely scenarios such as memory exceeding system limits, then they can resume NanoCaller at the indel calling stage, instead of having to do the entire SNP calling and phasing all over again.

We have removed the option for bed files for now. We found for whole genome variant calling that using a bed file would get rid of potential variants near the boundaries of the interval in a bed file. So we found it better to leave it up to a user to filter variants with a bed file using a more sophisticated tool like rtg vcffilter. But I can see having an option like bed file can be useful if you you dont want to spend extra time in calling variants that are outside the intervals in a bed file, and submitting individual calls for each line in a bed file can be cumbersome. We will revisit this issue in our team. Do you have any preferences or thoughts about this?

We provide an example here: https://github.com/WGLab/NanoCaller/tree/master/sample You can see the command we used in the file: https://github.com/WGLab/NanoCaller/blob/master/sample/sample_call

[umahsn]

Hi,

Regarding --bed, we changed how we used the bed file. Previously we were using it to filter out candidate sites after we generate a set of candidate sites from whole genome. But now it is meant to be used as a way to specify regions in which candidate sites are going to be generated and then we open the BAM file for each bed file entry. So if there are too many lines in the BED file then it will slow down NanoCaller significantly.

Whereas for indel calling failure, it might help to see if there are any phased SNP files created so we can narrow down where the failure happens. It seems like there is a problem with generating phased BAM file from whatshap.

Tuan, can you show me the error you received? Are you able to get phased SNP files? You can get more error details by replacing this line:
https://github.com/WGLab/NanoCaller/blob/master/nanocaller_src/indelCaller.py#L178

by

run_cmd("whatshap haplotag --ignore-read-groups --ignore-linked-read --reference %s %s %s --regions %s:%d-%d --tag-supplementary -o - | samtools view -b -1 --write-index -o %s" %(params['fasta_path'], output_contig_vcf, params['sam_path'], contig, start, end, phased_bam), verbose=True, output=True, error=True)