bcftools can't parse CONTIG ID containing a comma

[talshmaya]

I have vcf files with this line in the header:

##reference=<ID=hs37d5, Source=ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz>

bcftools fails to parse the file due to the space after the comma (before Source=). It works when I delete the space.

I get this error:

$ ./bcftools merge file1.vcf.gz file2.vcf.gz
Could not parse the header line: "##reference=<ID=hs37d5, Source=ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz>"
Could not parse the header line: "##reference=<ID=hs37d5, Source=ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz>"
[W::vcf_parse] INFO 'DPR' is not defined in the header, assuming Type=String
[W::vcf_parse] INFO 'NREF' is not defined in the header, assuming Type=String
[W::vcf_parse] FILTER 'avr010' is not defined in the header
[W::vcf_parse] INFO 'DPR' is not defined in the header, assuming Type=String
[W::vcf_parse] FILTER 'avr010' is not defined in the header
[W::vcf_parse] INFO 'XRX' is not defined in the header, assuming Type=String
Error: The INFO field is not defined in the header: DPR

[lskatz]

This is a similar bug that I just found with @andrewdhuang. We tried bcftools concat on a set of VCFs that have a comma at the end of the seqname O103H2084006_A,. This is the error we get:

Could not parse the header line: "##contig=<ID=O103H2084006_A,,IDX=0>"

We believe that it is attempting to put the ##contig header into the output file without quoting the ID. Without quoting the ID, the comma becomes a reserved letter and indicates a missing field in this header. I believe that there should either be an error with this kind of contig name or that the contig name should be quoted somehow.

Please forgive me if this is the wrong place to report this bug.

[pd3]

Hi, this is an issue in htslib, but the report here is OK. It is now fixed by pd3/htslib@45379c2 and pd3/htslib@d04b77a.

(This test file in bcftools will need update after this is merged, in the test fills sequence names on the fly https://github.com/samtools/bcftools/blob/develop/test/isec.tab.out)

Cheers

[jmarshall]

Original bug auto-closed now that samtools/htslib#214 has been merged; re-opening this issue to track the new comma/quoting issue.

[pd3]

I believe this is now solved, please reopen if not.

[zpingfeng]

I run bcftools on the vcf files generated from freebayes, but got problems on parsing the header lines (see below commands and error messages). I used bcftools version 1.8, but sorted with samtools version 1.3. Is it a problem?

bcftools consensus -f ../../Euc_RefSeq.fas -I test.vcf.gz -o test.unmask.fa

[E::bcf_hdr_parse_line] Could not parse the header line: "##contig=<ID=Egrandis_33521_Branched_chain_aminotransferase_BCAT1,_pyridoxal_phosphate_enzymes_type_IV_superfamily,length=1650>"

[W::bcf_hdr_parse] Could not parse header line: ##contig=<ID=Egrandis_33521_Branched_chain_aminotransferase_BCAT1,_pyridoxal_phosphate_enzymes_type_IV_superfamily,length=1650>

[E::bcf_hdr_parse_line] Could not parse the header line: "##contig=<ID=Egrandis_contig_40711_Uncharacterized_membrane_protein,_predicted_efflux_pump,length=1488>"

[W::bcf_hdr_parse] Could not parse header line: ##contig=<ID=Egrandis_contig_40711_Uncharacterized_membrane_protein,_predicted_efflux_pump,length=1488>

[E::bcf_hdr_parse_line] Could not parse the header line: "##contig=<ID=Egrandis_contig_40796__Protein_of_unknown_function_(DUF3675)_Zinc_finger,C3HC4_type(RING_finger),length=810>"

[W::bcf_hdr_parse] Could not parse header line: ##contig=<ID=Egrandis_contig_40796__Protein_of_unknown_function_(DUF3675)_Zinc_finger,C3HC4_type(RING_finger),length=810>

....

Note: the --sample option not given, applying all records

The fasta sequence does not match the REF allele at cpl_Euc_Mauve_Alignment_extraction_ndhF:196:

.vcf: [N]

.vcf: [N] <- (ALT)

.fa: [M]GAGTTCGGTCACTTAATAGATCCACTTACTTCTATTATGTTAATATTAATTACTACTGTTGGAATTTTGGTTCTTTTTTATAGTGACAATTATATGTCTCATGATCAAGGATATTTGAGATTTTTTGCTTATATGAGTTTTTTCAATACTTCCATGTTGGGATTAGTTACTAGTTCGAATTTGATACAAATTTATATTTTTTGGGAATTAGTTGGAATGTGTTCTTATCTATTAATAGGTTTTTGGTTCACACGACCTAGTGCGGCGACTGCTTGTCAAAAAGCGTTTGTAACGAATCGTGTAGGCGATTTTGGTTTATTATTAGGAATTTTAGGTCTTTATTGGATAACCGGTAGTTTTGAATTTCGGGATTTGTTCCAAATATTGAATAACTTGATTTATAATAATGAGGTTCCTTTTTTATTTCTTACTTTGTGTGCCTTTCTTTTATTTGCAGGTGCAGTTGCGAAATCGGCACAATTCCCCCTTCATGTATGGTTACCTGATGCCATGGAAGGCCCTACTCCCATTTCGGCTCTTATACATGCCGCTACTATGGTAGCAGCGGGCATTTTTCTTGTAGCTCGACTTCTTCCTCTTTTTATAATCATACCTTACATAATGAATTTCATATCTTTAATAGGTATAATAACAGTATTATTAGGAGCTACTTTAGCTCTTGCTCAAAAAGATATTAAAAGAGGTTTAGCTTATTCTACAATGTCTCAATTGGGTTATATGATGTTAGCTCTAGGTATGGGGTCTTATCGAGCCGCTTTATTTCATTTGATTACTCATGCTTATTCAAAAGCATTGTTGTTTTTAGGATCCGGATCAATTATTCATTCAATGGAAGCTATTGTTGGATATTCTCCAGATAAAAGTCAGAATATGGTTCTTATGGGAGGTTTAAAAAAGCATGTACCAATTACAAAAACTGCTTTTTTAGTAGGTACACTTTCTCTTTGTGGTATTCCCCCCCTTGCTTGTTTTTGGTCCAAAGATGAAATTCTTAATGATAGTTGGTTGTATTCACCTATTTTCGCAATAATAGCTTGTTCCACAGCAGGATTAACCGCATTTTATATGTTTCGAATCTATTTMCTTACTTTTGAGGGACATTTCAATGTTCATTTTCAAAATTACAATGGTCAAAAAAGTAGTTCCTGCTATTCAATATCTCTATGGGGAAAAGAAGTGCCAAAAAYGATTAAAAATCATTTTTGTTTATTAAGTTTATTRACAATGAATAATAATGAAAGGRCTTCTTTTTTTTCGAATAARACATATCAAATTGATGGTAATGGAAAAAACAGGATACGYCCTTTTATTACTATTACTMATTTTGTCACTAAAAAWACTTTCTCTTATCCTCATGAATCGGACAATACCATGTTRTTTTCTATGGTTATATTAGTGYYATTTACTTTGTTTGTTGGGGTCGTAGGAATTCCCTTTGCTTTTAATCAAGAAGAAATTCATTTGGATATATTATCTAAATTGTTAAATCCGTCTATAAACCTTTTACATCCGAATTCAAATAATTCGGTGGATTGGTATGAATTTGTGACAAATGCAAGTTTTTCTGTCAGWATAGCTTTTTTCGGAATATTTATAGSGTCTTTTTTATATAASCCTATTTATTCATCTTTACAAAATTTGAACTTACTRAATTCGTTTTCTAAAAGAGGTYCTAATMGAATTTTAGGGGACAGAATAAGAAATGGGATATATGATTGGTCATATAATCGTGGTTACATAGATGCTTTTTATACAATAYCCTTAACTCAGGGTATAAGAGGACTAGCTGAACTAATTCATTTTTTGGATAGACGASTAATTGATGGAATTACGAATGGTYTCG

bcftools consensus -f ../../Euc_RefSeq.fas -I -m test-region.bed test.vcf.gz -o test.mask.fa

[E::bcf_hdr_parse_line] Could not parse the header line: "##contig=<ID=Egrandis_33521_Branched_chain_aminotransferase_BCAT1,_pyridoxal_phosphate_enzymes_type_IV_superfamily,length=1650>"

[W::bcf_hdr_parse] Could not parse header line: ##contig=<ID=Egrandis_33521_Branched_chain_aminotransferase_BCAT1,_pyridoxal_phosphate_enzymes_type_IV_superfamily,length=1650>

[E::bcf_hdr_parse_line] Could not parse the header line: "##contig=<ID=Egrandis_contig_40711_Uncharacterized_membrane_protein,_predicted_efflux_pump,length=1488>"

[W::bcf_hdr_parse] Could not parse header line: ##contig=<ID=Egrandis_contig_40711_Uncharacterized_membrane_protein,_predicted_efflux_pump,length=1488>

[E::bcf_hdr_parse_line] Could not parse the header line: "##contig=<ID=Egrandis_contig_40796__Protein_of_unknown_function_(DUF3675)_Zinc_finger,C3HC4_type(RING_finger),length=810>"
....

Could not parse bed line: cpl_Egrandis_20012500 480

Failed to initialize mask regions

[Paul-rk-cruz]

Still issues in 2021...

[E::bcf_hdr_parse_line] Could not parse the header line: "##contig=<ID=JF781502 Human rhinovirus B strain HRV-B84_p1098_sR861_2008 polyprotein gene, complete cds,length=6941>"
[E::bcf_hdr_parse_line] Could not parse the header line: "##contig=<ID=KY369881 Human rhinovirus C2 strain SC9735, complete genome,length=7064>"
[E::faidx_adjust_position] The sequence "JQ837720 Human rhinovirus C strain HRV-C17_p1192_sR2967_2009 polyprotein gene, complete cds" was not found
[E::bam_plp_push] The input is not sorted (reads out of order)
[W::bcf_record_check] Bad BCF record at 544: Invalid CONTIG id 277

Any solutions?