bcftools norm -m-both lost some record after about 1K records

[wangyugui]

bcftools norm -m-both works correctly for the first 1K records , but lost some of them after about 1K records.

bcftools version 1.2 does NOT have this problem, but the last source in github have this problem.

the feature of this vcf file
1)VCF 4.2
2)it have 200 samples

[wangyugui]

Not only the second allele of multiallelic record may be lost, but also some record of single allele may be lost too.

[winni2k]

I suspect that the error is caused by a particular line in your VCF.

Can you post a minimal file (maybe as a gist?) that is made up or real that can be used to reproduce this error?

[winni2k]

PS Thank you for the bug report!

[wangyugui]

I failed to create a public gist. so I sent the vcf file as an attachment to wkretzsch@gmail.com.

Best Regards
Wang Yugui

[winni2k]

Thanks, I'll take a look.

[winni2k]

I can confirm the existence of a bug. I have narrowed down the problem to a small VCF with 79 sites. If I remove a site from the front or the end, then the bug disappears. I have posted the file here. However, I am not familiar with bcftools norm, so someone else needs to take a look at the source code.

This is the code I used to replicate the error:

function test {
    ../bcftools norm -m-both $1  | grep -v '^#' | wc -l
    grep -v '^#' $1 | wc -l
    grep -v '^#' $1 | cut -f5| grep -o ',' |wc -l
}
test issue_336_minimal.vcf

where the first number should be the same as the sum of the second and third number.

I am using bcftools version:

../bcftools --version
bcftools 1.2-157-g8deae27
Using htslib 1.2.1-226-g1e5c377
Copyright (C) 2015 Genome Research Ltd.
License Expat: The MIT/Expat license
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

[winni2k]

It appears as if the first site is being dropped (position 1:789241), but only if the entire file is read. Running bcftools norm -m-both on just a subset of the sites does not cause the omission.

[adamspargo]

Yes. Also the number of lines modified is not updated:

Lines total/modified/skipped: 78/0/0

I'm having a look, but also not familiar with bcftools norm.

[wangyugui]

The number of lines modified is not updated too for bcftools 1.2.

[adamspargo]

I can prevent specific examples from dropping lines by changing buffer limits (e.g. increase above 100 in vcfnorm.c line1486 to make the example above work), but there is a fundamental over-writing issue that will take a while to figure out.

[wangyugui]

vcfnorm.c line1486 is same as bcftools 1.2, so another fundamental over-writing issue maybe the real reason?

[adamspargo]

Sure.

[mcshane]

@wangyugui Should be fixed on the test case from you and @wkretzsch, but let us know if it hasn't fixed up the issue with your original VCF.

[winni2k]

PS I can confirm this code passes fine using bcftools 1.2 (using htslib 1.2.1).

[wangyugui]

This problem is fixed by this patch in my case.
Thanks for this patch.