Repository navigation
Validate headers before further operation? #2581
Answered
by
ymxlx
j-gillespie-dna
asked this question in
Q&A
|
I am running a merge on ~500 vcfs from WGS. As they are quite large, this is a lengthy process. I was in hour 7 of my HPC run when I received an error about a header and the process ended. Is there a way I can check headers for all 500 files before attempting another merge so I don't waste another 7 hours? For reference, the error I got is below. Note, I received these from a collaborator so I had no control over how the vcfs were produced. |
Answered by
ymxlx
Jul 15, 2026
Replies: 2 comments 2 replies
|
There are two useful checks here. The error you received is not purely a header-to-header incompatibility: it means a record uses You can first check whether all 500 headers can be merged: bcftools merge --print-header --file-list vcfs.txt >/dev/nullHowever, that alone will not catch an undefined tag which only appears late in a file. To detect those, each file has to be parsed through its records at least once. A lightweight check for one file is: bcftools view -Ou -o /dev/null sample.vcf.gzRun that over the file list as a scheduler array or with your usual parallel tool, keeping stderr separate per input file. It avoids doing the expensive 500-way merge while identifying which input emits the missing-header warning. Once you find the file, get the correct |
0 replies
Answer selected by
j-gillespie-dna
|
That worked perfectly! Found another few files wit "extra sample columns" at a few variants: As an additional question, I have never seen the above error so I pulled the offending line, is there a way to determine what caused the error? header (sorry, it's big): |
2 replies
|
That message means the record itself is missing its FORMAT/sample columns entirely — the parser found the 8 fixed VCF columns (up through INFO) but then nothing where the genotype field(s) should be, hence "0 vs 1". A couple of things worth checking:
(10 = 8 fixed columns + FORMAT + 1 sample; adjust if you have more samples). If it's a small, scattered set of lines, that points to truncation. If it's systematic (same NF every time, or tied to a specific tool step), it's more likely a bug in whatever last rewrote the file.
|
|
Thank you so much for the thorough explanation. The offending lines found by bcftools were the only ones and once they were removed from each file, bcftools could process the samples. As I didn't have access to any of the upstream files, removing the lines was the simplest solution. I will let my collaborator know some of their files may have issues. Thank you for your help. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
There are two useful checks here. The error you received is not purely a header-to-header incompatibility: it means a record uses
INFO/C, but that file does not declareCwith an##INFO=<ID=C,...>header line.You can first check whether all 500 headers can be merged:
bcftools merge --print-header --file-list vcfs.txt >/dev/nullHowever, that alone will not catch an undefined tag which only appears late in a file. To detect those, each file has to be parsed through its records at least once. A lightweight check for one file is:
Run that over the file list as a scheduler array or with your usual parallel tool, keeping stderr separate per input f…