Core dump when combining VCFs

[iqbal-lab]

Hi there
I thought i'd give vcfcombine another go, and on testing on small test VCfs it worked perfectly.
However when I tried this (on a clean git clone --recursive), combining 2600 VCFs, I got this

vcflib/bin/vcfcombine results/*vcf > outvcf
terminate called after throwing an instance of 'std::out_of_range'
what(): basic_string::substr
Aborted (core dumped)
bash-4.1$

[ekg]

What'd you find in "output"?

On Fri, Sep 18, 2015 at 4:33 PM, Zamin Iqbal notifications@github.com
wrote:

Hi there
I thought i'd give vcfcombine another go, and on testing on small test
VCfs it worked perfectly.
However when I tried this (on a clean git clone --recursive), combining
2600 VCFs, I got this

vcflib/bin/vcfcombine results/*vcf > outvcf
terminate called after throwing an instance of 'std::out_of_range'
what(): basic_string::substr
Aborted (core dumped)
bash-4.1$

—
Reply to this email directly or view it on GitHub
#103.

[iqbal-lab]

What do you mean in "output"?

[iqbal-lab]

Do you mean "outvcf" ?

[ekg]

Yes! outvcf :)

On Fri, Sep 18, 2015 at 4:36 PM, Zamin Iqbal notifications@github.com
wrote:

Do you mean "outvcf" ?

—
Reply to this email directly or view it on GitHub
#103 (comment).

[iqbal-lab]

It's empty!

[iqbal-lab]

[Thread debugging using libthread_db enabled]
terminate called after throwing an instance of 'std::out_of_range'
what(): basic_string::substr

Program received signal SIGABRT, Aborted.
0x0000003a11e32625 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.149.el6_6.9.x86_64 libgcc-4.4.7-11.el6.x86_64 libstdc++-4.4.7-11.el6.x86_64
(gdb) bt
#0 0x0000003a11e32625 in raise () from /lib64/libc.so.6
#1 0x0000003a11e33e05 in abort () from /lib64/libc.so.6
#2 0x0000003a146bea7d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib64/libstdc++.so.6
#3 0x0000003a146bcbd6 in ?? () from /usr/lib64/libstdc++.so.6
#4 0x0000003a146bcc03 in std::terminate() () from /usr/lib64/libstdc++.so.6
#5 0x0000003a146bcd22 in __cxa_throw () from /usr/lib64/libstdc++.so.6
#6 0x0000003a14661db7 in std::__throw_out_of_range(char const*) () from /usr/lib64/libstdc++.so.6
#7 0x00000000004199ae in vcflib::VariantCallFile::parseHeader(std::basic_string<char, std::char_traits, std::allocator >&) ()
#8 0x0000000000419b60 in vcflib::VariantCallFile::parseHeader() ()
#9 0x00000000004073ff in main ()
(gdb)

[iqbal-lab]

Anything useful I can query while in gdb?

[ekg]

If you can add '-g' to
https://github.com/ekg/vcflib/blob/master/Makefile#L100 and recompile, then
we can see exactly what line breaks.

It looks unhappy with the header somehow, but the exact line will explain
what's going on.

On Fri, Sep 18, 2015 at 4:40 PM, Zamin Iqbal notifications@github.com
wrote:

Anything useful I can query while in gdb?

—
Reply to this email directly or view it on GitHub
#103 (comment).

[iqbal-lab]

Here you go

0x0000003a11e32625 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.149.el6_6.9.x86_64 libgcc-4.4.7-11.el6.x86_64 libstdc++-4.4.7-11.el6.x86_64
(gdb) bt
#0 0x0000003a11e32625 in raise () from /lib64/libc.so.6
#1 0x0000003a11e33e05 in abort () from /lib64/libc.so.6
#2 0x0000003a146bea7d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib64/libstdc++.so.6
#3 0x0000003a146bcbd6 in ?? () from /usr/lib64/libstdc++.so.6
#4 0x0000003a146bcc03 in std::terminate() () from /usr/lib64/libstdc++.so.6
#5 0x0000003a146bcd22 in __cxa_throw () from /usr/lib64/libstdc++.so.6
#6 0x0000003a14661db7 in std::__throw_out_of_range(char const*) () from /usr/lib64/libstdc++.so.6
#7 0x00000000004199ae in _M_check (this=0x19ff280, hs="")
at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/bits/basic_string.h:309
#8 substr (this=0x19ff280, hs="") at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/bits/basic_string.h:2003
#9 vcflib::VariantCallFile::parseHeader (this=0x19ff280, hs="") at src/Variant.cpp:1178
#10 0x0000000000419b60 in vcflib::VariantCallFile::parseHeader (this=0x19ff280) at src/Variant.cpp:1172
#11 0x00000000004073ff in openFile (argc=2601, argv=) at src/Variant.h:100
#12 open (argc=2601, argv=) at src/Variant.h:93
#13 main (argc=2601, argv=) at src/vcfcombine.cpp:100

[iqbal-lab]

(gdb) frame 8
#8 substr (this=0x19ff280, hs="") at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/bits/basic_string.h:2003
2003 _M_check(__pos, "basic_string::substr"), __n); }
(gdb) l
1998 * is thrown.
1999 _/
2000 basic_string
2001 substr(size_type __pos = 0, size_type __n = npos) const
2002 { return basic_string(_this,
2003 _M_check(__pos, "basic_string::substr"), __n); }
2004
2005 /**
2006 * @brief Compare to a string.
2007 * @param str String to compare against.

[iqbal-lab]

frame 9
#9 vcflib::VariantCallFile::parseHeader (this=0x19ff280, hs="") at src/Variant.cpp:1178
1178 if (hs.substr(hs.size() - 1, 1) == "\n") {
(gdb) l
1173
1174 }
1175
1176 bool VariantCallFile::parseHeader(string& hs) {
1177
1178 if (hs.substr(hs.size() - 1, 1) == "\n") {
1179 hs.erase(hs.size() - 1, 1); // remove trailing newline
1180 }
1181 header = hs; // stores the header in the object instance
1182
(gdb) print hs
$1 = ""
(gdb)

[iqbal-lab]

frame 11
#11 0x00000000004073ff in openFile (argc=2601, argv=) at src/Variant.h:100
100 parsedHeader = parseHeader();
(gdb)

[iqbal-lab]

2601 means the very last file in the list - there are 2600 files

[ekg]

hs = "" means that the header string is empty. The error could definitely
be improved. Doing that now.

I'm concerned that the file name list is too long for some system limit.

On Fri, Sep 18, 2015 at 4:55 PM, Zamin Iqbal notifications@github.com
wrote:

2601 means the very last file in the list - there are 2600 files

—
Reply to this email directly or view it on GitHub
#103 (comment).

[iqbal-lab]

Hm - ok I'll try splitting in two!

[ekg]

Try the current git HEAD. It should show you what file failed to open now.

On Fri, Sep 18, 2015 at 4:59 PM, Zamin Iqbal notifications@github.com
wrote:

Hm - ok I'll try splitting in two!

—
Reply to this email directly or view it on GitHub
#103 (comment).

[iqbal-lab]

OK splitting in two has the same problem
But if I restrict to just 500 files, it does not core dump

[iqbal-lab]

Will dig around - thanks Erik

[ekg]

If my suspicion is correct, you'll see something with a truncated filename
when you re-run with the commit I just pushed. This means that the only way
to run the merge will be to use an input file, which would require a little
patch to vcfcombine.

On Fri, Sep 18, 2015 at 5:05 PM, Zamin Iqbal notifications@github.com
wrote:

Will dig around - thanks Erik

—
Reply to this email directly or view it on GitHub
#103 (comment).

[iqbal-lab]

OK, well for now I can manage - thanks a lot Erik. vcfcombine is lovelty and fast

[iqbal-lab]

Close.