Update to xg with graph PBWT support

[adamnovak]

No description provided.

[ekg]

Was this fixed via your other PR?

[adamnovak]

No, the other PR just loads paths into the vg graphs. This is pulling the new xg stuff from Japan.

[ekg]

Gotcha. So this needs to be pulled in to get level with xg.
On Mar 10, 2016 6:03 PM, "adamnovak" notifications@github.com wrote:

No, the other PR just loads paths into the vg graphs. This is pulling the
new xg stuff from Japan.

—
Reply to this email directly or view it on GitHub
#229 (comment).

[ekg]

The build failure suggests something wrong with the dynamic dependency or include path.

[adamnovak]

Yeah, it needs to copy the DYNAMIC headers to include. I should have finally put up the correct Makefile now.

[ekg]

The build works, but tests fail now because there are some very weird errors with find. I think I can figure out what's up. I need to get level with the new XG because I've got some modifications I need to apply there to change the way that path names are stored. I need to escape the block here: https://github.com/vgteam/xg/blob/master/src/xg.cpp#L1488 to ensure that it isn't causing contention when doing banded mapping.

[ekg]

We should move DYNAMIC into vg rather than xg. We can download it to do an xg-only build, but otherwise let's put it in vg/deps.

[ekg]

I'm getting segfaults in when vg tries to deallocate xg indexes. There may be other problems.

[adamnovak]

Hmm.

I was hoping I could do this without actually doing any work, but alas that
is not to be.

I will move the dependency and figure out what is wrong with the
destructors.

-Adam

On Fri, Mar 11, 2016 at 7:38 AM, Erik Garrison notifications@github.com
wrote:

I'm getting segfaults in when vg tries to deallocate xg indexes. There may
be other problems.

—
Reply to this email directly or view it on GitHub
#229 (comment).

[ekg]

I found the problem but don't understand it. Will post my stack trace in 15
mins. Something about trying to destroy dynamic's structures when they are
empty.

I have an updated merge for xg as well, changing the build.
On Mar 11, 2016 6:11 PM, "adamnovak" notifications@github.com wrote:

Hmm.

I was hoping I could do this without actually doing any work, but alas that
is not to be.

I will move the dependency and figure out what is wrong with the
destructors.

-Adam

On Fri, Mar 11, 2016 at 7:38 AM, Erik Garrison notifications@github.com
wrote:

I'm getting segfaults in when vg tries to deallocate xg indexes. There
may
be other problems.

—
Reply to this email directly or view it on GitHub
#229 (comment).

—
Reply to this email directly or view it on GitHub
#229 (comment).

[ekg]

Here we go:

#0  dyn::spsi<dyn::packed_vector, 8192u, 16u>::node::free_mem (this=0x0) at /home/erik/vg/include/dynamic/spsi.hpp:528
#1  0x0000000000466fd8 in dyn::spsi<dyn::packed_vector, 8192u, 16u>::~spsi (this=0xb5cf08, __in_chrg=<optimized out>) at /home/erik/vg/include/dynamic/spsi.hpp:156
#2  dyn::succinct_bitvector<dyn::spsi<dyn::packed_vector, 8192u, 16u> >::~succinct_bitvector (this=0xb5cf08, __in_chrg=<optimized out>) at /home/erik/vg/include/dynamic/succinct_bitvector.hpp:50
#3  dyn::wt_string<dyn::succinct_bitvector<dyn::spsi<dyn::packed_vector, 8192u, 16u> > >::node::~node (this=0xb5cef0, __in_chrg=<optimized out>) at /home/erik/vg/include/dynamic/wt_string.hpp:222
#4  0x0000000000467088 in dyn::wt_string<dyn::succinct_bitvector<dyn::spsi<dyn::packed_vector, 8192u, 16u> > >::~wt_string (this=0xb5cee8, __in_chrg=<optimized out>) at /home/erik/vg/include/dynamic/wt_string.hpp:27
#5  xg::XG::~XG (this=0xb5b720, __in_chrg=<optimized out>) at /home/erik/vg/include/xg.hpp:44
#6  0x000000000044fe9d in main_map (argc=argc@entry=17, argv=argv@entry=0x7fffffffe288) at src/main.cpp:4504
#7  0x0000000000413fd3 in main (argc=17, argv=0x7fffffffe288) at src/main.cpp:5388

I'm going to try to reproduce in xg itself.

[ekg]

In xg's test directory:

xg -v data/l.vg -o l.xg
xg -i l.xg -t 1
# segfault

Backtrace:

#0  0x000000000040bc89 in dyn::wt_string<dyn::succinct_bitvector<dyn::spsi<dyn::packed_vector, 8192u, 16u> > >::node::~node (this=0x971, __in_chrg=<optimized out>) at DYNAMIC/include/internal/wt_string.hpp:206
#1  0x000000000040bcb9 in dyn::wt_string<dyn::succinct_bitvector<dyn::spsi<dyn::packed_vector, 8192u, 16u> > >::node::~node (this=0x95e0c0, __in_chrg=<optimized out>) at DYNAMIC/include/internal/wt_string.hpp:217
#2  0x000000000040bc99 in dyn::wt_string<dyn::succinct_bitvector<dyn::spsi<dyn::packed_vector, 8192u, 16u> > >::node::~node (this=0x95cfd0, __in_chrg=<optimized out>) at DYNAMIC/include/internal/wt_string.hpp:210
#3  0x000000000040bc99 in dyn::wt_string<dyn::succinct_bitvector<dyn::spsi<dyn::packed_vector, 8192u, 16u> > >::node::~node (this=0x963610, __in_chrg=<optimized out>) at DYNAMIC/include/internal/wt_string.hpp:210
#4  0x000000000040bd78 in dyn::wt_string<dyn::succinct_bitvector<dyn::spsi<dyn::packed_vector, 8192u, 16u> > >::~wt_string (this=0x963608, __in_chrg=<optimized out>) at DYNAMIC/include/internal/wt_string.hpp:27
#5  xg::XG::~XG (this=0x961e40, __in_chrg=<optimized out>) at src/xg.hpp:44
#6  0x000000000040280a in main (argc=<optimized out>, argv=<optimized out>) at src/main.cpp:310

[adamnovak]

It's not clear to me how it's getting a bad value (0x971) for child1_. It
sets it to NULL as a default value, and only ever writes to it by assigning
from new or NULL. 0x971 should not be coming from either of those.

Is some kind of out-of-bounds access happening to corrupt things?

On Fri, Mar 11, 2016 at 11:33 AM, Erik Garrison notifications@github.com
wrote:

In xg's test directory:

xg -v data/l.vg -o l.xg
xg -i l.xg -t 1

segfault

Backtrace:

#0 0x000000000040bc89 in dyn::wt_string<dyn::succinct_bitvector<dyn::spsi<dyn::packed_vector, 8192u, 16u> > >::node::~node (this=0x971, __in_chrg=) at DYNAMIC/include/internal/wt_string.hpp:206
#1 0x000000000040bcb9 in dyn::wt_string<dyn::succinct_bitvector<dyn::spsi<dyn::packed_vector, 8192u, 16u> > >::node::~node (this=0x95e0c0, __in_chrg=) at DYNAMIC/include/internal/wt_string.hpp:217
#2 0x000000000040bc99 in dyn::wt_string<dyn::succinct_bitvector<dyn::spsi<dyn::packed_vector, 8192u, 16u> > >::node::~node (this=0x95cfd0, __in_chrg=) at DYNAMIC/include/internal/wt_string.hpp:210
#3 0x000000000040bc99 in dyn::wt_string<dyn::succinct_bitvector<dyn::spsi<dyn::packed_vector, 8192u, 16u> > >::node::~node (this=0x963610, __in_chrg=) at DYNAMIC/include/internal/wt_string.hpp:210
#4 0x000000000040bd78 in dyn::wt_string<dyn::succinct_bitvector<dyn::spsi<dyn::packed_vector, 8192u, 16u> > >::~wt_string (this=0x963608, __in_chrg=) at DYNAMIC/include/internal/wt_string.hpp:27
#5 xg::XG::~XG (this=0x961e40, __in_chrg=) at src/xg.hpp:44
#6 0x000000000040280a in main (argc=, argv=) at src/main.cpp:310

—
Reply to this email directly or view it on GitHub
#229 (comment).

[ekg]

Looks like the problem was soluble by forcing the dynamic wavelet tree to be allocated on the heap.