vg index segmantation fault

[shilpagarg]

I am doing vg index -x SK1+Y12.bwa.xg -g SK1+Y12.bwa.gcsa -k 16 SK1+Y12.bwa.vg

and here is bt of segmentation fault:

#0  0x00000000009945c3 in std::pair<sdsl::wt_pc<sdsl::balanced_shape, sdsl::int_vector<(unsigned char)1>, sdsl::rank_support_v<(unsigned char)1, (unsigned char)1>, sdsl::select_support_mcl<(unsigned char)1, (unsigned char)1>, sdsl::select_support_mcl<(unsigned char)0, (unsigned char)1>, sdsl::byte_tree<false> >::value_type, sdsl::wt_pc<sdsl::balanced_shape, sdsl::int_vector<(unsigned char)1>, sdsl::rank_support_v<(unsigned char)1, (unsigned char)1>, sdsl::select_support_mcl<(unsigned char)1, (unsigned char)1>, sdsl::select_support_mcl<(unsigned char)0, (unsigned char)1>, sdsl::byte_tree<false> >::size_type> sdsl::quantile_freq<sdsl::wt_pc<sdsl::balanced_shape, sdsl::int_vector<(unsigned char)1>, sdsl::rank_support_v<(unsigned char)1, (unsigned char)1>, sdsl::select_support_mcl<(unsigned char)1, (unsigned char)1>, sdsl::select_support_mcl<(unsigned char)0, (unsigned char)1>, sdsl::byte_tree<false> > >(sdsl::wt_pc<sdsl::balanced_shape, sdsl::int_vector<(unsigned char)1>, sdsl::rank_support_v<(unsigned char)1, (unsigned char)1>, sdsl::select_support_mcl<(unsigned char)1, (unsigned char)1>, sdsl::select_support_mcl<(unsigned char)0, (unsigned char)1>, sdsl::byte_tree<false> > const&, sdsl::wt_pc<sdsl::balanced_shape, sdsl::int_vector<(unsigned char)1>, sdsl::rank_support_v<(unsigned char)1, (unsigned char)1>, sdsl::select_support_mcl<(unsigned char)1, (unsigned char)1>, sdsl::select_support_mcl<(unsigned char)0, (unsigned char)1>, sdsl::byte_tree<false> >::size_type, sdsl::wt_pc<sdsl::balanced_shape, sdsl::int_vector<(unsigned char)1>, sdsl::rank_support_v<(unsigned char)1, (unsigned char)1>, sdsl::select_support_mcl<(unsigned char)1, (unsigned char)1>, sdsl::select_support_mcl<(unsigned char)0, (unsigned char)1>, sdsl::byte_tree<false> >::size_type, sdsl::wt_pc<sdsl::balanced_shape, sdsl::int_vector<(unsigned char)1>, sdsl::rank_support_v<(unsigned char)1, (unsigned char)1>, sdsl::select_support_mcl<(unsigned char)1, (unsigned char)1>, sdsl::select_support_mcl<(unsigned char)0, (unsigned char)1>, sdsl::byte_tree<false> >::size_type) ()
#1  0x00000000009861e5 in gcsa::LCP::min_lcp(gcsa::PathNode const&, gcsa::PathNode const&, unsigned int const*, unsigned int const*) const ()
#2  0x0000000000986611 in gcsa::PathRange::PathRange(unsigned long, unsigned long, std::pair<unsigned long, unsigned long>, gcsa::PathGraphMerger&) ()
#3  0x000000000098e48c in gcsa::PathGraphMerger::next() ()
#4  0x000000000098fd0e in gcsa::PathGraph::prune(gcsa::LCP const&, unsigned long) ()
#5  0x0000000000950da3 in gcsa::GCSA::GCSA(gcsa::InputGraph&, gcsa::ConstructionParameters const&) ()
#6  0x000000000069a214 in main_index(int, char**) () at src/subcommand/index_main.cpp:873
#7  0x00000000004182f2 in main () at src/main.cpp:7676

Here is the graph:
https://transfer.sh/YP4fM/sk1-y12.bwa.vg

[ekg]

What is the maximum node size in the graph?

…

On Thu, Feb 16, 2017, 8:46 PM shilpagarg ***@***.***> wrote: I am doing vg index -x SK1+Y12.bwa.xg -g SK1+Y12.bwa.gcsa -k 16 SK1+ Y12.bwa.vg and here is bt of segmentation fault: #0 0x00000000009945c3 in std::pair<sdsl::wt_pc<sdsl::balanced_shape, sdsl::int_vector<(unsigned char)1>, sdsl::rank_support_v<(unsigned char)1, (unsigned char)1>, sdsl::select_support_mcl<(unsigned char)1, (unsigned char)1>, sdsl::select_support_mcl<(unsigned char)0, (unsigned char)1>, sdsl::byte_tree >::value_type, sdsl::wt_pc<sdsl::balanced_shape, sdsl::int_vector<(unsigned char)1>, sdsl::rank_support_v<(unsigned char)1, (unsigned char)1>, sdsl::select_support_mcl<(unsigned char)1, (unsigned char)1>, sdsl::select_support_mcl<(unsigned char)0, (unsigned char)1>, sdsl::byte_tree >::size_type> sdsl::quantile_freq<sdsl::wt_pc<sdsl::balanced_shape, sdsl::int_vector<(unsigned char)1>, sdsl::rank_support_v<(unsigned char)1, (unsigned char)1>, sdsl::select_support_mcl<(unsigned char)1, (unsigned char)1>, sdsl::select_support_mcl<(unsigned char)0, (unsigned char)1>, sdsl::byte_tree > >(sdsl::wt_pc<sdsl::balanced_shape, sdsl::int_vector<(unsigned char)1>, sdsl::rank_support_v<(unsigned char)1, (unsigned char)1>, sdsl::select_support_mcl<(unsigned char)1, (unsigned char)1>, sdsl::select_support_mcl<(unsigned char)0, (unsigned char)1>, sdsl::byte_tree > const&, sdsl::wt_pc<sdsl::balanced_shape, sdsl::int_vector<(unsigned char)1>, sdsl::rank_support_v<(unsigned char)1, (unsigned char)1>, sdsl::select_support_mcl<(unsigned char)1, (unsigned char)1>, sdsl::select_support_mcl<(unsigned char)0, (unsigned char)1>, sdsl::byte_tree >::size_type, sdsl::wt_pc<sdsl::balanced_shape, sdsl::int_vector<(unsigned char)1>, sdsl::rank_support_v<(unsigned char)1, (unsigned char)1>, sdsl::select_support_mcl<(unsigned char)1, (unsigned char)1>, sdsl::select_support_mcl<(unsigned char)0, (unsigned char)1>, sdsl::byte_tree >::size_type, sdsl::wt_pc<sdsl::balanced_shape, sdsl::int_vector<(unsigned char)1>, sdsl::rank_support_v<(unsigned char)1, (unsigned char)1>, sdsl::select_support_mcl<(unsigned char)1, (unsigned char)1>, sdsl::select_support_mcl<(unsigned char)0, (unsigned char)1>, sdsl::byte_tree >::size_type) () #1 <#1> 0x00000000009861e5 in gcsa::LCP::min_lcp(gcsa::PathNode const&, gcsa::PathNode const&, unsigned int const*, unsigned int const*) const () #2 <#2> 0x0000000000986611 in gcsa::PathRange::PathRange(unsigned long, unsigned long, std::pair<unsigned long, unsigned long>, gcsa::PathGraphMerger&) () #3 <#3> 0x000000000098e48c in gcsa::PathGraphMerger::next() () #4 <#4> 0x000000000098fd0e in gcsa::PathGraph::prune(gcsa::LCP const&, unsigned long) () #5 <#5> 0x0000000000950da3 in gcsa::GCSA::GCSA(gcsa::InputGraph&, gcsa::ConstructionParameters const&) () #6 <#6> 0x000000000069a214 in main_index(int, char**) () at src/subcommand/index_main.cpp:873 #7 <#7> 0x00000000004182f2 in main () at src/main.cpp:7676 Here is the graph: https://transfer.sh/YP4fM/sk1-y12.bwa.vg — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#674>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAI4EUm8zooOiEbBm82UzEQi38L3yoqDks5rdKeXgaJpZM4MDdez> .

[jltsiren]

You may have run out of memory or disk space. The third doubling step creates a graph with 2.5 billion nodes and uses over 140 gigabytes of memory. The final graph is 10x smaller, and the order-128 index takes a little under a gigabyte. If you use only two doubling steps, you get a much more reasonable index.

[shilpagarg]

I chopped graph using

vg mod -X 64 SK1+Y12.bwa.vg > SK1+Y12.bwa_X.vg

and then index it using

vg run index -x SK1+Y12.bwa.xg -g SK1+Y12.bwa.gcsa -k 8 -X 2 SK1+Y12.bwa.vg

and I get

#0  0x000000000098de6d in gcsa::PathGraphMerger::read(gcsa::PriorityNode&) ()
#1  0x000000000098e04c in gcsa::PathGraphMerger::bufferNext() ()
#2  0x000000000098e295 in gcsa::PathGraphMerger::rangeEnd(unsigned long) ()
#3  0x000000000098e622 in gcsa::PathGraphMerger::first() ()
#4  0x000000000098f7f8 in gcsa::PathGraph::prune(gcsa::LCP const&, unsigned long) ()
#5  0x0000000000950da3 in gcsa::GCSA::GCSA(gcsa::InputGraph&, gcsa::ConstructionParameters const&) ()
#6  0x000000000069a214 in main_index(int, char**) () at src/subcommand/index_main.cpp:873
#7  0x00000000004182f2 in main () at src/main.cpp:7676

[glennhickey]

You're not passing SK1+Y12.bwa_X.vg to your index command in that example...

…

On Fri, Feb 17, 2017 at 1:23 PM, shilpagarg ***@***.***> wrote: I chopped graph using vg mod -X 64 SK1+Y12.bwa.vg > SK1+Y12.bwa_X.vg and then index it using vg run index -x SK1+Y12.bwa.xg -g SK1+Y12.bwa.gcsa -k 8 -X 2 SK1+ Y12.bwa.vg and I get #0 0x000000000098de6d in gcsa::PathGraphMerger::read(gcsa::PriorityNode&) () #1 0x000000000098e04c in gcsa::PathGraphMerger::bufferNext() () #2 0x000000000098e295 in gcsa::PathGraphMerger::rangeEnd(unsigned long) () #3 0x000000000098e622 in gcsa::PathGraphMerger::first() () #4 0x000000000098f7f8 in gcsa::PathGraph::prune(gcsa::LCP const&, unsigned long) () #5 0x0000000000950da3 in gcsa::GCSA::GCSA(gcsa::InputGraph&, gcsa::ConstructionParameters const&) () #6 0x000000000069a214 in main_index(int, char**) () at src/subcommand/index_main.cpp:873 #7 0x00000000004182f2 in main () at src/main.cpp:7676 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#674 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA2_7lCO7BEJOg3BPfZs0dNozToPgoMAks5rdeWhgaJpZM4MDdez> .

[shilpagarg]

sorry I copied the wrong command, but I ran the right one.

vg index -x SK1+Y12.bwa_X.xg -g SK1+Y12.bwa_X.gcsa -k 8 -X 2 SK1+Y12.bwa_X.vg

That is why the backtrace now is different than the previous one.

[shilpagarg]

I face the same issue for chrI as well and here is the chopped vg graph:

https://transfer.sh/nA8LE/sk1-y12.bwa-x.chri.vg

and backtrace

#0  0x000000000098de6d in gcsa::PathGraphMerger::read(gcsa::PriorityNode&) ()
#1  0x000000000098e04c in gcsa::PathGraphMerger::bufferNext() ()
#2  0x000000000098e295 in gcsa::PathGraphMerger::rangeEnd(unsigned long) ()
#3  0x000000000098e622 in gcsa::PathGraphMerger::first() ()
#4  0x000000000098f7f8 in gcsa::PathGraph::prune(gcsa::LCP const&, unsigned long) ()
#5  0x0000000000950da3 in gcsa::GCSA::GCSA(gcsa::InputGraph&, gcsa::ConstructionParameters const&) ()
#6  0x000000000069a214 in main_index(int, char**) () at src/subcommand/index_main.cpp:873
#7  0x00000000004182f2 in main () at src/main.cpp:7676

[glennhickey]

Using the latest vg, I just ran wget https://transfer.sh/nA8LE/sk1-y12.bwa-x.chri.vg vg index -x SK1+Y12.bwa_X.xg -g SK1+Y12.bwa_X.gcsa -k 8 -X 2 sk1-y12.bwa-x.chri.vg without any crash or error message. Took about 2min on 8 cores and 2G RAM. ls -l

…

-rw-rw-r-- 1 hickey hickey 6557138 Feb 17 15:54 sk1-y12.bwa-x.chri.vg -rw-rw-r-- 1 hickey hickey 26977467 Feb 17 15:56 SK1+Y12.bwa_X.gcsa -rw-rw-r-- 1 hickey hickey 14710097 Feb 17 15:56 SK1+Y12.bwa_X.gcsa.lcp -rw-rw-r-- 1 hickey hickey 21853206 Feb 17 15:55 SK1+Y12.bwa_X.xg

On Fri, Feb 17, 2017 at 2:43 PM, shilpagarg ***@***.***> wrote: I face the same issue for chrI as well and here is the chopped vg graph: https://transfer.sh/nA8LE/sk1-y12.bwa-x.chri.vg and backtrace #0 0x000000000098de6d in gcsa::PathGraphMerger::read(gcsa::PriorityNode&) () #1 0x000000000098e04c in gcsa::PathGraphMerger::bufferNext() () #2 0x000000000098e295 in gcsa::PathGraphMerger::rangeEnd(unsigned long) () #3 0x000000000098e622 in gcsa::PathGraphMerger::first() () #4 0x000000000098f7f8 in gcsa::PathGraph::prune(gcsa::LCP const&, unsigned long) () #5 0x0000000000950da3 in gcsa::GCSA::GCSA(gcsa::InputGraph&, gcsa::ConstructionParameters const&) () #6 0x000000000069a214 in main_index(int, char**) () at src/subcommand/index_main.cpp:873 #7 0x00000000004182f2 in main () at src/main.cpp:7676 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#674 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA2_7obMwyuhP2REDWRiM2K1i57quBGhks5rdfhKgaJpZM4MDdez> .