the problem in vg surject

[xuxingyubio]

I'm very sorry to bother you again.

My command goes like this: vg surject -x "$GRAPH" -t "$NUMCPU" -m -b -S -A -F "$REF_PATHS" "$GAMP" > "$BAM"
I try to use 'vg surject' to project down to linear reference.

I don't know if the following error is due to the command requiring a large amount of memory space, or if there are some issues with my input file, or some other issue. How can I solve this problem?

The error is as follows:
...
warning[vg::Watchdog]: Thread 7 has been checked in for 10 seconds processing: SRR1153470.16621002
warning[vg::Watchdog]: Thread 1 finally checked out after 26 seconds and 38447248 kb memory growth processing: SRR1153470.16349503
[BAMatrix::fill_matrix]: failed to allocate matrices of height 14978670 and width 32 for a total cell count of 479317440
[BAMatrix::fill_matrix]: is alignment problem too big for your virtual or physical memory?
warning[vg::Watchdog]: Thread 7 finally checked out after 16 seconds and 46538040 kb memory growth processing: SRR1153470.16621002
Unhandled exception: std::bad_alloc
Exception context: SRR1153470.16376079
━━━━━━━━━━━━━━━━━━━━
Crash report for vg v1.48.0 "Gallipoli"
Stack trace (most recent call last) in thread 2315526:
#11 Object "", at 0xffffffffffffffff, in
#10 Object "/home/users/xyxu/pantools/cactus-bin-v2.5.2/bin/vg", at 0x1f8f7f2, in __clone
#9 Object "/home/users/xyxu/pantools/cactus-bin-v2.5.2/bin/vg", at 0x1492898, in start_thread
#8 Object "/home/users/xyxu/pantools/cactus-bin-v2.5.2/bin/vg", at 0x1ec3ffd, in gomp_thread_start
#7 Object "/home/users/xyxu/pantools/cactus-bin-v2.5.2/bin/vg", at 0xca1d43, in void vg::io::for_each_parallel_implvg::MultipathAlignment(std::istream&, std::function<void (vg::MultipathAlignment&, vg::MultipathAlignment&)> const&, std::function<void (vg::MultipathAlignment&)> const&, std::function<bool ()> const&, unsigned long) [clone ._omp_fn.0]
#6 Object "/home/users/xyxu/pantools/cactus-bin-v2.5.2/bin/vg", at 0x1ec68d7, in gomp_team_barrier_wait_end
#5 Object "/home/users/xyxu/pantools/cactus-bin-v2.5.2/bin/vg", at 0x1ebdf9b, in gomp_barrier_handle_tasks
#4 Object "/home/users/xyxu/pantools/cactus-bin-v2.5.2/bin/vg", at 0xca16c1, in void vg::io::for_each_parallel_implvg::MultipathAlignment(std::istream&, std::function<void (vg::MultipathAlignment&, vg::MultipathAlignment&)> const&, std::function<void (vg::MultipathAlignment&)> const&, std::function<bool ()> const&, unsigned long) [clone ._omp_fn.1]
#3 Object "/home/users/xyxu/pantools/cactus-bin-v2.5.2/bin/vg", at 0x506970, in main_surject(int, char**)::{lambda(std::istream&)#9}::operator()(std::istream&) const::{lambda(vg::MultipathAlignment&)#2}::operator()(vg::MultipathAlignment) const [clone .cold]
#2 Object "/home/users/xyxu/pantools/cactus-bin-v2.5.2/bin/vg", at 0x5618ac, in vg::report_exception(std::exception const&) [clone .cold]
#1 Object "/home/users/xyxu/pantools/cactus-bin-v2.5.2/bin/vg", at 0x5c0007, in abort
#0 Object "/home/users/xyxu/pantools/cactus-bin-v2.5.2/bin/vg", at 0x149611b, in raise
ERROR: Signal 6 occurred. VG has crashed. Visit https://github.com/vgteam/vg/issues/new/choose to report a bug.
Please include this entire error log in your bug report!

[adamnovak]

It looks like we failed to allocate a 14 megabase by 32 base alignment matrix. That's a weirdly big alignment problem for us to be doing. (It also ought to fit in memory, but sometimes the allocator just refuses to allocate big stuff that seems to me like it ought to fit.)

I think if you throw enough memory at this it will probably succeed. But we should also add some mechanism to skip reads or parts of reads where the surjection is intractable.

If you could vg filter -M -n SRR1153470.16376079 $GAMP >one_read.gamp and send over that file and the graph, we could look at what about this particular read is making it want to do enormous alignments.

You should also be able to throw out that read from your file with vg filter --complement -M -n SRR1153470.16376079 $GAMP >all_other_reads.gamp, and maybe the other reads will be able to run through.

You also might consider reducing the number of threads, to make sure each thread has enough memory to do what it is trying to do.