SIGABRT Crash

[Razdeep]

I'm getting this error while running the program

System Configuration:

• Ubuntu : 20.04
• gcc : g++ (Ubuntu 9.3.0-10ubuntu2) 9.3.0
• boost : libboost-all-dev 1.71.0.0ubuntu2

Input:

./build_trie ef_trie 5 count --dir ../test_data --out ef_trie.count.bin

Output:

2020-07-15 10:47:41: Reading 1-grams counts
2020-07-15 10:47:41: Reading 2-grams counts
2020-07-15 10:47:41: Reading 3-grams counts
2020-07-15 10:47:41: Reading 4-grams counts
2020-07-15 10:47:41: Reading 5-grams counts
2020-07-15 10:47:41: Building vocabulary
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Aborted (core dumped)

GDB Backtrace Log:

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff7bcf859 in __GI_abort () at abort.c:79
#2  0x00007ffff7e55951 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007ffff7e6147c in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007ffff7e614e7 in std::terminate() () from /lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007ffff7e61799 in __cxa_throw () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007ffff7e55562 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#7  0x0000555555592e4f in __gnu_cxx::new_allocator<unsigned char>::allocate (this=0x7fffffffcd28, __n=33016750080)
    at /usr/include/c++/9/ext/new_allocator.h:102
#8  std::allocator_traits<std::allocator<unsigned char> >::allocate (__a=..., __n=33016750080)
    at /usr/include/c++/9/bits/alloc_traits.h:444
#9  std::_Vector_base<unsigned char, std::allocator<unsigned char> >::_M_allocate (this=0x7fffffffcd28, __n=33016750080)
    at /usr/include/c++/9/bits/stl_vector.h:343
#10 std::vector<unsigned char, std::allocator<unsigned char> >::reserve (this=0x7fffffffcd28, __n=33016750080)
    at /usr/include/c++/9/bits/vector.tcc:78
#11 0x0000555555592edd in tongrams::grams_counts_pool::grams_counts_pool (this=0x7fffffffcd00, num_bytes=<optimized out>)
    at /home/ubuntu/tongrams/include/utils/pools.hpp:104
#12 0x00005555555e7ec3 in tongrams::trie_count_lm<tongrams::single_valued_mpht<tongrams::hash_compact_vector<unsigned long>, emphf::jenkins64_hasher>, tongrams::identity_mapper, tongrams::sequence_collection, tongrams::indexed_codewords_sequence, tongrams::fast_ef_sequence, tongrams::ef_sequence>::builder::build_vocabulary (this=0x7fffffffdee0, counts_builder=...)
    at /home/ubuntu/tongrams/include/trie_count_lm.hpp:120
#13 0x00005555555eb33e in tongrams::trie_count_lm<tongrams::single_valued_mpht<tongrams::hash_compact_vector<unsigned long>, emphf::jenkins64_hasher>, tongrams::identity_mapper, tongrams::sequence_collection, tongrams::indexed_codewords_sequence, tongrams::fast_ef_sequence, tongrams::ef_sequence>::builder::builder (this=0x7fffffffdee0, input_dir=<optimized out>, order=<optimized out>, 
    remapping_order=<optimized out>) at /usr/include/c++/9/ext/new_allocator.h:89
#14 0x0000555555585f77 in main (argc=<optimized out>, argv=<optimized out>) at /home/ubuntu/tongrams/src/build_trie.cpp:159

[jermp]

Hello, very strange.
I re-tested the library on:

• my Mac (Catalina 10.15.2) with Boost-1.72.0 and Clang 11
• a Linux machine (Ubuntu 19.10) with Boost-1.67.0 and gcc 9.2.1
  and everything works just fine...

[Razdeep]

Could you please test it in a docker container?
I suspect that it is happening because of lesser RAM.

[Razdeep]

It runs smoothly on my Mac too.
I also tried to run it on an AWS instance, getting the same runtime error there also.

[jermp]

And what are the specifications of that AWS instance?

[Razdeep]

I used an AWS Marketplace AMI Ubuntu 20.04.
I tried it on three instances m5ad.4xlarge, m5ad.2xlarge and m5.4xlarge.
These instances have around 8 GB to 16 GB of RAM. Please Refer to this page for details.

[jermp]

If you give me access to one of those systems, I can investigate by myself.
But as long as there is not a way to reproduce the error, there is nothing I can do right now.

[Razdeep]

I cannot post the credentials publicly here.
Shall I send an email containing the private key and IP Address?
Please accept my connection request on linkedin https://www.linkedin.com/in/razdeeproychowdhury/

[jermp]

Sure, thanks.

[Razdeep]

Please check your email inbox

[jermp]

The problem is that

sysconf(_SC_PAGESIZE) * sysconf(_SC_PHYS_PAGES)

returns the physical memory of the machine, but the ram that is left available
for the system is often less.

So doing

grams_counts_pool unigrams_pool(available_ram * 0.8);

(which allocates a pool of 80% of the available ram) solves the problem.