Bad_alloc error on RPi

[philipktlin]

Expected behavior

Expect to run without exceptions

Actual behavior

Got std::bad_alloc error

Configuration

• Runs on Raspberry Pi 3 (CPU: 4× ARM Cortex-A53, 1.2GHz, 1GB LPDDR2 (900 MHz))
• Used recommended parameter
  // Optimization functions (

  rocksdb/options/options.cc

  Line 468 in 313e877

  DBOptions* DBOptions::OptimizeForSmallDb() {

  )
  DBOptions* DBOptions::OptimizeForSmallDb() {
  max_file_opening_threads = 1;
  max_open_files = 5000;
  return this;
  }

ColumnFamilyOptions* ColumnFamilyOptions::OptimizeForSmallDb() {
write_buffer_size = 2 << 20;
target_file_size_base = 2 * 1048576;
max_bytes_for_level_base = 10 * 1048576;
soft_pending_compaction_bytes_limit = 256 * 1048576;
hard_pending_compaction_bytes_limit = 1073741824ul;
return this;
}

• This is running inside a docker container
• This bad alloc error happens randomly and at least shows up after my container runs more than 1.5 hour.
• I don't see high memory consumption by the docker container.

Questions

• Turned on Dump Statistics; what data should I share/monitor?
• Any recommended configuration parameters I should use?
• Any config parameter can be used to limit max memory usage?

[siying]

Maybe we should update OptimizeForSmallDb() but I think if we want to limit the memory usage, it's a good idea to set following:

std::shared_ptr<Cache> cache = NewLRUCache(16 << 20);

Options options;

BlockBasedTableOptions table_options;
table_options.block_cache = cache;
table_options.cache_index_and_filter_blocks = true;
table_options.index_type = kTwoLevelIndexSearch;
options.table_factory.reset(new BlockBasedTableFactory(table_options));


std::shared_ptr<rocksdb::WriteBufferManager> write_buffer_manager =
   std::make_shared<rocksdb::WriteBufferManager>(0, cache);
options.write_buffer_manager = write_buffer_manager;

And I hope it will make things better. To debug where the memory is used, https://github.com/facebook/rocksdb/blob/master/include/rocksdb/utilities/memory_util.h#L20-L48 is something you can try out. It doesn't cover everything, but a good portion.

[siying]

My previous comment has a typo. I already edited it in-place. It should be options.cache_index_and_filter_blocks = true.

[siying]

I'm also trying to change the behavior of OptimizeForSmallDb() for future releases.

[philipktlin]

Thanks for your fast response. I will try out your suggested parameters today and run for few hours; will post back once test is done.

[philipktlin]

Try out recommended table option; but it doesn't work out. I still got bad_alloc error in my log.

"To debug where the memory is used, https://github.com/facebook/rocksdb/blob/master/include/rocksdb/utilities/memory_util.h#L20-L48 is something you can try out." Do you mean I update the code to call these method? why not use dump statistics? What should I look at first in the statistic dumps?

[siying]

@philipktlin memory dump will be even better!

[philipktlin]

I am new to RPi; do you know how to do memory dump on RPi?