Is there any limitation of WriteBatch?

[derekbit]

Hi,
I want to delete multiple key-val pairs (several millions to billion pairs) atomically from rocksdb by using WriteBatch method.
Is any limitation of the WriteBatch method, e.g. the maximum number of key-val pairs , memory usage or others?
Thanks

[BradBarnich]

would you be able to use DeleteRange?

[siying]

I can't think of a limit before you run out of memory. But it's not a good practice to insert large write batch. It's hard to predict the performance impact.

[adamretter]

@siying any guidance on what constitutes "large"?

[siying]

@adamretter good question. I don't know. Thinking about you are writing one such an entry to WAL, and inserting them to memtable in one operation. In the mean time, we double the memory usage. Or course, the impact will be workload specific. Maybe they don't care about slowing down other writes and have plenty of DRAM to waste. But I would be really cautious when one write batch is larger than a few MBs.

[derekbit]

Thanks! We reduced the on big write batch to multiple small write batches to avoid the large memory usages.