db_bench crashing on large number of entries

[cmumford]

Original issue 4 created by teoryn on 2011-05-10T23:17:44.000Z:

I ran a modified db_bench with --num=10 to --num=10^9 (by multiples of 10) to test the scaling of leveldb.
My modifications statically link snappy into the posix version of leveldb; I've attached the diff.
At --num=10^8 and --num=10^9 problems start occuring. stat.txt which shows the output of all the runs.

10^8 was killed during the overwrite benchmark at 23300000 ops, but I was not able to reproduce the error using 'db_bench --num=100000000 --benchmarks=overwrite'.

10^9 generated the following error during fillrandom:
put error: IO error: /tmp/dbbench/006484.log: Too many open files
Running 'db_bench --num=1000000000 --benchmarks=fillrandom' reproduced the error.

Built (patched) r27 of leveldb and r35 of snappy with:
$ gcc --version
gcc (Ubuntu/Linaro 4.4.4-14ubuntu5) 4.4.5

[cmumford]

Comment #1 originally posted by ashoemaker on 2011-06-24T09:41:46.000Z:

I cannot repro with leveldb/r33 and snappy/r43, though I can trigger it by decreasing my file descriptor limit. Is it possible that you had other files open when running db_bench? The default limit is 1024, and leveldb by default uses up to 1000. Did you try increasing your limit (ulimit -n 10000) or decreasing options.max_open_files?

[cmumford]

Comment #2 originally posted by teoryn on 2011-06-24T15:14:30.000Z:

This bug was fixed in leveldb/r28 by the merge of upstream release 1.2

[cmumford]

Comment #3 originally posted by tfarina@chromium.org on 2011-06-24T15:18:56.000Z:

Per comment 2, I'm closing this. Thanks.

[cmumford]

Comment #4 originally posted by mbherf on 2011-09-10T05:11:32.000Z:

For other OSes the soft limit can default to 256 (e.g., Solaris). Building a version that is aware of this limit is important for portability! Otherwise you have to run as root.