Berkeley DB Java Engine Tuning Tips

Here are some useful tips for the java BDB engine for Voldemort.
1. Recommended Boilerplate Config

See battle tested config at

config/prod_single_node_cluster/config/server.properties

2. If you have a large JVM, try allocating about 60% to the BDB cache .

Modify parameter bdb.cache.size inside the server.properties. This will significantly boost performance for operations by keeping more index in memory. Obviously you will want to tune this depending on your workload.

For a production server grade, JVM config, see

bin/voldemort-prod-server.sh

3. Look underneath the covers at BDB-JE Sensors.

See mbean located at:

voldemort.store.bdb.stats:type=<store-name>
voldemort.store.bdb.stats:type=aggregated-bdb

Run operation getEnvStatsAsString() and look at the javadocs for com.sleepycat.je.EnvironmentStats

- Now if you are experiencing high read times, pay attention to the nCacheMiss number, this indicates that your data access patterns going to disk instead of hitting memory.
- If you see the disk usage growing and growing, check the cleanerbacklog, percentage utilization mbeans. If 1 cleaner cannot catch up, try throwing more.

Example Output:

I/O: Log file opens, fsyncs, reads, writes, cache misses.
	bufferBytes=3,145,728
	endOfLog=0xa3/0x16b8e9b
	nBytesReadFromWriteQueue=0
	nBytesWrittenFromWriteQueue=0
	nCacheMiss=0
	nFSyncRequests=0
	nFSyncTimeouts=0
	nFSyncs=0
	nFileOpens=0
	nLogBuffers=3
	nNotResident=0
	nOpenFiles=3
	nRandomReadBytes=0
	nRandomReads=0
	nRandomWriteBytes=0
	nRandomWrites=0
	nReadsFromWriteQueue=0
	nRepeatFaultReads=0
	nSequentialReadBytes=0
	nSequentialReads=0
	nSequentialWriteBytes=0
	nSequentialWrites=0
	nTempBufferWrites=0
	nWriteQueueOverflow=0
	nWriteQueueOverflowFailures=0
	nWritesFromWriteQueue=0
Cache: Current size, allocations, and eviction activity.
	adminBytes=4,743
	cacheTotalBytes=3,225,039
	dataBytes=74,250
	lockBytes=318
	nBINsStripped=0
	nEvictPasses=0

3. Visit the Oracle Berkeley DB Java Edition Forums:

http://forums.oracle.com/forums/forum.jspa?forumID=273&start=150

4. Estimate your required BDB cache size….

Use the nice utility included in the java berkeley DB jar file.

http://www.oracle.com/technology/documentation/berkeley-db/je/java/com/sleepycat/je/util/DbCacheSize.html

Below tells us that we need 26 GB of memory allocated to the BDB cache if we need to keep 20 million entries in cache.

 $ java -jar je-4.0.92.jar DbCacheSize -records 20000000 -key 20 -data 1000

Inputs: records=20000000 keySize=20 dataSize=1000 nodeMax=128 density=80% overhead=10%

    Cache Size      Btree Size  Description
--------------  --------------  -----------
 1,746,091,324   1,571,482,192  Minimum, internal nodes only  
 2,364,425,475   2,127,982,928  Maximum, internal nodes only  
25,390,535,768  22,851,482,192  Minimum, internal nodes and leaf nodes  
26,008,869,920  23,407,982,928  Maximum, internal nodes and leaf nodes  

Btree levels: 4