Getting out of memory exception when inserting millions of records #64

Closed
ajermakovics opened this Issue Apr 6, 2012 · 1 comment

Projects

None yet

2 participants

@ajermakovics
        DB db = DBMaker.openFile("bigdb2").make();
        SortedMap<String, Data> map = db.getTreeMap("bigmap");
        if( map == null ) map = db.createTreeMap("bigmap");

        for(int i = 0; i < 1000000000; i++)
        {
            map.put("key"+i, new Data("data" + i) );

            if( i % 1000 == 0 )
            {
                System.out.println(i);
                db.commit(); // excepton on this line
            }
        }

        db.close();

Data class is a pojo with one string field, getter/setter and default constructor.
Exception thrown at around 4M inserted records:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
    at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
    at net.kotek.jdbm.BlockIo.setDirty(BlockIo.java:92)
    at net.kotek.jdbm.BlockIo.writeByte(BlockIo.java:155)
    at net.kotek.jdbm.RecordHeader.setCurrentSize(RecordHeader.java:65)
    at net.kotek.jdbm.PhysicalRowIdManager.write(PhysicalRowIdManager.java:314)
    at net.kotek.jdbm.PhysicalRowIdManager.update(PhysicalRowIdManager.java:87)
    at net.kotek.jdbm.DBStore.update2(DBStore.java:343)
    at net.kotek.jdbm.DBStore.update(DBStore.java:318)
    at net.kotek.jdbm.DBCache.updateCacheEntries(DBCache.java:135)
    at net.kotek.jdbm.DBCache.commit(DBCache.java:90)

Data class:

    public static class Data implements Serializable
    {
        String str;

        Data() { }
        public Data(String s) { str = s; }

        public String getStr() {
            return str;
        }

        public void setStr(String str) {
            this.str = str;
        }
    }
@jankotek
Owner
jankotek commented Apr 8, 2012

There is memory leak in MRU cache, use DBMaker.enableHardCache() for now.

@jankotek jankotek pushed a commit that referenced this issue Apr 8, 2012
@dan-g dan-g Fixed MRU memory leak #64 5214b21
@jankotek jankotek closed this Apr 8, 2012
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment