Huge Memory & CPU Usage

[KeKl]

Hello,

I compared three versions of a test application. I saved a lot (10^7) objects of a type with three properties of type double.
The first version was coded with saving to a file with json serialization (Newtonsoft Json). The second was with litedb. The third with simply saving to a txt-File (File.WriteAllLines).

Code LiteDB:

// Open database (or create if not exits)
using(var db = new LiteDatabase(@".\LiteDB.db"))
{
    // Get customer collection
    var col = db.GetCollection<Node>("nodes");
    col.Insert(nodes);
}

The results are:

Version	Memory Usage	Disk Space Usage	Time Needed [ms]
Newtonsoft Json	~500MB	673MB	68,630ms
LiteDB	~6 675MB	3 476MB	497,464ms
Text File	~500MB	950MB	38,913ms

Is there a faster way of saving bigger datas?

[mbdavid]

Hi @KeKl, thanks for your comment.

Performance and memory use drops in LiteDB when you are working with huge mass of data. This reason is because LiteDB implement a full transaction and journal backup save. Transactions are memory only (thats why use too many memory) and journal write in disk twice.

To works fast with huge data in LiteDB, try to use:

using(var db = new LiteDatabase(@"filename=.\LiteDB.db; journal=false"))
{
    var col = db.GetCollection<Node>("nodes");
    col.InsertBulk(nodes);
}

• nodes var must returns and IEnumerable<Node> and will be better if uses yield return

This code must run fast, but never as Newtonsoft or Text File. LiteDB implements indexes (at least _id index), data in pages, and has a much more complex data structure.

I have some plans to remove some features (like transactions and journals) in prior be simple, less memory usage and faster.

[KeKl]

Hi @mbdavid,

thank you for your answer.

I tested also your implementation. It´s faster.

Thank you!

Regards,
Kevin