Plain Datumfile input format for minimum memory usage

[immars]

When training with leveldb/lmdb, memory increases linearly with iterations and even with Next()s for DataLayer's rand_skip.

Here's simple plain datum file format via std::fstream to address this issue.

Pros

• RAM usage basically stay constant (<1G, on googlenet small batches) during training as expected
• datum file size between leveldb and lmdb format
• no noticeable impact on training speed because of prefetch
• concurrent read for multiple process

Cons

• no random read. But caffe does not need random key-value access anyway.

[sguada]

Why this value?

[immars]

It prevents a too large key_size or data_size read from file, maybe from file corruption.
I thought 10M for a datum is large enough, or maybe should be larger? 100M?

[immars]

Thanks for the review @sguada !

[weiliu89]

@immars Thanks for the pull! I have been using it, and found that when I start N training jobs accessing the same datumfile, each one only uses 100/N % of CPU. Is it normal? I am not sure if it is going to make the training slower or not.

[immars]

@weiliu89 this should not be happening, not according to my test. No locking is used, training process should not be IO bound either. Are you running N process? what's your iostat -kx 1 output ? or nvidia-smi?

[shelhamer]

Closing as better addressed by the Python layer. There are many types of data, and as long as it can be handled in Python it can be handled as a Python layer.