Loader bench

[kuenishi]

Add benchmark to compare cache class performance

$ time python3 main.py --preload-path ./dat
<class 'pfio.cache.multiprocess_file_cache.MultiprocessFileCache'>
Namespace(dataset_len=1024, batchsize=32, num_workers=8, num_trials=5, preload_path='./dat', preserve_path=None, cache_type='multiprocess')
trial:  1, elapsed-time(all): 3.061 sec, ave-elapsed-time(img+label): 1.352 msec
trial:  2, elapsed-time(all): 2.045 sec, ave-elapsed-time(img+label): 1.967 msec
trial:  3, elapsed-time(all): 1.953 sec, ave-elapsed-time(img+label): 2.072 msec
trial:  4, elapsed-time(all): 1.981 sec, ave-elapsed-time(img+label): 2.062 msec
trial:  5, elapsed-time(all): 1.951 sec, ave-elapsed-time(img+label): 2.114 msec

real    0m11.728s
user    0m28.036s
sys     0m37.290s
$ time python3 main.py --preload-path ./dat --cache-type "readonly"
Cache class: <class 'pfio.cache.mmap_file_cache.ReadOnlyFileCache'>
Namespace(dataset_len=1024, batchsize=32, num_workers=8, num_trials=5, preload_path='./dat', preserve_path=None, cache_type='readonly')
trial:  1, elapsed-time(all): 3.336 sec, ave-elapsed-time(img+label): 1.739 msec
trial:  2, elapsed-time(all): 1.949 sec, ave-elapsed-time(img+label): 1.737 msec
trial:  3, elapsed-time(all): 1.857 sec, ave-elapsed-time(img+label): 1.379 msec
trial:  4, elapsed-time(all): 1.873 sec, ave-elapsed-time(img+label): 1.396 msec
trial:  5, elapsed-time(all): 1.919 sec, ave-elapsed-time(img+label): 1.465 msec

real    0m11.703s
user    0m34.291s
sys     0m28.853s
$ time python3 main.py --preload-path ./dat --cache-type "file"
Cache class: <class 'pfio.cache.file_cache.FileCache'>
Namespace(dataset_len=1024, batchsize=32, num_workers=8, num_trials=5, preload_path='./dat', preserve_path=None, cache_type='file')
trial:  1, elapsed-time(all): 3.245 sec, ave-elapsed-time(img+label): 1.523 msec
trial:  2, elapsed-time(all): 1.862 sec, ave-elapsed-time(img+label): 1.587 msec
trial:  3, elapsed-time(all): 1.920 sec, ave-elapsed-time(img+label): 1.687 msec
trial:  4, elapsed-time(all): 1.959 sec, ave-elapsed-time(img+label): 1.892 msec
trial:  5, elapsed-time(all): 1.998 sec, ave-elapsed-time(img+label): 2.050 msec

real    0m11.727s
user    0m26.644s
sys     0m35.895s

[kuenishi]

Test failure is due to #228. After it's merged, restarting tests will make this success.

[kuenishi]

/test

[pfn-ci-bot]

Successfully created a job for commit 599d50d:

• Dashboard for commit 599d50d