MDB_PAGE_NOTFOUND Error while creating db

[hahne]

Hello,
I haved moved my job folder and everything was working fine but now I wanted to create a new database and I get the following error:

2017-02-16 12:57:03 [DEBUG] 200358 total lines in file
2017-02-16 12:57:03 [INFO ] 200358 valid lines in file
2017-02-16 12:57:03 [DEBUG] Category 0 has 100179 images.
2017-02-16 12:57:03 [DEBUG] Category 1 has 100179 images.
2017-02-16 12:57:05 [DEBUG] Processed 100/200358
2017-02-16 12:57:07 [DEBUG] Processed 200/200358
2017-02-16 12:57:09 [DEBUG] Processed 300/200358
2017-02-16 12:57:11 [DEBUG] Processed 400/200358
2017-02-16 12:57:13 [DEBUG] Processed 500/200358
2017-02-16 12:57:13 [ERROR] PageNotFoundError: mdb_put: MDB_PAGE_NOTFOUND: Requested page not found
Traceback (most recent call last):
File "/home/johannes/nvidia/digits/digits/tools/create_db.py", line 792, in
hdf5_dset_limit=args['hdf5_dset_limit'],
File "/home/johannes/nvidia/digits/digits/tools/create_db.py", line 294, in create_db
mean_files, **kwargs)
File "/home/johannes/nvidia/digits/digits/tools/create_db.py", line 358, in _create_lmdb
_write_batch_lmdb(db, batch, images_written)
File "/home/johannes/nvidia/digits/digits/tools/create_db.py", line 665, in _write_batch_lmdb
lmdb_txn.put(key, datum.SerializeToString())
lmdb.PageNotFoundError: mdb_put: MDB_PAGE_NOTFOUND: Requested page not found

Not sure if it is related anyhow to the job folder. Can you help me with this?

My DIGITS version is 5.1-dev

[lukeyeager]

I don't think I've ever seen that error before. BVLC/caffe#3731 seems to be working well for everyone, and we're basically doing the same thing in DIGITS.

How did you install LMDB? You didn't install from source or anything weird like that?

$ dpkg -l | grep lmdb
ii  liblmdb-dev:amd64                           0.9.17-3                                      amd64        Lightning Memory-Mapped Database development files
ii  liblmdb0:amd64                              0.9.17-3                                      amd64        Lightning Memory-Mapped Database shared library
ii  python-lmdb                                 0.87-2                                        amd64        Lightning Memory-Mapped Database python bindings

$ pip list | grep -i lmdb
lmdb (0.87)

[hahne]

Thank you for your help!

No I did not install LMDB from source or something. The thing is that it was working before but it is a few weeks ago that a created a new dataset.

I get the same output like you executing these commands.

[hahne]

Ok I was running create_db.py now manually and changed the output path to a local path. Here it is working.

Due to memory issues, I moved the job folder to a network drive. The network drive is a windows share mounted with cifs. It has a normal file path (/home/username/networks/data). But apparently this is causing issues for lmdb...

[lukeyeager]

Technically LMDB shouldn't ever be used over NFS (http://stackoverflow.com/a/24696604/2404152). But I've done it a lot and don't often see any issues.

[hahne]

You think it will work if I read only the database?

My quick solution for now is: I changed create_db.py that it will write the database temporarily on the local disk and moves it then to the job folder.