Cannot open db - LMDB, via python, Windows 10

[CarlosYeverino]

If you have a question or would like help and support, please ask at our
forums.

If you are submitting a feature request, please preface the title with [feature request].
If you are submitting a bug report, please fill in the following details.

Issue description

I was able to compile pytorch with the aim of using Caffe2. I have tested some libraries using python and it works well. However, I would like to use data bases for training but when I try to use lmdb there is an error:

RuntimeError                              Traceback (most recent call last)
<ipython-input-16-107e99931720> in <module>()
     17 
     18 # Initialize and create the training network
---> 19 workspace.RunNetOnce(train_model.param_init_net)
     20 workspace.CreateNet(train_model.net, overwrite=True)
     21 # Initialize and create validation network

D:\Yeverino\git_projects\pytorch\build\caffe2\python\workspace.pyc in RunNetOnce(net)
    197         C.Workspace.current._last_failed_op_net_position,
    198         GetNetName(net),
--> 199         StringifyProto(net),
    200     )
    201 

D:\Yeverino\git_projects\pytorch\build\caffe2\python\workspace.pyc in CallWithExceptionIntercept(func, op_id_fetcher, net_name, *args, **kwargs)
    176 def CallWithExceptionIntercept(func, op_id_fetcher, net_name, *args, **kwargs):
    177     try:
--> 178         return func(*args, **kwargs)
    179     except Exception:
    180         op_id = op_id_fetcher()

RuntimeError: [enforce fail at db.h:190] db_. Cannot open db: D:\caffe2_notebooks\tutorial_data\cifar10\training_lmdb of type lmdb Error from operator: 
output: "dbreader_D:\\caffe2_notebooks\\tutorial_data\\cifar10\\training_lmdb" name: "" type: "CreateDB" arg { name: "db_type" s: "lmdb" } arg { name: "db" s: "D:\\caffe2_notebooks\\tutorial_data\\cifar10\\training_lmdb" }

For the setup I used python 2.7 and installed all required packages. I installed the package lmdb as well. However, it seems that it is not recognized for the compilation of Caffe2. Furthermore, I set USE_LMDB=ON but it is not recognized by CMake since it has no impact in the CMake configuration that is printed out in the console. The file CMake_Cache.txt (attached)
CMakeCache.txt
also indicates that path for LMDB was not found LMDB_DIR:PATH=LMDB_DIR-NOTFOUND . How can I successfully use LMDB for Caffe2 using my configuration? I am stuck at this step for a while, any help would be very appreciated.

Provide a short description.
LMDB error - cannot open db

Code example

Follow Cifar10 or MNIST Tutorial.

Please try to provide a minimal example to repro the bug.
Error messages and stack traces are also helpful.

System Info

Please copy and paste the output from our
environment collection script
(or fill out the checklist below manually).

You can get the script and run it with:

wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py

• PyTorch or Caffe2: Pythorch cloned to be able to use Caffe2
• How you installed PyTorch (conda, pip, source): github
• Build command you used (if compiling from source):
• OS: Windows 10
• PyTorch version: current version
• Python version: 2.7
• CUDA/cuDNN version: 8.0/7.0.5
• GPU models and configuration: NVIDIA GeForce GTX 1050
• GCC version (if compiling from source):
• CMake version: cmake-3.12.0-rc2-win64-x64.msi
• Versions of any other relevant libraries: Visual Studio 2015

[pjh5]

Did you make sure to clear the cmake cache and all build artifacts before rebuilding with -DUSE_LMDB=ON ?

Can you tell me more about your lmdb? Where did you install it from? Where did you install it to? What version and how did you download it?

This is the logic used to find lmdb https://github.com/pytorch/pytorch/blob/master/cmake/Modules/FindLMDB.cmake and it's called from https://github.com/pytorch/pytorch/blob/master/cmake/Dependencies.cmake#L219 . If you want to iterate quickly, you can add a message(FATAL_ERROR "Stopping early") after this line https://github.com/pytorch/pytorch/blob/master/cmake/Dependencies.cmake#L228 and keep making changes until lmdb is found successfully.

[CarlosYeverino]

Hi @pjh5 ,

thanks for your help.

When the compilation fails, I delete the folder pytorch\build and then set configurations(including set USE_LMDB=ON) and finally execute build_windows.bat for the next try.

I installed lmdb via python 2.7 using:
pip install lmdb

lmdb is installed in path C:\Python27\Lib\site-packages. There are two folders:
lmdb and lmdb-0.94-py2.7.egg-info

There is no error for code lines like:
import lmdb
...
env = lmdb.open(lmdb_path, map_size=LMDB_MAP_SIZE)

However when the execution reaches the following:
workspace.RunNetOnce(train_model.param_init_net)
the above-mentioned error appears.

[zhaiyuqiang]

I have the same problem when I run the MNIST example and I install caffe2 by miniconda.
Many guys have this problem, If this is a bug, I hope someone can fix it as soon as possible.

error is:

RuntimeError: [enforce fail at db.h:190] db_. Cannot open db: /home/ubuntu/caffe2_notebooks/tutorial_data/mnist/mnist-train-nchw-lmdb of type lmdb
Error from operator: 
output: "dbreader_/home/ubuntu/caffe2_notebooks/tutorial_data/mnist/mnist-train-nchw-lmdb" name: "" type: "CreateDB" arg { name: "db_type" s: "lmdb" } arg { name: "db" s: "/home/ubuntu/caffe2_notebooks/tutorial_data/mnist/mnist-train-nchw-lmdb" }frame #0: c10::ThrowEnforceNotMet(char const*, int, char const*, std::string const&, void const*) + 0x59 (0x7fe4bd34cf49 in /home/ubuntu/miniconda2/lib/python2.7/site-packages/caffe2/python/../../torch/lib/libc10.so)
frame #1: <unknown function> + 0x135efa3 (0x7fe4e5b54fa3 in /home/ubuntu/miniconda2/lib/python2.7/site-packages/caffe2/python/../../torch/lib/libcaffe2.so)
frame #2: <unknown function> + 0x11a12b5 (0x7fe4e59972b5 in /home/ubuntu/miniconda2/lib/python2.7/site-packages/caffe2/python/../../torch/lib/libcaffe2.so)
frame #3: caffe2::SimpleNet::Run() + 0x1b2 (0x7fe4e5a8ad62 in /home/ubuntu/miniconda2/lib/python2.7/site-packages/caffe2/python/../../torch/lib/libcaffe2.so)
frame #4: caffe2::Workspace::RunNetOnce(caffe2::NetDef const&) + 0x25a (0x7fe4e5ac415a in /home/ubuntu/miniconda2/lib/python2.7/site-packages/caffe2/python/../../torch/lib/libcaffe2.so)
frame #5: <unknown function> + 0x53078 (0x7fe4e6db9078 in /home/ubuntu/miniconda2/lib/python2.7/site-packages/caffe2/python/caffe2_pybind11_state_gpu.so)
frame #6: <unknown function> + 0x88be8 (0x7fe4e6deebe8 in /home/ubuntu/miniconda2/lib/python2.7/site-packages/caffe2/python/caffe2_pybind11_state_gpu.so)
<omitting python frames>

my env is

PyTorch version: 1.0.0.dev20181029
Is debug build: No
CUDA used to build PyTorch: 8.0.61

OS: Ubuntu 14.04.5 LTS
GCC version: (Ubuntu 4.8.4-2ubuntu1~14.04.4) 4.8.4
CMake version: version 2.8.12.2

Python version: 2.7
Is CUDA available: Yes
CUDA runtime version: 8.0.44
GPU models and configuration: GPU 0: Quadro P4000
Nvidia driver version: 384.59
cuDNN version: Probably one of the following:
/usr/local/cuda-8.0/lib64/libcudnn.so.7.0.3
/usr/local/cuda-8.0/lib64/libcudnn_static.a

Versions of relevant libraries:
[pip] numpy (1.15.3)
[pip] torch (1.0.0.dev20181029)
[conda] cuda80                    1.0                  h205658b_0    pytorch
[conda] pytorch-nightly           1.0.0.dev20181029 py2.7_cuda8.0.61_cudnn7.1.2_0  [cuda80]  pytorch

[orionr]

@zhaiyuqiang looks like CMake can't find LMDB when building. Can you see if there's some adjustment to cmake/Modules/FindLMDB.cmake that you could do to have it find this? Maybe just try changing

if(MSVC)
  find_package(LMDB NO_MODULE)
else()
  find_path(LMDB_INCLUDE_DIR NAMES  lmdb.h PATHS "$ENV{LMDB_DIR}/include")
  find_library(LMDB_LIBRARIES NAMES lmdb   PATHS "$ENV{LMDB_DIR}/lib" )
endif()

to

find_path(LMDB_INCLUDE_DIR NAMES  lmdb.h PATHS "$ENV{LMDB_DIR}/include")
find_library(LMDB_LIBRARIES NAMES lmdb   PATHS "$ENV{LMDB_DIR}/lib" )

Unfortunately I don't think we test LMDB installs on CI.

[pjh5]

@zhaiyuqiang to add to this, the binaries are not built with lmdb, so if by "install caffe2 by miniconda" you mean you used "conda install" then it won't work. To use lmdb right now you'd have to build with source.

[CarlosYeverino]

Hello @orionr ,

I try changing the cmake file as you mentioned. However, it still does not work. I got the same error:

workspace.RunNetOnce(train_model.param_init_net)
 File "D:\Yeverino\git_projects\pytorch\build\caffe2\python\workspace.py", line 199, in RunNetOnce
   StringifyProto(net),
 File "D:\Yeverino\git_projects\pytorch\build\caffe2\python\workspace.py", line 178, in CallWithExceptionIntercept
   return func(*args, **kwargs)
RuntimeError: [enforce fail at db.h:190] db_. Cannot open db: ./data\mnist_mnistClassifier_net\mnist-train-nchw-lmdb of type lmdb Error from operator:
output: "dbreader_./data\\mnist_mnistClassifier_net\\mnist-train-nchw-lmdb" name: "" type: "CreateDB" arg { name: "db_type" s: "lmdb" } arg { name: "db" s: "./data\\mnist_mnistClassifier_net\\mnist-train-nchw-lmdb" } device_option { device_type: 1 cuda_gpu_id: 0 }
[F context_gpu.h:117] Check failed: error == cudaSuccess driver shutting down

I noticed that the LMDB flag was OFF even when I set it to ON before executing build_windows.bat.

Is it possible to get a LMDB fix/patch for the tag v0.4.0? I already developed some CNN predictors using that tag on ubuntu and I would like to extend the support for windows 10.

[orionr]

Apologies for the slow response. Unfortunately it would be difficult to get a patch in place for 0.4, but 1.0 will be moving from nightly to release shortly, so please verify everything works there and we can try to patch there as needed.