ImportError: cannot import name caffe2_pb2

[filipetrocadoferreira]

If this is a build issue, please fill out the template below.

System information

• Operating system: 16.04
• Compiler version: 5.4.0
• CMake version:3.5.1
• CMake arguments:None
• Relevant libraries/versions (e.g. CUDA): Cuda8,Cudnn6

CMake summary output

******** Summary ********
******** Summary ********
-- General:
--   CMake version         : 3.5.1
--   CMake command         : /usr/bin/cmake
--   Git version           : v0.8.1-1194-gd6757e3
--   System                : Linux
--   C++ compiler          : /usr/bin/c++
--   C++ compiler version  : 5.4.0
--   Protobuf compiler     : /usr/local/bin/protoc
--   Protobuf include path : /usr/local/include
--   Protobuf libraries    : optimized;/usr/local/lib/libprotobuf.so;debug;/usr/local/lib/libprotobuf.so;-pthread
--   BLAS                  : Eigen
--   CXX flags             :  -std=c++11 -O2 -fPIC -Wno-narrowing -Wno-invalid-partial-specialization
--   Build type            : Release
--   Compile definitions   : 
-- 
--   BUILD_BINARY          : ON
--   BUILD_DOCS            : OFF
--   BUILD_PYTHON          : ON
--     Python version      : 2.7.12
--     Python library      : /usr/lib/x86_64-linux-gnu/libpython2.7.so
--   BUILD_SHARED_LIBS     : ON
--   BUILD_TEST            : ON
--   USE_ATEN              : OFF
--   USE_ASAN              : OFF
--   USE_CUDA              : ON
--     CUDA version        : 8.0
--     CuDNN version       : 6.0.21
--     CUDA root directory : /usr/local/cuda
--     CUDA library        : /usr/lib/x86_64-linux-gnu/libcuda.so
--     CUDA NVRTC library  : /usr/local/cuda/lib/libnvrtc.so
--     CUDA runtime library: /usr/local/cuda/lib64/libcudart.so
--     CUDA include path   : /usr/local/cuda/include
--     NVCC executable     : /usr/local/cuda/bin/nvcc
--     CUDA host compiler  : /usr/bin/cc
--   USE_EIGEN_FOR_BLAS    : 1
--   USE_FFMPEG            : OFF
--   USE_GFLAGS            : ON
--   USE_GLOG              : ON
--   USE_GLOO              : ON
--   USE_LEVELDB           : ON
--     LevelDB version     : 1.18
--     Snappy version      : 1.1.3
--   USE_LITE_PROTO        : OFF
--   USE_LMDB              : ON
--     LMDB version        : 0.9.17
--   USE_METAL             : OFF
--   USE_MKL               : 
--   USE_MOBILE_OPENGL     : OFF
--   USE_MPI               : ON
--   USE_NCCL              : ON
--   USE_NERVANA_GPU       : OFF
--   USE_NNPACK            : ON
--   USE_OBSERVERS         : ON
--   USE_OPENCV            : ON
--     OpenCV version      : 3.3.1
--   USE_OPENMP            : OFF
--   USE_PROF              : OFF
--   USE_REDIS             : OFF
--   USE_ROCKSDB           : OFF
--   USE_THREADS           : ON
--   USE_ZMQ               : OFF

#Error

python caffe2/python/operator_test/relu_op_test.py
Traceback (most recent call last):
  File "caffe2/python/operator_test/relu_op_test.py", line 21, in <module>
    from caffe2.python import core
  File "/home/ubuntu/caffe2/caffe2/python/core.py", line 29, in <module>
    from caffe2.proto import caffe2_pb2
ImportError: cannot import name caffe2_pb2

[kenkyusha]

try running:
python2 -c 'from caffe2.python import workspace; print(workspace.NumCudaDevices())'
if that does not print >0 (more than 0) then something is wrong with your build and it does not employ GPU

[filipetrocadoferreira]

I found the error.If you run the installations steps when you have an empty PYTHON PATH it will generate a corrupted one because of double ':'

[chenlh14]

@filipetrocadoferreira I run into the same problem. Could you tell me how you fixed it?

[pjh5]

@chenlh14 how did you install Caffe2?

There should be a caffe2 directory under your site-packages, something like /usr/local/lib/python2.7/site-packages/caffe2 . There should also be a libcaffe2.so located depending on how you installed, in this example it would be at /usr/local/lib/libcaffe2.so . The path to the lib folder containing libcaffe2.so should be in LD_LIBRARY_PATH

[sdalxn]

@pjh5
I also have this problem.

I have not a caffe2 directory under my /python2.7/site-packages, but I have it under /usr/local/lib/python2.7/dist-packages/ and the files in this folder are correctly installed from the installation date and time.
I also have a libcaffe2.so at /usr/local/lib/.

echo $LD_LIBRARY_PATH, I get:
/usr/local/lib:/usr/local/cuda-8.0/lib64:

echo $PYTHONPATH, I get
/usr/local::/home/lxn/caffe2/build
when;
~/caffe2$ python caffe2/python/operator_test/relu_op_test.py

I got
Traceback (most recent call last):
File "caffe2/python/operator_test/relu_op_test.py", line 21, in
from caffe2.python import core
File "/home/lxn/caffe2/caffe2/python/core.py", line 29, in
from caffe2.proto import caffe2_pb2
ImportError: cannot import name caffe2_pb2
what's wrong ?

Thanks for your continuous help!

[sdalxn]

OK , It is really because PYTHONPATH has double ':'.

[ezreal1129]

@pjh5 excuse me ,i also have the problem and i have not a caffe2 directory in /usr/local/lib/python2.7/site-packages/ and i have not libcaffe2 in /usr/local/lib/,so how to fix it

[suchaoxiao]

I have solved it in my ubuntu16.04, my way is : sudo gedit ~/.bashrc add at the end #caffe2
#echo $PYTHONPATH
export PYTHONPATH=/usr/local:$PYTHONPATH
export PYTHONPATH=$PYTHONPATH:yourcaffe2rootdir/caffe2/build
#echo $LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH

notes: $PYTHONPATH:# yourcaffe2rootdir/caffe2/build yourcaffe2rootdir you can find the build see where is it .such as mine: /home/xsc/caffe2/build
save then
source ~/.bashrc
good luck!!

[pjh5]

Everyone, please do not edit PYTHONPATH or LD_LIBRARY_PATH unless you need to. The install instructions said only to edit those if you were having import problems, because they only apply in certain cases. I have removed all of these commands from the installation instructions.

Please read https://caffe2.ai/docs/faq.html#why-do-i-get-import-errors-in-python-when-i-try-to-use-caffe2 and understand what your PYTHONPATH does before changing it.

[youngxiao]

double :
when execute the following command, then double : appears.

young@yang-xiao:~$ echo $PYTHONPATH
/usr/bin/python:/usr/local::/home/young/pytorch/caffe2/build

So delete PYTHONPATH and add new PYTHONPATH to .bashrc

sudo gedit ~/.bashrc

Add new pythonpath

#echo $PYTHONPATH
export PYTHONPATH=/usr/local
export PYTHONPATH=$PYTHONPATH:/home/young/pytorch/caffe2/build
export PYTHONPATH=$PYTHONPATH:/usr/bin/python
 
#echo $LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/usr/local/lib

Then

young@yang-xiao:~$ echo $PYTHONPATH
/usr/local:/home/young/pytorch/caffe2/build:/usr/bin/python

[Zhang-Wenwen]

I don't have a double: , and all the paths are right,but the error is still there ,what can I do ?

[pjh5]

Have you made sure your Python follows all the steps in https://caffe2.ai/docs/faq.html#why-do-i-get-import-errors-in-python-when-i-try-to-use-caffe2 ?

[Zhang-Wenwen]

I am new to caffe2,and I build it from docker image, and run it in a docker container, could it be possible to lack packages?

[CarlosYeverino]

Hi @zhangwewenwen ,

I had the same issue as you but I built caffe2 from source. I fixed it in the following way:
delete the build folder and then follow the steps from @youngxiao first. After this steps run the compilation for Caffe2 and it will be ok and no more impport issues.

[manuelli]

Hi,

I just built caffe2 from source successfully in an ubuntu 16.04 docker image following these instructions.

When I attempted to test my installation using the suggested command I get an import error on caff2e_pb2. The full error is

>>> from caffe2.python import core
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/caffe2/python/__init__.py", line 2, in <module>
    from caffe2.proto import caffe2_pb2
ImportError: cannot import name caffe2_pb2
>>> 

The location of the caffe2 python package that is being loaded is as follows.

>>> caffe2.__file__
'/usr/local/lib/python2.7/dist-packages/caffe2/__init__.pyc'
>>> import caffe2.proto 
>>> caffe2.proto.__file__
'/usr/local/lib/python2.7/dist-packages/caffe2/proto/__init__.pyc'
>>> 

The problem seems to be that caffe2_pb2 is not in /usr/local/lib/python2.7/dist-packages/caffe2/proto. Searching around for caffe2_pb2.py in the pytorch source tree shows that it appears 3 places. The following is the result of find . | grep caffe2_pb2.

./build/caffe2/proto/caffe2_pb2.py
./torch/lib/tmp_install/lib/python2.7/dist-packages/caffe2/proto/caffe2_pb2.py
./torch/lib/python2.7/dist-packages/caffe2/proto/caffe2_pb2.py

It seems like caffe2_pb2.py exists in the build tree but for some reason didn't get copied over to the install location. Do you know what might be causing this issue and/or how to fix it?

Other notes:

• I built from source using system python. I am also running my python interpreter with system python /usr/bin/python. I know this is not recommended but I am inside a docker image so I am not going to mess up my system python permanently. I can try building in a virtualenv if you think that is the issue.

• I looked at the FAQ but didn't find my answers there.

• With regards to the Summary output of cmake, is there a file that contains this information? I saw it flash past on

Summary

-- ******** Summary ********
-- General:
--   CMake version         : 3.5.1
--   CMake command         : /usr/bin/cmake
--   Git version           : v0.1.11-10353-gea0ee77
--   System                : Linux
--   C++ compiler          : /usr/bin/c++
--   C++ compiler version  : 5.4.0
--   BLAS                  : MKL
--   CXX flags             : -msse3 -msse4.1 -msse4.2   --std=c++11  -fvisibility-inlines-hidden -fopenmp -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -Wno-unused-but-set-variable -Wno-maybe-uninitialized
--   Build type            : Release
--   Compile definitions   : ONNX_NAMESPACE=onnx_torch;USE_GCC_ATOMICS=1;HAVE_MMAP=1;_FILE_OFFSET_BITS=64;HAVE_SHM_OPEN=1;HAVE_SHM_UNLINK=1;HAVE_MALLOC_USABLE_SIZE=1
--   CMAKE_PREFIX_PATH     : /usr/lib/python2.7/dist-packages
--   CMAKE_INSTALL_PREFIX  : /home/manuelli/code/data_volume/sandbox/pytorch/torch/lib/tmp_install
-- 
--   BUILD_ATEN_MOBILE     : OFF
--   BUILD_BINARY          : OFF
--   BUILD_CUSTOM_PROTOBUF : ON
--     Link local protobuf : ON
--   BUILD_DOCS            : OFF
--   BUILD_PYTHON          : ON
--     Python version      : 2.7.12
--     Python executable   : /usr/bin/python
--     Pythonlibs version  : 2.7.12
--     Python library      : /usr/lib/python2.7
--     Python includes     : /usr/include/python2.7
--     Python site-packages: lib/python2.7/dist-packages
--   BUILD_SHARED_LIBS     : ON
--   BUILD_TEST            : ON
--   USE_ASAN              : OFF
--   USE_CUDA              : 1
--     CUDA static link    : 0
--     USE_CUDNN           : ON
--     CUDA version        : 8.0
--     cuDNN version       : 7.2.1
--     CUDA root directory : /usr/local/cuda
--     CUDA library        : /usr/local/nvidia/lib64/libcuda.so
--     cudart library      : /usr/local/cuda/lib64/libcudart_static.a;-pthread;dl;/usr/lib/x86_64-linux-gnu/librt.so
--     cublas library      : /usr/local/cuda/lib64/libcublas.so;/usr/local/cuda/lib64/libcublas_device.a
--     cufft library       : /usr/local/cuda/lib64/libcufft.so
--     curand library      : /usr/local/cuda/lib64/libcurand.so
--     cuDNN library       : /usr/lib/x86_64-linux-gnu/libcudnn.so.7
--     nvrtc               : /usr/local/cuda/lib64/libnvrtc.so
--     CUDA include path   : /usr/local/cuda/include
--     NVCC executable     : /usr/local/cuda/bin/nvcc
--     CUDA host compiler  : /usr/bin/cc
--     USE_TENSORRT        : OFF
--   USE_ROCM              : OFF
--   USE_EIGEN_FOR_BLAS    : ON
--   USE_FFMPEG            : OFF
--   USE_GFLAGS            : OFF
--   USE_GLOG              : OFF
--   USE_LEVELDB           : ON
--     LevelDB version     : 1.18
--     Snappy version      : 1.1.3
--   USE_LITE_PROTO        : OFF
--   USE_LMDB              : ON
--     LMDB version        : 0.9.17
--   USE_METAL             : OFF
--   USE_MKL               : 
--   USE_MOBILE_OPENGL     : OFF
--   USE_NCCL              : ON
--     USE_SYSTEM_NCCL     : OFF
--   USE_NERVANA_GPU       : OFF
--   USE_NNPACK            : 1
--   USE_OBSERVERS         : ON
--   USE_OPENCL            : OFF
--   USE_OPENCV            : OFF
--   USE_OPENMP            : OFF
--   USE_PROF              : OFF
--   USE_REDIS             : OFF
--   USE_ROCKSDB           : OFF
--   USE_ZMQ               : OFF
--   USE_DISTRIBUTED       : ON
--     USE_MPI             : OFF
--     USE_GLOO            : ON
--     USE_GLOO_IBVERBS    : OFF
--   Public Dependencies  : Threads::Threads
--   Private Dependencies : nnpack;cpuinfo;/usr/lib/x86_64-linux-gnu/liblmdb.so;/usr/lib/x86_64-linux-gnu/libleveldb.so;/usr/lib/x86_64-linux-gnu/libsnappy.so;/usr/lib/x86_64-linux-gnu/libnuma.so;gloo;gloo;aten_op_header_gen;onnxifi_loader;rt;gcc_s;gcc;dl
-- Configuring done

[sdalxn]

@manuelli ,
You can inspect your PYTHONPATH.
In Terminal, type:echo $PYTHONPATH
You should get：
echo $LD_LIBRARY_PATH, I get:
/usr/local/lib:/usr/local/cuda-8.0/lib64

echo $PYTHONPATH, I get
/usr/local:/home/yourname/caffe2/build

[pjh5]

@manuelli what you described was indeed a bug which has since been fixed.

@sdalxn you probably shouldn't have PYTHONPATH set to anything. Setting it to your build folder will cause python to search in build/*.py files and also in copies of those files in your python site-packages, which may diverge over time if you are editting locally. /usr/local/lib should also not be on your LD_LIBRARY_PATH, unless you have a separate reason for having it there and understand your use case.

[akanazawa]

@pjh5 Hi, can you please elaborate on this bug that has been fixed? I just pulled from the latest repo and I still have this issue. It could be for a different issue, but could you please point to the fix?
Thanks!

[pjh5]

@akanazawa the fix was something along the lines of https://github.com/pytorch/pytorch/blob/master/setup.py#L1255 , basically setuptools wasn't including the Caffe2 python source files in the installation directory for the binary builds in certain cases. If you're still getting the error then you have a different problem.

[akanazawa]

@pjh5 Thank you for the update! I think this could've fixed it. I ended up going around this issue with docker.