pycaffe requires caffe_pb2, which breaks Python 3 (?)

[AmNotGod]

Hi, everyone. I have to use pycaffe for python3.4 currently. The process of compiling is all right by following the guidance of cmake installation. However, something is wrong when I tried to use pycaffe:

Failed to include caffe_pb2, things might go wrong!
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/protobuf-3.0.0a3.dev0-py3.4.egg/google/protobuf/internal/python_message.py", line 890, in MergeFromString
File "/usr/local/lib/python3.4/dist-packages/protobuf-3.0.0a3.dev0-py3.4.egg/google/protobuf/internal/python_message.py", line 912, in InternalParse
File "/usr/local/lib/python3.4/dist-packages/protobuf-3.0.0a3.dev0-py3.4.egg/google/protobuf/internal/decoder.py", line 189, in ReadTag
TypeError: unsupported operand type(s) for &: 'str' and 'int'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "../../tools/detect.py", line 16, in
from fast_rcnn.config import cfg
File "/home/amax/Data/qingxin/fast-rcnn/tools/../lib/fast_rcnn/init.py", line 9, in
from . import train
File "/home/amax/Data/qingxin/fast-rcnn/tools/../lib/fast_rcnn/train.py", line 10, in
import caffe
File "/home/amax/Data/qingxin/fast-rcnn/tools/../caffe-fast-rcnn/python/caffe/init.py", line 4, in
from .proto.caffe_pb2 import TRAIN, TEST
File "/home/amax/Data/qingxin/fast-rcnn/tools/../caffe-fast-rcnn/python/caffe/proto/caffe_pb2.py", line 608, in
options=_descriptor._ParseOptions(descriptor_pb2.FieldOptions(), '\020\001')),
File "/usr/local/lib/python3.4/dist-packages/protobuf-3.0.0a3.dev0-py3.4.egg/google/protobuf/descriptor.py", line 834, in _ParseOptions
File "/usr/local/lib/python3.4/dist-packages/protobuf-3.0.0a3.dev0-py3.4.egg/google/protobuf/message.py", line 185, in ParseFromString
File "/usr/local/lib/python3.4/dist-packages/protobuf-3.0.0a3.dev0-py3.4.egg/google/protobuf/internal/python_message.py", line 896, in MergeFromString
google.protobuf.message.DecodeError: Truncated message.

---

I can't figure it out. Any help will be appreciated!

[longjon]

Note that Python 3 is not (yet) tested, so support is not guaranteed (though we aim not to break it).

It looks to me like #1790 might be causing this issue by forcing the caffe module to import caffe_pb2, though I would have expected to have heard more noise if that were so. (@shelhamer?)

Can you reproduce this with a simple import caffe in the latest master?

[AmNotGod]

I suppose so. The version of caffe I am using is the latest one.....

[shelhamer]

It looks to me like #1790 might be causing this issue by forcing the caffe module to import caffe_pb2, though I would have expected to have heard more noise if that were so. (@shelhamer?)

This sounds like it could be the problem. I only did that to avoid a parallel declaration between proto / C++ / Python. I haven't encountered any trouble, but I haven't tried protobuf 3 or Python 3. We could instead hardcode the constants or do some kind of conditional import (is there such a thing as caffe_pb3 for protobuf 3? I'm not caught up on the latest.)

[CyanVladimirS]

What's the current status ? I'm also trying latest caffe with Python 3.4. When I run example Caffe code I got same error: "TypeError: unsupported operand type(s) for &: 'str' and 'int'" But it looks like Google.Protobuf error? Is anyone running Caffe with python3?

[CaffeBin]

Same issue here. Have you managed to overcome this issue?

[CyanVladimirS]

I switched back to Python 2.7 ...

[CaffeBin]

I left python 3.5 as well and went with python 3.3

Seem to be working in 3.3

[paalge]

I also ran into this problem (python 3.4), and as I don't have any of the other libraries I've compiled in python 2.7, this breaks caffe for me (I know that python 3 isn't supported yet). I tried protocolbuffers/protobuf#1276 (without anaconda), but couldn't get it to work.

[andrey-iliyov]

I also have this problem.
Works for me: I installed python3.5 with conda 4.0.5. Please take a look: protocolbuffers/protobuf#1276
Steps 2,3 help me.

[botev]

I'm trying to construct benchmarks for several different deep learning packages. However on the cluster we need to use python3 so there is no resolution to that.
Could someone just point out if something can be hard-coded modified to make this work?

[willyd]

I use caffe with python35 and a patched protobuf 3.1.0 daily without any problems. The patch is for MSVC 2015 support because I am on windows. I would upgrade protobuf to 3.1 or 3.2 and see if the issue goes away.

[botev]

@willyd Could you elaborate? Because I have python3.4 with protobuf 2.5 installed.

[xxy19404]

At the beginning, I use anaconda3, the version of python is 3.5, it doesn't work, anyway. But i use python 2, everything is ok. If python3 is not necessary. I suggest turn to python2.

[shelhamer]

Closing as fixed—Caffe is compatible with py35.

[ZubairKhan001]

i have compiled caffe on Ubuntu 16.04 LTS both with python 2 and python, it is really a hectic process for beginners, for caffe with python 2.7 make sure that your /.local/lib/python2.7/site-packages only contains a version same as you protoc version. protoc version can be checked by entering

protoc --version

for me i got problems with python 2.7 because by mistake i installed protbuf 3.0 alpha in python 2.7 site packages, so there was a protobuf error after removing it from python 2.7 site packages everything worked fine
also install python-dev version

apt-get install the python3.x-dev

and for python 3.5 make sure you have python 3.0 alpha, if you have both python 2.7 and python 3.x in the system use pip3 otherwise use pip

pip3 install protobuf==3.0.0-alpha-3

or another protobuf such as 3.5 version of protobuf.
also do this

pip3 install python-dateutil --upgrade

after that do

make clean

make all -j8
make runtest -j8

make pycaffe

after performing make all make runtest steps don't forget to do that

sudo ldconfig

hope everything works OK for you.