Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Protobuf 2.6.0 incompatibility issues with Python 3.x #7

Closed
KJaunay opened this issue Sep 2, 2014 · 18 comments
Closed

Protobuf 2.6.0 incompatibility issues with Python 3.x #7

KJaunay opened this issue Sep 2, 2014 · 18 comments

Comments

@KJaunay
Copy link

@KJaunay KJaunay commented Sep 2, 2014

I've run into a problem when using Python 3.4 and protobuf 2.6.0.
I get several errors including:
protobuf/python/google/protobuf/internal/decoder.py, line 167
_DecodeVarint = _VarintDecoder((1 << 64) - 1, long)
NameError: name 'long' is not defined

@xfxyjwf xfxyjwf added this to the Python 3 support milestone Sep 3, 2014
@gpshead
Copy link

@gpshead gpshead commented Sep 3, 2014

Blanket Python 3 support was unintentionally listed in these release notes. We need to fix that up for a 2.6.x release. The code itself is ready with some 2to3 use, the setup.py and deps are not done yet. Sorry about that!

Here's my summary:

  • Support for Python 3 is almost in place. Our setup.py does not yet do the right thing for 2to3 conversions with Python 3 but for the adventurous, manually run the following files through 2to3 and it should work in Python 3.3+:

protobuf/descriptor.py
protobuf/internal/cpp_message.py
protobuf/internal/decoder.py
protobuf/internal/python_message.py
protobuf/internal/type_checkers.py
protobuf/internal/message_cpp2_test.py
protobuf/internal/message_factory_test.py
protobuf/internal/message_factory_cpp2_test.py
protobuf/internal/message_factory_python_test.py
protobuf/internal/message_python_test.py
protobuf/internal/message_test.py
protobuf/internal/reflection_cpp2_generated_test.py
protobuf/internal/reflection_python_test.py
protobuf/internal/reflection_test.py
protobuf/internal/test_util.py
protobuf/internal/text_format_test.py
protobuf/message_factory.py
protobuf/text_encoding.py
protobuf/text_format.py

setup.py also specifies an external dep on google-apputils needed in order to run the unittests. We need to push out an updated version of that with Python 3 support.

@brettcannon
Copy link

@brettcannon brettcannon commented Sep 4, 2014

Any specific reason for wanting to use 2to3 instead of something like https://github.com/python-modernize/python-modernize to do the transition once?

@mrovner
Copy link

@mrovner mrovner commented Sep 6, 2014

It says
"Modernizes Python code for eventual Python 3 migration. Build on top of
2to3"

and it's not apparent why it's better ;)

Also 2to3 support is in setuptools already.

On Thu, Sep 4, 2014 at 6:59 AM, Brett Cannon notifications@github.com
wrote:

Any specific reason for wanting to use 2to3 instead of something like
https://github.com/python-modernize/python-modernize to do the transition
once?


Reply to this email directly or view it on GitHub
#7 (comment).

Thanks,
--Mike

@brettcannon
Copy link

@brettcannon brettcannon commented Sep 7, 2014

By unifying the code base to work on both Python 2 and 3 it makes development easier and installation faster (unless you release separate Python 2 and 3 packages). By not having to wait for 2to3 you eliminate a compile step during development. As for installation, you can forgo that step entirely. Plus the coding practices used in converted source is more in line with modern best practices and syntax.

@Kentzo
Copy link

@Kentzo Kentzo commented Sep 26, 2014

I wonder if we can reuse code from my https://github.com/GreatFruitOmsk/protobuf-py3

@tseaver
Copy link
Contributor

@tseaver tseaver commented Nov 18, 2014

I'm +1 on @Kentzo's approach, and would volunteer to help review #66. As @brettcannon says, avoiding 2to3 is pretty much the "widely accepted" best practice for libraries which need to support both Python2 and Python3: e.g., see Lennart Regebro's excellent outline of strategies.

@virtuald
Copy link

@virtuald virtuald commented Nov 21, 2014

I would love to see a python 3 compatible official protobuf available by Christmas.

@virtuald
Copy link

@virtuald virtuald commented Nov 30, 2014

FWIW, using the current trunk of protobuf, the only change I had to make to the package after installation was change a number of str("", "utf-8") references in descriptor_pb2.py. Oddly enough, the protobuf I have installed from homebrew (2.6.1) generates _pb2.py files with a function that looks like this:

_b=sys.version_info[0]<3 and (lambda x:x) or (lambda x:x.encode('latin1'))

And it uses that to encode strings instead like this:

default_value=_b("").decode('utf-8')

So.. not sure why the descriptor_pb2.py file is out of date. If someone could push those changes into that file, then I believe trunk would at least work on python3 -- but obviously #66 would be even better.

@tseaver
Copy link
Contributor

@tseaver tseaver commented Nov 30, 2014

That function should not be encoding literals as latin under Py3k, especially since they are going to be immediately decoded as utf-8. Use of latin1 in anything but edgiest cases (decoding wonky / unknown data, or encoding for antique wire protocols) is a sign of papered-over trouble: "mojibake ahead", it could read.

@virtuald
Copy link

@virtuald virtuald commented Nov 30, 2014

So, I stand corrected. When I install protobuf properly, it installs/runs from trunk without any noticable problems now. Python 3.4.2 on OSX. Now someone just needs to push an updated package to pypi... :)

It would be nice to get better python3 support in trunk to support the edge cases properly, as @tseaver mentions.

@Ivoz
Copy link
Contributor

@Ivoz Ivoz commented Dec 2, 2014

@gpshead would you be fine with only supporting python 2.6+? Current readme.txt mentions 2.4.

Single-source cross python support is, FTR, like 10x easier if you only support python 2.6+ and 3.3+.

@gpshead
Copy link

@gpshead gpshead commented Dec 14, 2014

Absolutely, 2.6 is a good minimum. Anything older is obsolete (I consider 2.6 obsolete as well, but it is easy enough to support).

One of the reasons we started with some files needing to be run through 2to3 is that we have a few internal things still running on the old App Engine 2.5 legacy runtime that needed proto2 code and using 2to3 made maintaining 2.5 compatibility easier. I believe we have dealt with those internally now but I need to double check. We should absolutely be able to make a single source 2.6, 2.7, >= 3.3+ compatible version in the future.

@tseaver
Copy link
Contributor

@tseaver tseaver commented Jan 30, 2015

FWIW, #164 modernizes the python code to drop all the PY25 stuff.

@thanatos
Copy link

@thanatos thanatos commented Jul 13, 2016

Why is this bug marked as Closed & obsolete? I'm currently trying to get a third-party package (heka_py) that uses protobuf to work, and near as I can tell, the latest version is not Python 3 compatible.

There's a package python3_protobuf on PyPI (I have no idea if this is official or not) but it is on Protobuf 2.5, and unfortunately Heka assumes 2.6; so that doesn't work. protobuf itself doesn't support 3, so that also doesn't work. Near as I can tell, protocol buffers and Python 3 don't mix.

@jleclanche
Copy link

@jleclanche jleclanche commented Jul 13, 2016

@thanatos https://pypi.python.org/pypi/protobuf pip install protobuf==3.0.0b3 is python3-compatible.

@thanatos
Copy link

@thanatos thanatos commented Jul 15, 2016

@thanatos https://pypi.python.org/pypi/protobuf pip install protobuf==3.0.0b3 is python3-compatible.

While technically true, my understanding is that the major number here is bumped because it isn't backwards-compatible with compiled protocol buffers from protobuf v2; empirically, this seems to be the case; e.g., <a Message derived class>.INTEGER doesn't exist in protobuf==3.0.0b3.

(Did it change to VALUE_INTEGER_FIELD_NUMBER? (Nope.) Perhaps if this is the only change that heka_py requries, I might be able to make a small patch to heka_py to make it protobuf==3.0.0b3 compatible? But the problem seems to still stand — if I pull in a library that relies on protobuf, that restricts me to legacy Python.)

@xfxyjwf
Copy link
Contributor

@xfxyjwf xfxyjwf commented Jul 15, 2016

On Fri, Jul 15, 2016 at 2:41 PM, Roy Wellington Ⅳ notifications@github.com
wrote:

@thanatos https://github.com/thanatos
https://pypi.python.org/pypi/protobuf pip install protobuf==3.0.0b3 is
python3-compatible.

While technically true, my understanding is that the major number here is
bumped because it isn't backwards-compatible with compiled protocol buffers
from protobuf v2; empirically, this seems to be the case; e.g., .INTEGER doesn't exist in protobuf==3.0.0b3.

We bumped the major version number because of the new proto3 syntax
support, not because it's incompatible. If you find incompatible issues
with old generated code, please help file a bug for it.


You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
#7 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AIJ6iiG4RJcNqEfQ0dyXn0benZXh40lIks5qV_6bgaJpZM4Cdsma
.

@Laowai01
Copy link

@Laowai01 Laowai01 commented Feb 9, 2018

i have compiled caffe on Ubuntu 16.04 LTS both with python 2 and python, it is really a hectic process for beginners, for caffe with python 2.7 make sure that your /.local/lib/python2.7/site-packages only contains a version same as you protoc version. protoc version can be checked by entering
protoc --version
for me i got problems with python 2.7 because by mistake i installed protbuf 3.0 alpha in python 2.7 site packages, so there was a protobuf error after removing it from python 2.7 site packages everything worked fine
also install python-dev version
apt-get install the python3.x-dev
and for python 3.5 make sure you have python 3.0 alpha, if you have both python 2.7 and python 3.x in the system use pip3 otherwise use pip

pip3 install protobuf==3.0.0-alpha-3

or another protobuf such as 3.5 version of protobuf.
also do this
pip3 install python-dateutil --upgrade
after that do
make clean
make all
make runtest
make pycaffe
after performing make all make runtest steps don't forget to do that
sudo ldconfig

hope everything works OK for you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet