Incompatible with CPP protobuf implementation for Python

[junghoahnsc]

Hello,

(
I originally filed this to googleapis/google-cloud-python#2971, but this seems to be an issue in protobuf.

cc: @dhermes, @bjwatson, @geigerj
)

I'm trying to use C++ protobuf implementation for Python since pure Python proto is too slow.
But importing gcloud.bigtable package throws an exception TypeError: Descriptors should not be created directly, but only retrieved from their parent..

Is there any way to fix this problem?

Thanks,

1. OS type and version
   Tested on ubuntu 14.04 / osx 10.12.

2. Python version and virtual environment information python --version
   Python 2.7.6

3. google-cloud-python version pip show google-cloud, pip show google-<service> or pip freeze
   google-cloud 0.22.0 / 0.18.3

4. Stacktrace if available

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jungho.ahn/pyproto/protobuf/xx/local/lib/python2.7/site-packages/google/cloud/bigtable/__init__.py", line 18, in <module>
    from google.cloud.bigtable.client import Client
  File "/home/jungho.ahn/pyproto/protobuf/xx/local/lib/python2.7/site-packages/google/cloud/bigtable/client.py", line 33, in <module>
    from google.longrunning import operations_grpc
  File "/home/jungho.ahn/pyproto/protobuf/xx/local/lib/python2.7/site-packages/google/longrunning/operations_grpc.py", line 47, in <module>
    options=None),
  File "/home/jungho.ahn/pyproto/protobuf/python/google/protobuf/descriptor.py", line 494, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors should not be created directly, but only retrieved from their parent.

5. Steps to reproduce

// Install CPP proto implementation.
$ git clone https://github.com/google/protobuf.git
$ cd protobuf
$ ./autogen.sh
$ CXXFLAGS="-fPIC -g -O2" ./configure
$ make -j12
$ export PROTOC=$PWD/src/protoc
$ cd python
$ python setup.py bdist_wheel --cpp_implementation --compile_static_extension
$ pip uninstall protobuf
$ pip install dist/<wheel file name> (e.g., protobuf-3.2.0-cp27-cp27mu-linux_x86_64.whl)

// Enable CPP proto implementation.
$ export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp
$ python -c 'from gcloud import bigtable'

6. Code example
   from gcloud import bigtable

[gfernandessc]

The same issue happens when using google.cloud.bigtable from google-cloud 0.22 or 0.23.

Can we get some resolution here please?

[dhermes]

/cc @lukesneeringer

[TeBoring]

protobuf's cpp and python implementation has different undefined behavior. I think the google-cloud api depends on these undefined behaviors. Unless google-cloud api has a new release that removes these api, you can not use protobuf cpp implementation.

…

On Mon, Feb 27, 2017 at 09:33 Danny Hermes ***@***.***> wrote: /cc @lukesneeringer <https://github.com/lukesneeringer> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2646 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AE9H5Z5MCiBOWe3wm1s7TnpVBIAMoiERks5rgwj3gaJpZM4Lxwos> .

[dhermes]

@TeBoring The error is from a Python file generated via protoc. We don't have any custom behavior, it's all just vanilla usage of protobuf.

[TeBoring]

May I see the stack trace?

…

On Mon, Feb 27, 2017 at 09:59 Danny Hermes ***@***.***> wrote: @TeBoring <https://github.com/TeBoring> The error is from a Python file generated via protoc. We don't have any custom behavior, it's all just vanilla usage of protobuf. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2646 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AE9H5YEuffUOjR82wzOVSo4ans4G1KBmks5rgw8IgaJpZM4Lxwos> .

[dhermes]

I am referring to the stack trace in the beginning of this issue:
#2646 (comment)

[xfxyjwf]

@dhermes The stack trace doesn't reveal any generated protobuf files. Can you provide the .proto file that can reproduce the error?

[xfxyjwf]

The google/longrunning/operations_grpc.py file in the stack trace might be a generated file of grpc. @haberman mentioned he might know what's going on.

[dhermes]

Here is the proto.

For comparison, here is the vanilla protoc generated file and here is the protoc generated file with the gprc plugin. (@lukesneeringer's team generates these so I'm not 100% sure how it's done.)

[lukesneeringer]

I actually dealt with this issue a couple weeks ago:

• That error happens in protobuf 3.2.x if there is a descriptor being generated in a filename that does not end with _pb2.
• google/longrunning/operations_grpc.py is provided by google-cloud-core, if I recall correctly. If it is creating descriptors, we need to rename the module (and then expose it in the old module path).

[lukesneeringer]

@xfxyjwf @gfernandessc Can you verify (with pip freeze) that you are using googleapis-common-protos with a version of 1.5.2 or above? I pushed out a fix a couple weeks ago for this issue, but if you are still on 1.5.1, you will not have the fix.

If it is still broken after upgrading to 1.5.2, let me know and I will investigate.

Also tagging the relevant pull request for reference.

[gfernandessc]

Thanks Luke! I can confirm it works now with 1.5.2. Thank you everyone for digging into this over the past day.

[kpayson64]

@dhermes
The _grpc.py file linked here
Seems to be out of date. I'm not sure if its already resolved at head, but using the latest version of grpcio-tools generates a valid _grpc.py file (no protobuf descriptors)

[lukesneeringer]

Yeah. I fixed that in 1.5.2.
You can close this, since one of the original folks who raised it has confirmed that it works in the updated version.

[xfxyjwf]

Thanks for confirming!