-
Notifications
You must be signed in to change notification settings - Fork 15.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
MergeFrom error when using pure python message implementation #7408
Comments
I have the exact same issue on Raspberry Pi with python 3.7.3 and python 3.7.5. Somehow, it is does not happen on my Ubuntu laptop with Python 3.7.5. I run the exact same code on both but the isinstance check fails on Raspberry Pi. The class type name is the same on both Raspberry Pi and my Ubuntu PC, but the class object id is different on Raspberry Pi. I know python3 is compiled on ARM for Raspberry Pi and compiled for AMD64 on my laptop. Maybe there is a difference there. |
met same problems too. The weird thing i met is that, when running the same code on x86 platform, everything works fine. While when running code on aarch platform like jetson tx2 or xavier nx, the problem occurs. |
I had the exact same problem. In my case is simple, I used Linux Apline, the error occured. And if I use Ubuntu, everythin is fine. |
There appears to be an assumption baked into the generator code for python at
This explicitly references the module it expects the generated code to be available under. Combined with how the message classes are constructed dynamically via descriptors, this can result in 2 different object id's for the same message class definition appearing under two different modules. Taking the example above for route.proto: package a.b.c;
message Route {
string id = 1; // uuid, generated by minion
string dst = 2;
string via = 3;
string link_name = 4;
int32 table = 5;
} and network.proto:
The generated python code will look something like:
file: a/b/c/network_pb2.py
Now this all works fine if the files are actually structured such as However if you are placing the files elsewhere, or changing the prefix so that the import path is Part of what happens is that when I uncovered this by a snippet a bit like the following to a unit test that was failing on trying to upgrade to python 3.9.1:
combined with adding the line protobuf/python/google/protobuf/internal/python_message.py Lines 1318 to 1322 in d16bf91
I don't know enough of what is happening, I could see that there were 2 different imports appearing in sys.modules for the library I was working with, and the class id was different for each one. That allowed me to work out that if I imported the file that would register and then import using the module path hardcoded in the files would result in using the same ones in all locations.
It appears once rpc.network.route_pb2 is imported, there is an instance added to sys.modules as 'a.b.c.route' which is used by the other files that are part of the protobuf definition, so importing from |
#1491, #4614 and #7470 appear to suggest there is a whole bunch of fun around import paths and python generated code. I suspect the divergence between what the generated code is assuming layout to be absolute and how usage of the resulting python code may assume that it can be placed under a different root to suit application layout or even be able to provide the generated protobuf code as a library is at the core of this issue. Having popped up various times and disappearing when import load orders end up switching slightly in python versions resulting in getting the needed class rather than the alternative definition. |
I did some more digging and it turned out our local code had injected additional paths into Given the important of paths in python module imports, and the assumptions about the base package name made by the generated code, which isn't made by the generated golang code, because all files in the same directory (ignoring test files for now) are in the same package which is defined as a simple path component, I think supporting relative path imports will make this class of error go away for use in python. Might also be nice if the |
I have been facing the same issue, where the CopyFrom or MergeFrom works fine on ubuntu system by has the same error as metioned by OP on a raspberry pi. A workaround i found to work was something like this default_response = GetDefaultGatewayResponse()
route = Route()
default_response.route.ParseFromString(route.SerializeToString()) |
Got the same issue on jetson-NX, The mergeFrom function is work on my x86 laptop, but run failed on getson-NX.
|
The problem is related to |
Today I faced same issue on ARM 64 based hardware (Strange things was its working fine in Ubuntu X86 hardware) but @electrofelix workaround worked perfectly. Workaround was to just import these pb2 , pb2_grpc file from * instead of from a.b.* |
Had the same issue, in my case, VSCode added automatically an import, while our codebase usually adds the path to sys paths, so full path is not required. |
We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please add a comment. This issue is labeled |
We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please reopen it. This issue was closed and archived because there has been no new activity in the 14 days since the |
What version of protobuf and what language are you using?
Version: 3.6.1
Language: Python 3.7
What operating system (Linux, Windows, ...) and version?
What runtime / compiler are you using (e.g., python version or gcc version)
Python 3.7
What did you do?
I have two proto file. In our actual project, these two files have a lot of content, only two message are pasted to illustrate the problem.
One is route.proto:
Another is network.proto:
I run the following command in Interactive Python:
Then the following exception was thrown:
But I add
print(type(msg), cls)
got same value is<class 'route_pb2.Route'>
but id of that different.However, there is no such problem with cpp message implementation.
What did you expect to see
What did you see instead?
Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).
Anything else we should know about your project / environment
It is similar to another issue #5272 ?
The text was updated successfully, but these errors were encountered: