relative imports are not supported

[ierezell]

When using -I to import other proto files (common declaration in my use-case) the python generated file does not import correctly the generated file.

E.g :

makefile
A 
--- A.proto
Common
--- common.proto

In the makefile :

./protoc/bin/protoc  -I Common -I A
	--python_out=$(COMPILED_PROTO_DIRECTORY) \
	--mypy_out=$(COMPILED_PROTO_DIRECTORY) \
	--grpc_out=$(COMPILED_PROTO_DIRECTORY) \
	--mypy_grpc_out=$(COMPILED_PROTO_DIRECTORY) \
	A/A.proto

Which generate a file (a_pb2.py) with :

"""
@generated by mypy-protobuf.  Do not edit manually!
isort:skip_file
"""
import builtins
import common_pb2

However common_pb2 is not a module in my path ! At least it should be from common import common_pb2 (which would in my case work)

I know it's a known problem but using https://github.com/danielgtaylor/python-betterproto seems to fix it (however it comes with other things I don't want and seems less maintained than this repo)

Do you have any fix / advice

Thanks in advance
Have a great day

[nipunn1313]

I did my best to try and repro, and I found that the pyi file from mypy-protobuf matches the py file from protoc itself.

Eg - in my toy example where I tried to mirror your sample:

python-betterproto is a nice alternative for python generated code, but AFAICT, the one built into protoc doesn't support relative imports without some fumbling with pythonpath. See #32 and protocolbuffers/protobuf#1491 (comment)

As far as I can tell, mypy-protobuf is behaving to spec with the protoc generated python code, though I agree that to-spec generated code leaves things to be desired.

If I'm missing your point entirely, could you provide a repo with a full repro example and reopen?

[ierezell]

Hi @nipunn1313,

Thanks a lot for having a look at that :)

I cannot share all my code but I will place a snippet below.

packages
---- Commons
     | ------ Commons
                 | ------ Common.proto
---- MyPackage
     | -------.venv
     | ------- MakeFile
     | ------- MyModule
                   | ------- Protos
                               | ------- InputProto.proto
                               | ------- OutputProto.proto    // uses Common.proto
                               | ------- ServerProto.proto    // uses Output.proto which uses Common.proto

In the make file I'm using :

        -I MyModule/Protos \
        -I ../Commons/commons \
	--plugin=protoc-gen-mypy=.venv/bin/protoc-gen-mypy \
	--plugin=protoc-gen-mypy-grpc=.venv/bin/protoc-gen-mypy-grpc \
	--plugin=protoc-gen-grpc=.venv/bin/protoc-gen-python_grpc \
	--python_out=MyModule/Protos \
	--mypy_out=MyModule/Protos  \
	--grpc_out=MyModule/Protos  \
	--mypy_grpc_out=MyModule/Protos  \
	MyModule/Protos /InputProto.proto	MyModule/Protos /ServerProto.proto  MyModule/Protos /ServerProto.proto

which generates :

protos_output_pb2.py with import common_pb2 as common__pb2
However as you can see, common is not in my path (as it's a poetry package) so at runtime, it will not find it except with some tweaking/hacking ...

So until google fixes protobuf for python (which hasn't been in 4 years), we should use ugly hacks?
Maybe this plugin could modify generated imports (as betterproto and if you find it useful)?

Thanks for your time.
I guess the issue can be closed as (except adding a huge feature out of the scope of this package) nothing can be done.

Have a great day

[nipunn1313]

Hiya.

Sounds like you'd like protos_output_pb2.py to work differently. Unfortunately, that's something that's built into protoc (via python_out), so that's not something we can help out with in this plugin. It's the python code that won't work, not the mypy stubs.

betterproto is intended to be a replacement for --python_out (generates different _pb2.py) while mypy-protobuf is a complement to --python_out.

You can consider writing all of your protos out to the same directory (eg --mypy_out=generated/). Another option is to add all your output directories to PYTHONPATH or equivalently sys.path.

export PYTHONPATH=MyModule/Protos:Commons/common

or

sys.path.append("Commons/common")

you can look at the discussion at protocolbuffers/protobuf#1491 to see what protobuf's dev team thinks of this issue. I believe they consider your workflow supported via PYTHONPATH or sys.path - though the quirk doesn't seem well documented.

[cpcloud]

Another option is to use the tool I built (https://github.com/cpcloud/protoletariat) to automatically rewrite stock protobuf imports to relative imports. The tool rewrites imports using FileDescriptorSets, so it's aware of the generated code's structure.