Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python: use relative imports in generated modules #1491

Closed
little-dude opened this issue May 5, 2016 · 69 comments
Assignees
Labels

Comments

@little-dude
Copy link

@little-dude little-dude commented May 5, 2016

I have a package foo that looks like this:

.
├── data
│   ├── a.proto
│   └── b.proto
└── generated
    ├── a_pb2.py
    ├── b_pb2.py
    └── __init__.py
# a.proto
package foo;
# b.proto
import "a.proto";

package foo;

Generate the code: protoc -I ./data --python_out=generated data/a.proto data/b.proto.
Here is the failure:

Python 3.5.1 (default, Mar  3 2016, 09:29:07) 
[GCC 5.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from generated import b_pb2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/corentih/repro/generated/b_pb2.py", line 16, in <module>
    import a_pb2
ImportError: No module named 'a_pb2'

This is beacuse the generated code looks like this:

import a_pb2

If the import was relative it would actually work:

from . import a_pb2
@little-dude little-dude changed the title use relative imports in generated modules python: use relative imports in generated modules May 6, 2016
@goldenbull

This comment has been minimized.

Copy link

@goldenbull goldenbull commented May 10, 2016

I have the exactly same problem, hope to be fixed

little-dude added a commit to little-dude/protobuf that referenced this issue May 10, 2016
@little-dude

This comment has been minimized.

Copy link
Author

@little-dude little-dude commented May 10, 2016

@goldenbull I submitted a fix, let's see if it makes it through. I'm just not sure: are there cases where we don't want relative imports?

@goldenbull

This comment has been minimized.

Copy link

@goldenbull goldenbull commented May 11, 2016

@little-dude how about if a_pb2.py is generated into a different folder as b_pb2.py?

@little-dude

This comment has been minimized.

Copy link
Author

@little-dude little-dude commented May 11, 2016

Could you provide a small example of what you're thinking about, so that I try it with my change?

@goldenbull

This comment has been minimized.

Copy link

@goldenbull goldenbull commented May 11, 2016

.
├── proto
│   ├── a.proto
│   └── b.proto
├── pkg_a
│   ├── a_pb2.py
│   └── __init__.py
└── pkg_b
     ├── b_pb2.py
     └── __init__.py

maybe this is not a good case, I don't have enough knowledge about how protobuf/python/etc. deal with the importing

@little-dude

This comment has been minimized.

Copy link
Author

@little-dude little-dude commented May 11, 2016

I don't think this is actually possible because the generated modules follow the hierarchy of the proto files.
However we could imagine that we have the following:

.
└── data
    ├── a.proto
    ├── b.proto
    └── sub
        ├── c.proto
        └── sub
             └── d.proto

with the following:

# a.proto
package foo;
import "b.proto";
import "sub/c.proto";
import "sub/sub/d.proto";

# b.proto
package foo;
import "sub/c.proto";
import "sub/sub/d.proto";

# sub/c.proto
package foo;
import "sub/d.proto";

# sub/sub/d.proto
package foo;

We generate the code with:

protoc -I data -I data/sub -I data/sub/sub --python_out=generated data/a.proto data/b.proto data/sub/c.proto data/sub/sub/d.proto

which generated the following:

.
└── generated
    ├── a_pb2.py
    ├── b_pb2.py
    └── sub
        ├── c_pb2.py
        └── sub
            └── d_pb2.py

But this is a more complex case than what I am trying to fix.

Edit: I'm not even sure this is a valid case but here is the error I'm getting with the master branch (4c6259b):

In [1]: from generated import a_pb2
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1-f28bccc761b6> in <module>()
----> 1 from generated import a_pb2

/home/corentih/repro/generated/a_pb2.py in <module>()
     14 
     15 
---> 16 import b_pb2 as b__pb2
     17 from sub import c_pb2 as sub_dot_c__pb2
     18 from sub.sub import d_pb2 as sub_dot_sub_dot_d__pb2

/home/corentih/repro/generated/b_pb2.py in <module>()
     14 
     15 
---> 16 from sub import c_pb2 as sub_dot_c__pb2
     17 from sub.sub import d_pb2 as sub_dot_sub_dot_d__pb2
     18 

/home/corentih/repro/generated/sub/c_pb2.py in <module>()
     14 
     15 
---> 16 from sub import d_pb2 as sub_dot_d__pb2
     17 
     18 

/home/corentih/repro/generated/sub/sub/d_pb2.py in <module>()
     20   package='foo',
     21   syntax='proto2',
---> 22   serialized_pb=_b('\n\x0fsub/sub/d.proto\x12\x03\x66oo')
     23 )
     24 _sym_db.RegisterFileDescriptor(DESCRIPTOR)

TypeError: __init__() got an unexpected keyword argument 'syntax'
@little-dude

This comment has been minimized.

Copy link
Author

@little-dude little-dude commented Jun 1, 2016

@haberman is there any chance for this to be fixed before next release? It's quite limiting for python 3.

@drats

This comment has been minimized.

Copy link

@drats drats commented Jun 17, 2016

I have exactly the same problem (using protobuf v3beta3), the generated imports do not conform to PEP 328 (finalized in 2004), restated in the Python docs: https://docs.python.org/3/tutorial/modules.html#intra-package-references - the 12 yr-old specification is enforced in Python3, so generated protobufs are unusable without further modification.

@asgoel

This comment has been minimized.

Copy link

@asgoel asgoel commented Jul 15, 2016

any updates on this?

@haberman

This comment has been minimized.

Copy link
Contributor

@haberman haberman commented Jul 29, 2016

Yikes, sorry for the slow reply on this.

Wouldn't relative imports break the case that you are importing protos from a different pip package?

For example, the well-known types come from the google-protobuf pip package. If we merge a change to use relative imports, imports of google/protobuf/timestamp.proto (for example) would be broken.

@drats

This comment has been minimized.

Copy link

@drats drats commented Aug 8, 2016

@haberman This bug has to do with protos importing protos in the same package, even in the same directory. The compiler converts this into relative imports with defective syntax under Python 3, so the generated code cannot execute at all. I don't see how you can get away without using relative imports in this case. I've had to manually edit the compiler generated pb2.py files to get them to work at all.

@oc243

This comment has been minimized.

Copy link

@oc243 oc243 commented Oct 22, 2016

+1 for fixing this bug. It's stopping me migrating from python2 to python3.

@brynmathias

This comment has been minimized.

Copy link

@brynmathias brynmathias commented Nov 15, 2016

+1 for fixing as well

@27359794

This comment has been minimized.

Copy link

@27359794 27359794 commented Nov 18, 2016

+1, as far as I can tell this completely prevents proto imports in Python 3. Seems extremely worrying that this isn't fixed.

EDIT: this is not quite right, see my comment below.

@ylwu

This comment has been minimized.

Copy link

@ylwu ylwu commented Nov 18, 2016

+1 for fixing

@xfxyjwf

This comment has been minimized.

Copy link
Contributor

@xfxyjwf xfxyjwf commented Nov 18, 2016

I believe protobuf is working as intended in this case. The python package generated for a .proto file mirrors exactly the relative path of the .proto file itself. For example, if you have a .proto file "proto/a.proto", the generated python code must be "proto/a_pb2.py" and it must be in the "proto" package. In @little-dude 's example, if you want the generated code in the "generated" package, the .proto files themselves must be put in the "generated" directory:

└── generated
    ├── a.proto
    ├── b.proto

with protoc invoked as:

$ protoc --python_out=. generated/a.proto generated/b.proto

This way, the output will have the correct import statements (it will be "import generated.a_pb2" rather than "import a_pb2").

Using relative imports only solves the problem when all generated py code is put in the same directory. That's not the case when you import protos from other projects though (e.g., use protobuf's well-known types). It will likely break more than it fixes.

@haberman

This comment has been minimized.

Copy link
Contributor

@haberman haberman commented Nov 19, 2016

I am confused by the claims that this is totally broken in Python 3. We have Python 3 tests (and have for a while) that are passing AFAIK. Why would Python 3 require relative imports?

@27359794

This comment has been minimized.

Copy link

@27359794 27359794 commented Nov 21, 2016

The issue that I'm having and that I believe others are having is that the proto code import "foo.proto" compiles into the Python3 code import foo_pb2. However, implicit relative imports were disabled in Python3, so relative imports must be of the form from . import foo_pb2. Manually changing the generated proto code to this form after proto compilation fixes the issue.

There are already multiple existing issues concerning this problem, and it first seems to have been recognised in 2014 (!!!): #90, #762, #881, #957

@27359794

This comment has been minimized.

Copy link

@27359794 27359794 commented Nov 25, 2016

I read a bit more about Python 3's import rules and I think I can give a better explanation.

In Python 3 the syntax import foo imports from the interpreter's current working directory, from $PYTHONPATH, or from an installation-dependent default. So if you compile proto/foo.proto to gen/foo_pb2.py, the syntax import foo_pb2 works only if the current working directory is gen/ or if you placed gen/ on your python path.

If you are compiling protos as part of a Python package (which is the case in most non-trivial Python projects), the interpreter's current working directory is the directory of your main module (suppose the directory is mypackage/), and modules in the package must either use fully-qualified absolute imports (e.g. import mypackage.gen.foo_pb2) or relative imports (e.g. from .gen import foo_pb2).

In Python 2, a module inside gen/ could do import foo_pb2 and this would import mypackage.gen.foo_pb2 into its namespace, regardless of the current working directory. This is an implicit relative import.

In Python 3, implicit relative imports don't exist and import foo_pb2 will not find foo_pb2.py, even if the module importing foo_pb2 is inside gen/. This is the issue that people are complaining about in the thread.


The root of this problem seems to be that import "foo.proto"; needs to compile into from <absolute or relative package path> import foo_pb2 when the proto is inside a package, and import foo_pb2 otherwise. Neither syntax will work in both scenarios. The proto compiler ignores the package name in the proto file and only observes the directory structure of the proto files, so if you want the from <path> import foo_pb2 output you need to place your protos in a directory structure mirroring the Python structure. For instance, if you have the following directory structure and you set the proto path to proto_files/ and python_out to mypackage/proto/, the correct import line is generated, but the compiled python is put in the wrong directory.

Pre-compilation:

proto_files/
  mypackage/
    proto/
      foo.proto  # import "mypackage/proto/bar.proto";
      bar.proto
mypackage/
  qux/
    mymodule.py  # import mypackage.proto.foo_pb2
  proto/

Post-compilation:

proto_files/
  mypackage/
    proto/
      foo.proto  # import "mypackage/proto/bar.proto";
      bar.proto
mypackage/
  qux/
    mymodule.py  # import mypackage.proto.foo_pb2`
  proto/
    mypackage/
      proto/
        foo_pb2.py  # from mypackage.proto import bar_pb2 (the import we want! but file should be in ../../)
        bar_pb2.py

This is close to the desired result, but not quite it, because now the absolute reference to the compiled file is mypackage.proto.my_package.proto.foo_pb2 rather than mypackage.proto.foo_pb2.

In this instance you can actually get it to produce the right output by specifying the python output path mypackage/. Here, the compiler detects that it doesn't need to create mypackage/proto because it already exists, and it just plops the generated files in that directory. However, this doesn't play nicely when the project directory structure makes use of symlinks. e.g. if mypackage/proto is a symlink to somewhere else and you actually want to dump the compiled protos there instead.

I think the 'correct' fix is to make use of the proto package rather than the location of the proto in the directory structure.

@haberman

This comment has been minimized.

Copy link
Contributor

@haberman haberman commented Nov 30, 2016

@DanGoldbach Thanks very much for all of the detail. I think a lot of the confusion here has been a result of not fully explaining all of the background and assumptions we are making. The more full description really helps clarify things.

Let me first respond to this:

I think the 'correct' fix is to make use of the proto package rather than the location of the proto in the directory structure.

Can you be more specific about exactly what fix you are proposing? An example would help.

One thing people seem to want, but that doesn't seem to work in practice, is that a message like this:

package foo.bar;

message M {}

...can be imported like this in Python:

from foo.bar import M

That is a very natural thing to want, but doesn't work out, as I described here: grpc/grpc#2010 (comment)

Overall, your directory structure appears to be more complicated than what we generally do at Google (which is the environment where all this behavior was designed/evolved). At Google we generally have a single directory structure for all source files, including protos. So we would anticipate something more like this:

Pre-compilation:

mypackage/
  foo.proto  # import "mypackage/bar.proto";
  bar.proto
  qux/
    mymodule.py  # import mypackage.foo_pb2

Post-compilation:

mypackage/
  foo.proto  # import "mypackage/proto/bar.proto";
  foo_pb2.py # import mypackage.proto.bar_pb2
  bar.proto
  bar_pb2.py
  qux/
    mymodule.py  # import mypackage.proto.foo_pb2

Because protobuf thinks in terms of this single, flat namespace, that's why we get a little confused when people talk about needing relative imports. I haven't wrapped my head around why this is necessary. Why doesn't the scheme I outlined above work for your use case?

@27359794

This comment has been minimized.

Copy link

@27359794 27359794 commented Nov 30, 2016

Thanks, I understand much better now.

Can you be more specific about exactly what fix you are proposing?

I meant that it would be nice if the compiled proto module hierarchy mirrored the package hierarchy specified in the proto source file. As you pointed out in the grpc thread, this isn't feasible right now. Maybe in the future, the one-to-one restriction between proto sources and gens can be relaxed.

It sounds like protos work best when the generated files compile to the same directory as the source files, as per your example. Our directory structure has a separate build/ directory for generated code which isn't indexed by source control.

/build/  # generated code directory
  proto/
    # compiled protos go here
/python/  # parent directory for python projects
  my_python_pkg/  # root of this python package
    proto -> /build/proto/  # symlink to compiled proto dir
    main.py  # import my_python_pkg.proto.compiled_proto_pb2

We explicitly keep generated and source files separate, so your scheme doesn't suit our current repo layout.

We would also like the option of using those protos in multiple distinct Python packages in the future, so generating compiled protos into one particular Python package isn't ideal. At Google this isn't an issue because IIRC the entire repo acts like one massive Python package and blaze provides you with the bits of the repo that you need.

I think we'll get around this by either adding the compiled proto directory to our Python path or by writing a build command to manually edit the imports in the generated protos to be package-relative imports.

Hopefully this helps other people reading the thread.

@haberman

This comment has been minimized.

Copy link
Contributor

@haberman haberman commented Nov 30, 2016

Cool, glad we're getting closer to understanding the problems.

Maybe in the future, the one-to-one restriction between proto sources and gens can be relaxed.

I think this would be difficult to do. Right now we guarantee that:

$ protoc --python_out=. foo.proto bar.proto

...is equivalent to:

$ protoc --python_out=. foo.proto
$ protoc --python_out=. bar.proto

This is important because it's what allows the build to be parallelized. At Google we have thousands of .proto files (maybe even tens or hundreds of thousands of files, haven't checked lately) that all need to be compiled for a given build. It's not practical to do one big protoc run for all of them.

It's also not practical to try and ensure that all .proto files with a given (protobuf) package get compiled together. Protobuf doesn't require the file/directory to match the package, so .proto files for package foo could exist literally anywhere in the whole repo. So we have to allow that two different protoc runs will both contain messages for the same package.

So with these constraints we're a bit stuck. It leads to the conclusion that we can't have more than one .proto file put symbols into the same Python module, because the two protoc runs would overwrite the same output file.

We would also like the option of using those protos in multiple distinct Python packages in the future, so generating compiled protos into one particular Python package isn't ideal.

Usually for this case we would put the generated code for those protos into a package that multiple other packages can use. Isn't that usually the solution when you want to share code?

If you have foo/bar.proto that you want to share across multiple packages, can't you put it in a package such that anyone from any package can import it as foo.bar_pb2?

@27359794

This comment has been minimized.

Copy link

@27359794 27359794 commented Nov 30, 2016

I hadn't considered the constraints placed on the proto compiler by Google's scale and parallelism requirements, but that makes sense.

I guess I can compile the protos into their own proto-only package in build/ and then import that package from wherever I need it. I think you still need to add that the parent of that package to the python path.

@hindman

This comment has been minimized.

Copy link

@hindman hindman commented Jan 10, 2017

@DanGoldbach Thanks for your example -- it helped me solve a problem.

I think your example work as desired if you run protoc like this:

protoc --python_out . --proto_path proto_files proto_files/mypackage/proto/*.proto

It generates correct import lines and places the _pb2.py files in the correct location.

@W35170

This comment has been minimized.

Copy link

@W35170 W35170 commented Mar 27, 2019

@haberman I'm having issues with the #1491 (comment) solution. I am generating C++ and python code from the same .proto files.

If I use the mentioned comment's structure, the Python works but the C++ autogenerated code fails to compile. This is what my setup looks like:

cpp_source/
    - MyClass.cpp #include "proto/bar.pb.h"
    - MyClass.hpp
    - Makefile
    - proto/
        - bar.proto # import "proto/foo.proto" [BAD]
        - bar.pb.cc #include "proto/bar.pb.h" [BAD]
        - bar.pb.h  #include "proto/foo.pb.h" [BAD]
        - foo.proto
        - foo.pb.cc
        - foo.pb.h
...

python_package/
    - my_module.py # import python_package.proto.bar [OK]
    - proto/
        - bar_pb2.py # from proto import foo_pb2 as proto_dot_foo__pb2 [OK]
        - foo_pb2.py

I am compiling the proto files on the makefile by doing:

protoc --python_out=/path/to/python_package/ --cpp_out=. proto/*.proto

As you can see, the includes in the autogenerated C++ code are bad.

Originally, when I was only generating C++ code, the imports inside the .proto files DID NOT have the proto/ prefix and I was compiling the proto files by:

protoc --proto_path=proto --cpp_out=proto proto/*.proto

which resulted in:

cpp_source/
    - MyClass.cpp #include "proto/bar.pb.h"
    - MyClass.hpp
    - Makefile
    - proto/
        - bar.proto # import "foo.proto"
        - bar.pb.cc #include "bar.pb.h"
        - bar.pb.h  #include "foo.pb.h"
        - foo.proto
        - foo.pb.cc
        - foo.pb.h
@mkobit

This comment has been minimized.

Copy link

@mkobit mkobit commented Apr 19, 2019

Another frustration we have ran into is that the Python protobuf generator treats filenames as part of the contract

If you have a file named my-stuff.proto then the my-stuff is used in generation, instead of just using the types and the package and other information actually contained in the protobuf contract itself to generate the module.

@chiuup

This comment has been minimized.

Copy link

@chiuup chiuup commented Apr 24, 2019

@haberman I'm having issues with the #1491 (comment) solution. I am generating C++ and python code from the same .proto files.

If I use the mentioned comment's structure, the Python works but the C++ autogenerated code fails to compile. This is what my setup looks like:

cpp_source/
    - MyClass.cpp #include "proto/bar.pb.h"
    - MyClass.hpp
    - Makefile
    - proto/
        - bar.proto # import "proto/foo.proto" [BAD]
        - bar.pb.cc #include "proto/bar.pb.h" [BAD]
        - bar.pb.h  #include "proto/foo.pb.h" [BAD]
        - foo.proto
        - foo.pb.cc
        - foo.pb.h
...

python_package/
    - my_module.py # import python_package.proto.bar [OK]
    - proto/
        - bar_pb2.py # from proto import foo_pb2 as proto_dot_foo__pb2 [OK]
        - foo_pb2.py

I am compiling the proto files on the makefile by doing:

protoc --python_out=/path/to/python_package/ --cpp_out=. proto/*.proto

As you can see, the includes in the autogenerated C++ code are bad.

Originally, when I was only generating C++ code, the imports inside the .proto files DID NOT have the proto/ prefix and I was compiling the proto files by:

protoc --proto_path=proto --cpp_out=proto proto/*.proto

which resulted in:

cpp_source/
    - MyClass.cpp #include "proto/bar.pb.h"
    - MyClass.hpp
    - Makefile
    - proto/
        - bar.proto # import "foo.proto"
        - bar.pb.cc #include "bar.pb.h"
        - bar.pb.h  #include "foo.pb.h"
        - foo.proto
        - foo.pb.cc
        - foo.pb.h

you can separate the generation into two commands which uses different root/out folders.

@iamliamc

This comment has been minimized.

Copy link

@iamliamc iamliamc commented May 24, 2019

I postprocess generated files with:

sed -i -E 's/^import.*_pb2/from . \0/' *.py

This makes all imports relative. Works great!

My group had to got a few steps beyond that.. to handle the build of multiple proto files and then do the replacement in a series of subfolders... we haven't quite nailed it down i'm sure some bash master might gag but but it's working for now.

micro_service_components
    |__ definitions ___
                      |__ proto_folder_1 -> proto_folder_1.proto
                      |__ proto_folder_2 -> proto_folder_2.proto
   |__ gen __
             |__ proto_folder_1 -> proto_folder_1_pb2.py, proto_folder_1_pb2_grpc.py
             |__ proto_folder_2 -> proto_folder_2_pb2.py, proto_folder_2_pb2_grpc.py

# Re-build protocol
#
echo ""
echo "---------------------Generating protocols----------------------"
for D in definitions/*; do
    if [ -d "${D}" ]; then
        echo "Processing folder $D:"
        serviceName="${D#definitions/}"
        outDir="gen"

        for file_name in $D/*; do
            proto_file="${file_name}"
            echo "------ converting $proto_file --> $outDir"
            if [ ! -d $outDir ]; then
              mkdir -p $outDir;
            fi
            python -m grpc_tools.protoc --python_out=$outDir --grpc_python_out=$outDir $proto_file --proto_path definitions
            temp="${file_name#$D/}"
            no_extension="${temp%.*}"

            gsed -i -r "s/(from\s)(.*)/from ..\2/g"  "gen/$serviceName/${no_extension}_pb2_grpc.py" || echo "Broke on replacement relative might be nothing to do"

        done
    else
        serviceName="${D#definitions/}"
        extension="${serviceName##*.}"
        if [ "${extension}" = "proto" ]; then
            file_name="${serviceName}"
            outDir="gen"
            moduleDir="gen"
            proto_file="${file_name#$D/}"
            echo "------ converting $proto_file --> $outDir"
            if [ ! -d $outDir ]; then
              mkdir -p $outDir;
            fi
            python -m grpc_tools.protoc -I definitions --python_out=$outDir --grpc_python_out=$outDir $proto_file
            temp="${file_name#$D/}"
            no_extension="${temp%.*}"

            gsed -i -r "s/(from\s)(.*)/from ..\2/g"  "gen/${no_extension}_pb2_grpc.py" || echo "Broke on replacement relative might be nothing to do"
        fi
    fi
done

echo "------------------------------Done------------------------------"
echo ""
aykevl added a commit to karpenoktem/kninfra that referenced this issue Jun 17, 2019
There was a problem with sharing protobuf messages between files. It has
been fixed by following the structure described in this comment:
protocolbuffers/protobuf#1491 (comment)
aykevl added a commit to karpenoktem/kninfra that referenced this issue Jun 25, 2019
There was a problem with sharing protobuf messages between files. It has
been fixed by following the structure described in this comment:
protocolbuffers/protobuf#1491 (comment)
hueyjj added a commit to greenmochi/ultimate-proto that referenced this issue Jul 2, 2019
We need to do this because the protobuf tool doesn't yet compile the correct relative paths for python files.

Currently, using this hack grpc/grpc#9575 (comment)

Apparently, there is a fix, but it doesn't work for us since we need a central repo to handle all the protobufs.
protocolbuffers/protobuf#1491 (comment)

Most likely, we will switch to a hacky bash scrip with sed and awk to fix the generated stubs...
@keepsimple1

This comment has been minimized.

Copy link

@keepsimple1 keepsimple1 commented Jul 26, 2019

with grpcio-tools 1.22.0 and Python 3.7.2, I still have this issue, i.e. grpc generated import line does not work due to relative imports.

Sorry this thread is so long, what was the final resolution?

@akselilukkarila

This comment has been minimized.

Copy link

@akselilukkarila akselilukkarila commented Jul 31, 2019

So is this fixed or what are the correct workarounds not involving modifying the generated files?

import YsAppStatusService_pb2 as YsAppStatusService__pb2 causes ModuleNotFound error with Python 3.

@mitar

This comment has been minimized.

Copy link

@mitar mitar commented Jul 31, 2019

This is not fixed. Not sure why it is closed.

@keepsimple1

This comment has been minimized.

Copy link

@keepsimple1 keepsimple1 commented Jul 31, 2019

Just to share that, the following comment from another similar thread seems to be a good answer, grpc/grpc#9575 (comment) . It works for me for now. Still think protoc can be improved regarding this issue.

@mitar

This comment has been minimized.

Copy link

@mitar mitar commented Jul 31, 2019

This are not relative imports. If you use generated code as a git submodule, for example, you need relative imports. The best solution I know of is running little postprocessing.

@JasonGilholme

This comment has been minimized.

Copy link

@JasonGilholme JasonGilholme commented Aug 17, 2019

Reading the Packages section of this page: https://developers.google.com/protocol-buffers/docs/proto3 specifically the python section:

In Python, the package directive is ignored, since Python modules are organized according to their location in the file system.

That's all good but it ignores half the problem. Yes, python modules are organized in that manner, but their location affects the import statements. IMO, if you have a package specified in your proto file, it should create a package directory inside the directory provided to the python_out arg. For example, this command:

protoc --python_out=generated ./messages.proto

should result in this dir structure:

generated
    - package_name
        - messages_pb2.py

the imports would then use the package name:

import package_name.messages_pb2

edit: I did just stumble on the fact that the import statements will reflect the directory structure of the .proto files. This seems like the concept of packages is being handled in two ways - one from the proto dir structure, and one from the package directive in the proto language.

@elvizlai

This comment has been minimized.

Copy link

@elvizlai elvizlai commented Aug 19, 2019

proto-project
    foo
        a.proto # package foo and has import "bar/b.proto"
    bar
        b.proto # package bar

the generate code only work in root dir, for example:

foo
    a_pb2.py
bar
    b_pb2.py
main.py # from foo import a_pb2

but I want put generated code in libgen, but this not works.

libgen
    foo
        a_pb2.py
    bar
        b_pb2.py
main.py # from libgen.foo import a_pb2

the result error is "missing module bar", a way to fix the generated .py code in b_pb.py should using

from ..foo XXXX # the generated is from foo

instead.

Is there any special params to specify when generate python?

jspahrsummers added a commit to jspahrsummers/ibdatastream that referenced this issue Sep 7, 2019
This has a long and storied history:
grpc/grpc#9575
protocolbuffers/protobuf#1491

In our repo, do this:
python -m grpc_tools.protoc -I. --python_out=. --grpc_python_out=. ibdatastream/ibdatastream.proto
@andreycizov

This comment has been minimized.

Copy link

@andreycizov andreycizov commented Sep 16, 2019

Just in case anyone is looking for solutions, here's a project that does the re-rooting of the generated bindings and also additionally provides a way to generate python class wrappers for protobuf: https://github.com/andreycizov/python-protobuf-gen

@justinfx

This comment has been minimized.

Copy link

@justinfx justinfx commented Sep 23, 2019

I just hit this as well in a project that currently supports python2/3. The import is broken for py3 because of the import statement, after being generated from a proto file in another part of the project.

Why does the solution have to involve mucking around with the path hierarchy? Can the python proto plugin not support an option like --python_out=import_from=myApp.sub:out ?

@andreycizov

This comment has been minimized.

Copy link

@andreycizov andreycizov commented Sep 24, 2019

@justinfx Google has stated multiple times that if someone likes to have this functionality they are free to submit pull requests. Nobody did that yet, as I believe?

@dendisuhubdy

This comment has been minimized.

Copy link

@dendisuhubdy dendisuhubdy commented Sep 24, 2019

Still stuck on this, any clues on where to start to build a PR for this?

@justinfx

This comment has been minimized.

Copy link

@justinfx justinfx commented Sep 24, 2019

@andreycizov fair enough. I must have missed that point and only focused on the majority of replies that talked about the path hierarchy as the solution. I ended up symlinking my proto file from the other part of the project to the python location and then using the includes/proto path combination that produces the right output.

@meyer9

This comment has been minimized.

Copy link

@meyer9 meyer9 commented Oct 15, 2019

I'm using this solution. Add an __init__.py to the directory with your protobufs and then add the following code:

import sys
sys.path.append("./pb") # this should be the directory of the generated files relative to where you're running it from
@chdsbd

This comment has been minimized.

Copy link

@chdsbd chdsbd commented Oct 29, 2019

The relative path worked for me until I called my code from a different directory. This variation using the full path worked for me.

# pb/__init__.py
import sys
from pathlib import Path
sys.path.append(str(Path(__file__).parent))
@ddfabbro

This comment has been minimized.

Copy link

@ddfabbro ddfabbro commented Oct 30, 2019

The relative path worked for me until I called my code from a different directory. This variation using the full path worked for me.

# pb/__init__.py
import sys
from pathlib import Path
sys.path.append(str(Path(__file__).parent))

So far, this is the best fix as it is independent from how I name my protobuf submodule or the name of the symbolic link to that submodule.

@danielgtaylor

This comment has been minimized.

Copy link

@danielgtaylor danielgtaylor commented Oct 31, 2019

I used the sys.path modification in an __init__.py method for a while as well but it always seems too hacky. If you're using a recent version of Python (3.7+) I've got a new protoc plugin I've been working on that solves this issue by using relative imports:

https://github.com/danielgtaylor/python-betterproto

It will also take care of generating __init__.py files as needed so imports will work.

@dashdanw

This comment has been minimized.

Copy link

@dashdanw dashdanw commented Nov 12, 2019

Just as a note, here's a small sed command I use to alter the imports of the generated files
sed -i -r 's/import (.+_pb2.*)/from . import \1/g' *_pb2*.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.