Build python grpc?

[king--kong]

By using io.grpc:protoc-gen-grpc-java as an artifact for grpc plugin, I can generate grpc libraries in Java.

But I could not figure out how to generate python grpc libraries with protobuf-gradle-plugin.

Is it not supported yet?

[zhangkun83]

The python code generator is in the main grpc repository. Unlike the Java gRPC code generator which provides precompiled artifacts, you will need to compile the python generator by yourself. Feel free to ask questions on grpc-io@googlegroups.com on how to do it.

[kdubezerra]

As an alternative, there's this plugin, which seems to support building for both java an python: https://github.com/andrewkroh/gradle-protobuf-plugin

[gnz00]

This would be a cool addition. Personally could use it for cpp.

[mkobit]

Is there an easy way to execute the Python gRPC generation using the Gradle plugin? I tried searching the forum, but did not find much of anything. Or, is there something similar to #177 for Python gRPC?

[gnz00]

@mkobit You can try this (haven't run it in awhile):

project(":proto") {
    apply plugin: "java"
    apply plugin: "com.google.protobuf"

    buildDir = "$rootProject.buildDir/proto"

    protobuf {
        protoc {
            artifact = "com.google.protobuf:protoc:${protoVersion}"
        }
        plugins {
            grpc_java {
                artifact = "io.grpc:protoc-gen-grpc-java:${grpcVersion}"
            }
            grpc_python {
                path = getPluginPath("python")
            }
            grpc_ruby {
                path = getPluginPath("ruby")
            }
            grpc_cpp {
                path = getPluginPath("cpp")
            }
        }
        generateProtoTasks {
            all()*.builtins {
                java {}
                python {}
                ruby {}
                cpp {}
            }
            all()*.plugins {
                grpc_java {
                    outputSubDir = "java"
                }
                grpc_python {
                    outputSubDir = "python"
                }
                grpc_ruby {
                    outputSubDir = "ruby"
                }
                grpc_cpp {
                    outputSubDir = "cpp"
                }
            }
        }

        generatedFilesBaseDir = "$buildDir/generated/src"
    }

    dependencies {
        compile "com.google.protobuf:protobuf-java:${protoVersion}"
        compile "io.grpc:grpc-protobuf:${grpcVersion}"
        compile "io.grpc:grpc-netty:${grpcVersion}"
        compile "io.grpc:grpc-stub:${grpcVersion}"
    }
}

static def getPluginPath(name) {
    def path = "which grpc_${name}_plugin".execute()
    path.waitFor()
    path = path.in.text.trim()
    return path
}

[gnz00]

@mkobit Also looks like theres a Python example in the README.md

Edit: Added the getPluginPath function definition above.

[mkobit]

@gnz00 thanks, the example you provided is helpful. Hopefully I can take that and run with it. I did see the Python in the README but haven't figured out yet how to get gRPC plugin-side of it to work.

I'm looking through some Python gRPC documentation, but I am really struggling trying to find the path to the plugin that is installed. Everything I see is doing python -m grpc_tools.protoc <args> with no reference to using it as a plugin to protoc (which looks like it is executed by Java in this case). I feel like with my pip installation I should be finding a grpc_python_plugin binary somewhere but I'm definitely just stupidly missing something.

[gnz00]

@mkobit
Here is how I was building with Docker on RHEL. Mac OSX I think you just need brew install grpc protobuf.

RUN yum install -y autoconf                   \
                   automake                   \
                   build-essential            \
                   gcc-c++                    \
                   gflags-devel               \
                   glibc-headers              \
                   git                        \
                   java-1.8.0-openjdk-devel   \
                   libtool                    \
                   make                       \
                   unzip                      \
                   wget                       \
                   && yum -y clean all

ENV APP_DIR=...
ENV PROTOBUF_DIR=/opt/protobuf
ENV GRPC_DIR=/opt/grpc

ARG PROTOBUF_VERSION=3.4.0
ARG GRPC_VERSION=1.6.6
ARG GRADLE_VERSION=4.2.1

RUN wget https://services.gradle.org/distributions/gradle-$GRADLE_VERSION-bin.zip
RUN unzip -d /opt/gradle gradle-$GRADLE_VERSION-bin.zip
RUN git clone --depth 1 --branch v$PROTOBUF_VERSION  https://github.com/google/protobuf $PROTOBUF_DIR
RUN git clone --depth 1 --branch v$GRPC_VERSION  https://github.com/grpc/grpc/ $GRPC_DIR

ENV PATH=$PATH:/opt/gradle/gradle-$GRADLE_VERSION/bin/
ENV PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib/pkgconfig:/usr/lib/pkgconfig:/usr/share/pkgconfig
ENV LD_LIBRARY_PATH=$LD_LIBARY_PATH:/usr/local/lib/

# Install protobuf
WORKDIR $PROTOBUF_DIR
RUN git submodule update --init --recursive
RUN ./autogen.sh
RUN ./configure
RUN make
RUN sudo make install
RUN sudo ldconfig

# Install GRPC
WORKDIR $GRPC_DIR
RUN git submodule update --init --recursive
RUN make
RUN sudo make install
RUN sudo ldconfig

COPY . $APP_DIR
WORKDIR $APP_DIR

# Build projects
RUN gradle :proto:build
RUN gradle build

[gnz00]

@mkobit You might just need: python -m pip install grpcio-tools

[mkobit]

Thanks @gnz00 - if I install grpcio-tools (latest releases) I don't find grpc_python_plugin anywhere on the path or in any of the installed directories. It looks like there is an entire protoc-like executable that is built as part of the Python setup.py installation process. I'm just kind of figuring this out by looking at the gRPC source code, but I am still new to it so could be off on some of this. I'm wondering if some of the tooling is different now?

To give a summary of what I'm triyng to do - I'm attempting to generate Python server and client gRPC code as well as Java client code by using this Gradle plugin. I'm not having much luck doing them both, so I might just have to shell out to the Python gRPC tooling with python -m grpc_tools.protoc.

[ejona86]

It appears grpcio-tools uses an .so version of the plugin, not a normal binary.

CC @nathanielmanistaatgoogle

[mehrdada]

@mkobit As @ejona86 pointed out the Python grpcio-tools package compiles protoc and the plugin into a single Python extension, unlike most other language packages. This has some advantages like the whole thing being installable via pip install even for esoteric architectures. We still do have the Python plugin available if you need it, but you'll need to manually checkout the gRPC source tree and run make plugins -j12 to build it.

[mkobit]

Thanks @ejona86 and @mehrdada for confirming what I suspected. I would like to see the plugin be its own distribution (or be a part of the grpcio-tools package), I can move forward with the grpc_tools.protoc approach.

Thanks everybody for fast responses!

[pbarker]

would be nice if this played well with things like prototool, its difficult to generate stubs for a variety of languages with these off cases

[doctorpangloss]

@mkobit You can try this (haven't run it in awhile):

project(":proto") {
    apply plugin: "java"
    apply plugin: "com.google.protobuf"

    buildDir = "$rootProject.buildDir/proto"

    protobuf {
        protoc {
            artifact = "com.google.protobuf:protoc:${protoVersion}"
        }
        plugins {
            grpc_java {
                artifact = "io.grpc:protoc-gen-grpc-java:${grpcVersion}"
            }
            grpc_python {
                path = getPluginPath("python")
            }
            grpc_ruby {
                path = getPluginPath("ruby")
            }
            grpc_cpp {
                path = getPluginPath("cpp")
            }
        }
        generateProtoTasks {
            all()*.builtins {
                java {}
                python {}
                ruby {}
                cpp {}
            }
            all()*.plugins {
                grpc_java {
                    outputSubDir = "java"
                }
                grpc_python {
                    outputSubDir = "python"
                }
                grpc_ruby {
                    outputSubDir = "ruby"
                }
                grpc_cpp {
                    outputSubDir = "cpp"
                }
            }
        }

        generatedFilesBaseDir = "$buildDir/generated/src"
    }

    dependencies {
        compile "com.google.protobuf:protobuf-java:${protoVersion}"
        compile "io.grpc:grpc-protobuf:${grpcVersion}"
        compile "io.grpc:grpc-netty:${grpcVersion}"
        compile "io.grpc:grpc-stub:${grpcVersion}"
    }
}

static def getPluginPath(name) {
    def path = "which grpc_${name}_plugin".execute()
    path.waitFor()
    path = path.in.text.trim()
    return path
}

For people using this solution and would like something that doesn't require installing dependencies, only a valid docker environment, you can use something along the lines of

protobuf {
    protoc {
        artifact = 'com.google.protobuf:protoc:3.13.0'
    }
    plugins {
        grpc_java {
            ...
        }
        grpc_csharp {
            path = project.projectDir.absolutePath + "/bin/grpc_csharp_plugin"
        }
    }
    generateProtoTasks {
        all()*.builtins {
            java {}
            csharp {}
        }
        all()*.plugins {
            grpc_java {
                outputSubDir = "java"
            }
            grpc_csharp {
                outputSubDir = "csharp_grpc"
            }
        }
    }
}

where an executable file is placed in your project at ./bin/grpc_csharp_plugin for example:

#!/usr/bin/env bash
docker run -v "$(pwd)":/defs -w "/defs" --rm --entrypoint /usr/bin/grpc_csharp_plugin namely/protoc-all "$@"

I am unsure if Docker for Desktop on Windows will (1) add docker to your path and (2) correctly expand pwd. If you need Windows and modify the command / author a valid .ps1 that is treated as executable, please post it here.