Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to generate gRPC descriptor set #36897

Closed
edeandrea opened this issue Nov 6, 2023 · 13 comments · Fixed by #36930
Closed

Unable to generate gRPC descriptor set #36897

edeandrea opened this issue Nov 6, 2023 · 13 comments · Fixed by #36930
Labels
area/grpc gRPC area/kubernetes kind/enhancement New feature or request
Milestone
3.6.0.CR1

Comments

@edeandrea
Copy link
Contributor

Describe the bug

I would like to be able to generate the Protobuf descriptor set alongside the source code. Generally this is done as an additional flag to the protoc compiler process.

This can be done in Maven by configuring the protobuf-maven-plugin (see the Output Of Binary File Descriptor Sets at the bottom of that link).

I'm having a hard time being able to do that in Quarkus. I've followed the steps in https://quarkus.io/version/main/guides/grpc-generation-reference#skipping-code-generation but it seems to clash with Quarkus' code generation.

When I disable Quarkus' code generation by doing either -Dgrpc.codegen.skip=true or setting quarkus.grpc.codegen.skip=true in application.properties, the application no longer compiles. It seems the Mutiny classes generated by Quarkus are no longer generated and therefore my application no longer compiles.

Expected behavior

No response

Actual behavior

No response

How to Reproduce?

  1. Clone https://github.com/quarkusio/quarkus-super-heroes
  2. cd into quarkus-super-heroes/rest-fights
  3. Open pom.xml
  4. Replace it's contents with this:
<?xml version="1.0"?>
<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <modelVersion>4.0.0</modelVersion>
  <groupId>io.quarkus.sample.super-heroes</groupId>
  <artifactId>rest-fights</artifactId>
  <version>1.0</version>
  <name>Quarkus Sample :: Super-Heroes :: Fights Microservice</name>
  <properties>
    <assertj.version>3.24.2</assertj.version>
    <compiler-plugin.version>3.11.0</compiler-plugin.version>
    <grpc.version>1.57.2</grpc.version>
    <grpcmock.version>0.10.1</grpcmock.version>
    <mapstruct.version>1.5.5.Final</mapstruct.version>
    <maven.compiler.parameters>true</maven.compiler.parameters>
    <maven.compiler.release>17</maven.compiler.release>
    <microcks.version>0.1.3</microcks.version>
    <os-maven-plugin.version>1.7.1</os-maven-plugin.version>
    <pact.version>1.1.0</pact.version>
    <pact.writer.overwrite>true</pact.writer.overwrite>
    <protobuf-maven-plugin.version>0.6.1</protobuf-maven-plugin.version>
    <protoc.version>3.22.0</protoc.version>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
    <quarkus.platform.artifact-id>quarkus-bom</quarkus.platform.artifact-id>
    <quarkus.platform.group-id>io.quarkus.platform</quarkus.platform.group-id>
    <quarkus.platform.version>3.5.0</quarkus.platform.version>
    <surefire-plugin.version>3.2.1</surefire-plugin.version>
    <wiremock.version>3.3.1</wiremock.version>
  </properties>
  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>${quarkus.platform.group-id}</groupId>
        <artifactId>${quarkus.platform.artifact-id}</artifactId>
        <version>${quarkus.platform.version}</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-hibernate-validator</artifactId>
    </dependency>
    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-smallrye-openapi</artifactId>
    </dependency>
    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-smallrye-reactive-messaging-kafka</artifactId>
    </dependency>
    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-resteasy-reactive-jackson</artifactId>
    </dependency>
    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-arc</artifactId>
    </dependency>
    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-rest-client-reactive-jackson</artifactId>
    </dependency>
    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-smallrye-fault-tolerance</artifactId>
    </dependency>
    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-smallrye-health</artifactId>
    </dependency>
    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-micrometer-registry-prometheus</artifactId>
    </dependency>
    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-jacoco</artifactId>
    </dependency>
    <dependency>
      <groupId>org.mapstruct</groupId>
      <artifactId>mapstruct</artifactId>
      <version>${mapstruct.version}</version>
    </dependency>
    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-container-image-docker</artifactId>
    </dependency>
    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-openshift</artifactId>
    </dependency>
    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-kubernetes</artifactId>
    </dependency>
    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-minikube</artifactId>
    </dependency>
    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-apicurio-registry-avro</artifactId>
    </dependency>
    <dependency>
      <groupId>io.strimzi</groupId>
      <artifactId>kafka-oauth-client</artifactId>
    </dependency>
    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-mongodb-panache</artifactId>
    </dependency>
    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-liquibase-mongodb</artifactId>
    </dependency>
    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-smallrye-stork</artifactId>
    </dependency>
    <dependency>
      <groupId>io.smallrye.stork</groupId>
      <artifactId>stork-service-discovery-static-list</artifactId>
    </dependency>
    <dependency>
      <groupId>io.smallrye.stork</groupId>
      <artifactId>stork-service-discovery-kubernetes</artifactId>
    </dependency>
    <dependency>
      <groupId>io.smallrye.stork</groupId>
      <artifactId>stork-service-discovery-eureka</artifactId>
    </dependency>
    <dependency>
      <groupId>io.smallrye.stork</groupId>
      <artifactId>stork-service-discovery-consul</artifactId>
    </dependency>
    <dependency>
      <groupId>io.smallrye.stork</groupId>
      <artifactId>stork-service-discovery-knative</artifactId>
    </dependency>
    <dependency>
      <groupId>io.smallrye.stork</groupId>
      <artifactId>stork-load-balancer-random</artifactId>
    </dependency>
    <dependency>
      <groupId>io.smallrye.stork</groupId>
      <artifactId>stork-load-balancer-least-requests</artifactId>
    </dependency>
    <dependency>
      <groupId>io.smallrye.stork</groupId>
      <artifactId>stork-load-balancer-least-response-time</artifactId>
    </dependency>
    <dependency>
      <groupId>io.smallrye.stork</groupId>
      <artifactId>stork-load-balancer-power-of-two-choices</artifactId>
    </dependency>
    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-kubernetes-client</artifactId>
    </dependency>
    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-opentelemetry</artifactId>
    </dependency>
    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-info</artifactId>
    </dependency>
    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-grpc</artifactId>
    </dependency>
    <dependency>
      <groupId>io.github.microcks.quarkus</groupId>
      <artifactId>quarkus-microcks</artifactId>
      <version>${microcks.version}</version>
      <scope>provided</scope>
    </dependency>
    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-junit5</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-panache-mock</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>io.rest-assured</groupId>
      <artifactId>rest-assured</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.assertj</groupId>
      <artifactId>assertj-core</artifactId>
      <version>${assertj.version}</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.wiremock</groupId>
      <artifactId>wiremock</artifactId>
      <version>${wiremock.version}</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.grpcmock</groupId>
      <artifactId>grpcmock-junit5</artifactId>
      <version>${grpcmock.version}</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>io.smallrye.reactive</groupId>
      <artifactId>smallrye-reactive-messaging-in-memory</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-test-kafka-companion</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>io.quarkiverse.pact</groupId>
      <artifactId>quarkus-pact-consumer</artifactId>
      <version>${pact.version}</version>
      <scope>provided</scope>
    </dependency>
    <dependency>
      <groupId>io.quarkiverse.pact</groupId>
      <artifactId>quarkus-pact-provider</artifactId>
      <version>${pact.version}</version>
      <scope>provided</scope> <!-- See https://github.com/quarkiverse/quarkus-pact/issues/28 -->
    </dependency>
  </dependencies>
  <build>
    <extensions>
      <extension>
        <groupId>kr.motd.maven</groupId>
        <artifactId>os-maven-plugin</artifactId>
        <version>${os-maven-plugin.version}</version>
      </extension>
    </extensions>
    <plugins>
      <plugin>
        <groupId>${quarkus.platform.group-id}</groupId>
        <artifactId>quarkus-maven-plugin</artifactId>
        <version>${quarkus.platform.version}</version>
        <extensions>true</extensions>
        <executions>
          <execution>
            <goals>
              <goal>build</goal>
              <goal>generate-code</goal>
              <goal>generate-code-tests</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>${compiler-plugin.version}</version>
        <configuration>
          <parameters>${maven.compiler.parameters}</parameters>
          <annotationProcessorPaths>
            <path>
              <groupId>org.mapstruct</groupId>
              <artifactId>mapstruct-processor</artifactId>
              <version>${mapstruct.version}</version>
            </path>
            <path>
              <groupId>io.quarkus</groupId>
              <artifactId>quarkus-panache-common</artifactId>
              <version>${quarkus.platform.version}</version>
            </path>
          </annotationProcessorPaths>
        </configuration>
      </plugin>
      <plugin>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>${surefire-plugin.version}</version>
        <configuration>
          <systemPropertyVariables>
            <java.util.logging.manager>org.jboss.logmanager.LogManager</java.util.logging.manager>
            <maven.home>${maven.home}</maven.home>
            <pact_do_not_track>true</pact_do_not_track>
            <pact.writer.overwrite>true</pact.writer.overwrite>
          </systemPropertyVariables>
        </configuration>
      </plugin>
      <plugin>
        <artifactId>maven-failsafe-plugin</artifactId>
        <version>${surefire-plugin.version}</version>
        <executions>
          <execution>
            <goals>
              <goal>integration-test</goal>
              <goal>verify</goal>
            </goals>
            <configuration>
              <systemPropertyVariables>
                <native.image.path>${project.build.directory}/${project.build.finalName}-runner</native.image.path>
                <java.util.logging.manager>org.jboss.logmanager.LogManager</java.util.logging.manager>
                <maven.home>${maven.home}</maven.home>
              </systemPropertyVariables>
            </configuration>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <groupId>org.xolstice.maven.plugins</groupId>
        <artifactId>protobuf-maven-plugin</artifactId>
        <version>${protobuf-maven-plugin.version}</version>
        <configuration>
          <writeDescriptorSet>true</writeDescriptorSet>
          <descriptorSetFileName>locationservice-v1.dsc</descriptorSetFileName>
          <descriptorSetOutputDirectory>src/test/resources/wiremock/grpc</descriptorSetOutputDirectory>
          <protocArtifact>com.google.protobuf:protoc:${protoc.version}:exe:${os.detected.classifier}</protocArtifact>
          <pluginId>grpc-java</pluginId>
          <pluginArtifact>io.grpc:protoc-gen-grpc-java:${grpc.version}:exe:${os.detected.classifier}</pluginArtifact>
          <protocPlugins>
            <protocPlugin>
              <id>quarkus-grpc-protoc-plugin</id>
              <groupId>io.quarkus</groupId>
              <artifactId>quarkus-grpc-protoc-plugin</artifactId>
              <version>${quarkus.platform.version}</version>
              <mainClass>io.quarkus.grpc.protoc.plugin.MutinyGrpcGenerator</mainClass>
            </protocPlugin>
          </protocPlugins>
        </configuration>
        <executions>
          <execution>
            <id>compile</id>
            <goals>
              <goal>compile</goal>
            </goals>
          </execution>
          <execution>
            <id>test-compile</id>
            <goals>
              <goal>test-compile</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
  <profiles>
    <profile>
      <id>native</id>
      <activation>
        <property>
          <name>native</name>
        </property>
      </activation>
      <properties>
        <quarkus.package.type>native</quarkus.package.type>
      </properties>
    </profile>
  </profiles>
</project>

Output of uname -a or ver

No response

Output of java -version

No response

Quarkus version or git rev

No response

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response
@edeandrea edeandrea added the kind/bug Something isn't working label Nov 6, 2023
@quarkus-bot
Copy link

quarkus-bot bot commented Nov 6, 2023

/cc @alesj (grpc), @cescoffier (grpc), @geoand (knative,kubernetes,minikube,openshift), @iocanel (knative,kubernetes,openshift)

@edeandrea
Copy link
Contributor Author

@cescoffier and I discussed this on https://quarkusio.zulipchat.com/#narrow/stream/187030-users/topic/gRPC.20generate.20descriptor.20set.20at.20build.20time.3F as well.

@edeandrea
Copy link
Contributor Author

Maybe a good enhancement would be to add the ability to generate the descriptor set (& customize its name/location) when running protoc in the Quarkus gRPC code generator?

Essentially this configuration:

<writeDescriptorSet>true</writeDescriptorSet>
<descriptorSetFileName>locationservice-v1.dsc</descriptorSetFileName>
<descriptorSetOutputDirectory>src/test/resources/wiremock/grpc</descriptorSetOutputDirectory>

@alesj
Copy link
Contributor

alesj commented Nov 6, 2023

Why do you need this?

@edeandrea
Copy link
Contributor Author

For now, because WireMock requires it in order to mock gRPC endpoints.

@edeandrea
Copy link
Contributor Author

https://wiremock-community.slack.com/archives/C03N1E6HFPY/p1699047977297779?thread_ts=1699025967.381439&cid=C03N1E6HFPY

@cescoffier
Copy link
Member

Seems easy enough to add. The target directory should be configurable, but not the name as you may have several protos. For the name, we should use the proto name.

@edeandrea
Copy link
Contributor Author

If you both don't mind, maybe I'll take a stab at it this week.

@edeandrea
Copy link
Contributor Author

edeandrea commented Nov 7, 2023 •
edited

Also @cescoffier

The target directory should be configurable, but not the name as you may have several protos. For the name, we should use the proto name.

There is ever only 1 descriptor set file that describes all the .proto files that were part of the generation, so therefore the name should be configurable. See https://docs.streamsets.com/portal/platform-datacollector/latest/datacollector/UserGuide/Data_Formats/Protobuf-Prerequisites.html

@cescoffier
Copy link
Member

Go ahead @edeandrea

@edeandrea
Copy link
Contributor Author

@cescoffier Within GrpcCodeGen how do I get a reference to the project's root directory? I would think that a user might want to configure the output dir for the descriptor set to be somewhere relative to the project root dir (like src/test/resources).

When I try to reference CodeGenContext.applicationModel().getApplicationModule() it returns null, so I'm not sure how to go about getting a reference?

There's CodeGenContext.workDir(), which says "typically the main build directory of the project", but I'm not sure its ok to rely on CodeGenContext.workDir()/.. as a reference to the project's root dir?

@cescoffier
Copy link
Member

I don't believe we can access the module root dir. However protoc generally requires absolute paths, so might be a way to deal with the problem.

@edeandrea
Copy link
Contributor Author

I have a way to do it that i'm working on. I had to add something to both the maven & gradle plugins to pass it in.

I'll create a draft PR in a bit once i finish some tests and you can take a look.

@edeandrea edeandrea mentioned this issue Nov 7, 2023
Merged
@gsmet gsmet added kind/enhancement New feature or request and removed kind/bug Something isn't working labels Nov 10, 2023
@quarkus-bot quarkus-bot bot added this to the 3.6 - main milestone Nov 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/grpc gRPC area/kubernetes kind/enhancement New feature or request
Projects
None yet
Milestone
3.6.0.CR1
Development

Successfully merging a pull request may close this issue.

Ability to generate gRPC descriptor set edeandrea/quarkus
4 participants
@alesj @edeandrea @cescoffier @gsmet