[Need Help] How can I generate Swagger/OpenAPI v3 spec from .proto files and vice versa

[krishna-birla]

It seems that only binary protocol buffers are supported for interconversion between protobuf and swagger. How can I convert binary protocol buffers (.pb) to and from proto3 protocol buffers (.proto), is that even a sane question to ask.

Moreover, my final intent is to be able to interconvert between .proto and OpenAPI2/3 spec (.json/.yaml) files, either using annotations in the .proto file, some tags or otherwise.

Thanks in advance!

Edits:

Modifying the issue title a little to highlight the use case much better.

External links:

Stack Overflow | How to generate swagger3 (OpenAPI3) spec in (.json/.yaml) from protobuf (.proto) files?

[LorenzHW]

You can go from .json/.yaml to .proto with gnostic-grpc.

[krishna-birla]

Thanks for the quick response!!

This project is a very nice initiative, and I am super glad to see it incubate in GSOC! But there are a few issues that I think are fundamental given the description of the project is clear:

This tool converts an OpenAPI v3.0 API description into a description of a gRPC service that can be used to implement that API...

So

So a major feature added in OpenAPI3 over the previous version 2 was the ability to have multiple schemas for the same HTTP verb + path, using the oneOf, anyOf and allOf constructs. There are constructs in protobuf such as oneOf, using different combinations of these can simulate all OpenAPI3 constructs. The current version of the project does not support it.

The current conversion logic loses a lot of information from the spec (it is lossy), and hence converting back is neither possible nor supported (as of now). So generating servers and clients from these specs is not possible.

Moreover, some of my test examples were missing some tags from the annotations like body tag in results. A lot of swagger3 fields are today not supported. And above all, this is NOT an inter-conversion library, it does not convert to and from, where as the more practical use case is proto to swagger3 and not the other way around, since most implementations wrap an HTTP(S) server around gRPC server, and not the other way around.

But

But clearly this project is WIP, and what it already does is very helpful in some use cases. I would love to contribute to the project and be a part of it. This has good scope, it is trying to deal with a problem everyone has but no one talks about.

Meanwhile

Meanwhile, are there anymore "production" usable alternatives to solve my original problem?

[balnice]

@krishna-birla My team is also looking for the same and we've found 2 more alternatives for evaluation so far:

• openapi2proto only supports swagger/openapi spec 2.0
• openapi-generator only support protocol buffers version 3

Hope these help.

[krishna-birla]

Thanks @balnice
The second option looks promising, but the fundamental requirement of interconversion is still on hang. These libraries don't convert .proto to OpenAPI v3 spec, and only do it the other way around.

The most common and main use case is to wrap an HTTPS server around a gRPC server. This means that the developer writes the .proto files and the rest of the steps (i.e. generate gRPC server, generate gRPC client, generate OpenAPI v3 spec, generate HTTPS server, generate HTTPS client) should be automated.

The second repository you suggested definitely helps, but the question still remains open.

[shikag]

@krishna-birla Were you able to resolve this? I also have the same thing: Rest APIs in Java Spring Boot using Protobuf classes as arguments, and looking to generate Swagger UI.

Have you tried using protoc to generate .pb files from .proto and then using gnostic to generate OpenAPI V3 (something like reverse of this)

[krishna-birla]

@shikag It didn't work for me, I also thought that .pb is just intermediate code for protobuf, but it doesn't seem like it.

[shellsong]

@krishna-birla have you try this which can convert *.proto to swagger json schema

[krishna-birla]

@shellsong Yes, it supports OpenAPIv2, I am looking for v3.

[shikag]

Thanks!

[mpokryva]

You can run the generated OpenAPI v2 code through https://github.com/getkin/kin-openapi/tree/master/openapi2conv to get OpenAPI v3 code. This would work if you don't want to build something "cleaner," and you don't need any OpenAPI v3 features, just the syntax.

[krishna-birla]

I need the anyOf and oneOf features. That is the catch.

[mpokryva]

I see. One thing that should be easier to do would be to convert the v2 spec to v3 as I said above and then fork grpc-gateway and use gnostic's v3 proto spec to implement anyOf and oneOf annotations. See https://github.com/grpc-ecosystem/grpc-gateway/blob/master/protoc-gen-swagger/options/annotations.proto for the v2 equivalent. Not "clean," but it should work.

[timburks]

Hi @krishna-birla, do you have any end-to-end examples that illustrate your goals? Specifically I'm curious to see an OpenAPI spec that has examples of your usage of anyOf and oneOf, and even better if you have a corresponding .proto file - I'm guessing that since there's no automated conversion available, you might be manually mapping from proto to OpenAPI in handwritten code. Anyway, I think an example would be very helpful. Thanks!

[krishna-birla]

Hi @timburks
Thanks for the prompted reply.

A simple example could be something like this:
from

...
message CreateAnimalRequest {
  oneof Aminals {
    Cat cat = 1;
    Dog dog = 2;
    Lizard lizard = 3;
  }
}
...

to

...
requestBody:
        content:
          application/json:
            schema:
              oneOf:
              - $ref: '#/components/schemas/Cat'
              - $ref: '#/components/schemas/Dog'
              - $ref: '#/components/schemas/Lizard'
          application/yaml:
            schema:
              oneOf:
              - $ref: '#/components/schemas/Cat'
              - $ref: '#/components/schemas/Dog'
              - $ref: '#/components/schemas/Lizard'
...

The application/yaml and application/json parts can be taken from tags, so please ignore those.

Yes, I am forced to write a script of my own given the lack of a library, but I would prefer using one from openAPI or googleapis. I will be more than happy to contribute!

[timburks]

@krishna-birla One of the basic limitations currently is that the conversion from .proto (gRPC) to REST/JSON is done in a standard way that probably does not support oneofs in proto message bodies (I haven't tried to verify this but am not aware of this ever being done). So perhaps before we can usefully generate an OpenAPI v3 model, we should verify that an appropriate REST API is being created. Do you know if HTTP transcoding (described in AIP-127) will usefully transcode .protos like your sample?

[krishna-birla]

Sorry for the delayed response, I will investigate deeper and reply soon.

[krishna-birla]

@timburks Apologies for the delayed response.

The transcoding in https://google.aip.dev/127 may not be able to completely cater to the oneOf use case. Currently, annotations is a map, and takes HTTP spec of the GRPC. But it does not support accepting multiple types of bodies for the GRPC method. This gap prevents from using one of the key additions/features from OpenAPI v3.

For example, something expected as shown in:

rpc CreateBook(CreateBookRequest) returns (Book) {
  option (google.api.http) = {
    post: "/v1/books/*"
    body: "book"
  };
}

message CreateBookRequest {
  // The publisher who will publish this book.
  // When using HTTP/JSON, this field is automatically populated based
  // on the URI, because of the `{parent=publishers/*}` syntax.
  string parent = 1;

  // The type of the book, academic or fun time.
  // When using HTTP/JSON, this field is populated based on the HTTP body,
  // because of the `body: "book"` syntax.
  string bookType = 1;

  // The book to create.
  // When using HTTP/JSON, this field is populated based on the HTTP body,
  // because of the `body: "book"` syntax.
  // If body is an academic book, bookA is populated
  // If body is a fun time book, bookB is populated
  oneOf BookTypes {
    BookAcademic bookA = 2;
    BookFunTime bookF = 3;
  }

  // The user-specified ID for the book.
  // When using HTTP/JSON, this field is populated based on a query string
  // argument, such as `?book_id=foo`. This is the fallback for fields that
  // are not included in either the URI or the body.
  string book_id = 4;
}

[krishna-birla]

Also, similarly HTTP/YAML will also be supported for above AIPs, right?

[zwiedmann-isp]

Same desire, currently using protoc-gen-swagger since we don't currently have any oneOf types. There's an open issue for openapiv3 support in grpc-gateway (houses protoc-gen-swagger as well grpc-ecosystem/grpc-gateway#441)

@krishna-birla They need some help getting it done, but someone did make a start. Might be our best bet....

[krishna-birla]

@zwiedmann-isp
I am more than willing to contribute towards this effort.

[javidang]

@krishna-birla My team is also looking for the same and we've found 2 more alternatives for evaluation so far:

• openapi2proto only supports swagger/openapi spec 2.0
• openapi-generator only support protocol buffers version 3

Hope these help.

hi @balnice . were you able to use openapi-generator for openAPI v3 generation? we currently use similar grpc-bridge that takes care of swagger generation during runtime, it seems like openapi-generator is more of "do it yourself" style approach to the problem.

[balnice]

No idea what you mean by "do it yourself". If you need help with openapi-generator, simply open an issue in that repo.

[johnwayne19860314]

@krishna-birla My team is also looking for the same and we've found 2 more alternatives for evaluation so far:

• openapi2proto only supports swagger/openapi spec 2.0
• openapi-generator only support protocol buffers version 3

Hope these help.

hi @balnice . were you able to use openapi-generator for openAPI v3 generation? we currently use similar grpc-bridge that takes care of swagger generation during runtime, it seems like openapi-generator is more of "do it yourself" style approach to the problem.

could this grpc-bridge generate the openapiv3 and solve the problem with oneof?

[javidang]

@krishna-birla My team is also looking for the same and we've found 2 more alternatives for evaluation so far:

• openapi2proto only supports swagger/openapi spec 2.0
• openapi-generator only support protocol buffers version 3

Hope these help.

hi @balnice . were you able to use openapi-generator for openAPI v3 generation? we currently use similar grpc-bridge that takes care of swagger generation during runtime, it seems like openapi-generator is more of "do it yourself" style approach to the problem.

could this grpc-bridge generate the openapiv3 and solve the problem with oneof?

nope, that is why I was looking into openapi-generator. We ended up moving away from automatic swagger generation at the end.