Proposal: Expand OpenAPI to include RPC APIs

[timburks]

The OpenAPI 2.0 Specification states that its goal is “to define a standard, language-agnostic interface to REST APIs”. REST is an extremely popular style for implementing APIs that run over HTTP and related protocols (HTTPS, HTTP/2, QUIC). But other transport mechanisms are common and APIs are often defined in ways that correspond to procedure calls. In some ways, procedure calls are more fundamental, as REST APIs are often implemented by systems that convert REST requests into procedure calls.

At times it is desirable to use remote procedure calls (RPCs) as the primary form of an API. RPC systems allow expression of semantics that go outside the range of the HTTP verbs used by REST. RPC systems map naturally to the function calls that are used to implement most software systems, and RPC provides a design pattern that can be use to guide system implementations. The RPC abstraction is also much older than REST, and new systems such as gRPC and Thrift show its enduring usefulness.

With this in mind, we think it would be beneficial to expand the scope of the OpenAPI specification to include remote procedure call semantics. Here we propose an expanded specification that builds on our experience using RPCs to connect very large distributed systems while being general enough to apply to a broad range of variations.

Background

Protocol Buffers

We give special consideration to the Protocol Buffer message representation. At Google, Protocol Buffers are used to send and receive tens of billions of messages each second. Protocol Buffers are also used for messages in the open-source gRPC framework and other more specialized RPC frameworks [1] [2] [3].

Protocol Buffers are typically represented in a binary form, and the latest version (proto3) also allows mapping to representations in JSON and YAML formats. Messages can be generally treated as collections of typed fields, just as they are in other serialization formats including Thrift, Avro, and Cap’n Proto. Thus message description can be treated similarly for all of these representations, including the JSON and YAML that OpenAPI uses to describe request and response objects.

Protocol Buffer interfaces are defined using a special description language (.proto) that describes messages and the services that use them. This language is different from the JSON and YAML used for OpenAPI specifications. To keep the specification and tooling simple, this proposal replaces this language with extensions to OpenAPI that carry the same meanings. It seeks to express as much as possible of the .proto language within OpenAPI so that if desired, .proto files can be converted into OpenAPI RPC descriptions and vice versa.

RPC Elements

Messages

Remote procedure calls are described in terms of the messages that they send and receive. Messages contain fields that have names, types, and other attributes, often including an integer field number that is used in message serialization.

Services

In RPC terminology, "Services" are collections of remote procedure calls that send and receive messages. Each call has a single input and output message, and some RPC implementations will allow input and output messages to be streamed in a single call. Thus each remote procedure call is described by its name, input type, output type, and whether or not the input and output messages are streamed.

Example

Here we show an example .proto description of an API. For illustration, we've specified the GetBook RPC as a streaming call that accepts a stream of book requests and returns a stream of books. This is indicated with the stream keyword that appears before the request and response types.

syntax = "proto3";

package examples.bookstore;

import "google/protobuf/empty.proto";
import "google/protobuf/struct.proto";

service Bookstore {
  rpc ListShelves(google.protobuf.Empty) returns (ListShelvesResponse) {}
  rpc CreateShelf(CreateShelfRequest) returns (Shelf) {}
  rpc GetShelf(GetShelfRequest) returns (Shelf) {}
  rpc DeleteShelf(DeleteShelfRequest) returns (google.protobuf.Value) {}
  rpc ListBooks(ListBooksRequest) returns (ListBooksResponse) {}
  rpc CreateBook(CreateBookRequest) returns (Book) {}
  rpc GetBook(stream GetBookRequest) returns (stream Book) {}
  rpc DeleteBook(DeleteBookRequest) returns (google.protobuf.Value) {}
}

message Shelf {
  string name = 2;
  string theme = 3;
}

message Book {
  string author = 2;
  string name = 3;
  string title = 4;
}

message ListShelvesResponse {
  repeated Shelf shelves = 1;
}

message CreateShelfRequest {
  Shelf shelf = 1;
}

message GetShelfRequest {
  int64 shelf = 1;
}

message DeleteShelfRequest {
  int64 shelf = 1;
}

message ListBooksRequest {
  int64 shelf = 1;
}

message ListBooksResponse {
  repeated Book books = 1;
}

message CreateBookRequest {
  int64 shelf = 1;
  Book book = 2;
}

message GetBookRequest {
  int64 shelf = 1;
  int64 book = 2;
}

message DeleteBookRequest {
  int64 shelf = 1;
  int64 book = 2;
}

Open API Representation

Messages

RPC messages are described in the OpenAPI definitions section. To these we add a few new fields to provide protocol buffer semantics. New fields are prefixed x- but in the accepted proposal, we would expect all x- prefixes to be deleted.

Properties of a message correspond to fields in a protocol buffer message. x-field-number (or fieldNumber) is a required integer property that associates a unique field number with each property. x-repeated is an optional boolean property (by default false) that, when true, indicates that a field may occur more than once.

Here we show an example OpenAPI representation of the messages in the Bookstore API.

 definitions:
  "Shelf":
    type: object
    properties:
      name: 
        type: string
        x-field-number: 2
      theme:
        type: string
        x-field-number: 3
  "Book":
    type: object
    properties:
      author: 
        type: string
        x-field-number: 2
      name:
        type: string
        x-field-number: 3
      title:
        type: string
        x-field-number: 4
  "ListShelvesResponse":
    type: object
    properties:
      shelves:
        allOf:
          - $ref: "#/definitions/Shelf"
          - x-repeated: true
          - x-field-number: 1
  "CreateShelfRequest":
    type: object
    properties:
      shelf:
        allOf:
          - $ref: "#/definitions/Shelf"
          - x-field-number: 1
  "GetShelfRequest":
    type: object
    properties:
      shelf:
        type: integer
        format: int64
        x-field-number: 1
  "DeleteShelfRequest":
    type: object
    properties:
      shelf:
        type: integer
        format: int64
        x-field-number: 1
  "ListBooksRequest":
    type: object
    properties:
      shelf:
        type: integer
        format: int64
        x-field-number: 1
  "ListBooksResponse":
    type: object
    properties:
      books:
        allOf:
          - $ref: "#/definitions/Book"
          - x-repeated: true
          - x-field-number: 1
  "CreateBookRequest":
    type: object
    properties:
      shelf:
        type: integer
        format: int64
        x-field-number: 1
      book:
        allOf:
          - $ref: "#/definitions/Book"
          - x-field-number: 2
  "GetBookRequest": 
    type: object
    properties:
      shelf:
        type: integer
        format: int64
        x-field-number: 1
      book:
        type: integer
        format: int64
        x-field-number: 2
  "DeleteBookRequest": 
    type: object
    properties:
      shelf:
        type: integer
        format: int64
        x-field-number: 1
      book:
        type: integer
        format: int64
        x-field-number: 2

Services

Services are a distinct new entity that we represent using the x-services key at the top level of the OpenAPI description. This allows an API to include both RPC and REST representations side-by-side.

Services are described by their name and the procedures they contain. Procedures are described by the objects they accept and return. Streamed values are indicated with x-streaming, an optional boolean property (by default false).

Here is an example OpenAPI representation of the services in the Bookstore API (in this case, a single service named "Bookstore").

x-services:
  "Bookstore":
    x-procedures:
      "ListShelves":
        x-accepts: 
          $ref: "google-protobuf.yaml#/Empty"
        x-returns:
          $ref: "#/definitions/ListShelvesResponse"
      "CreateShelf":
        x-accepts:
          $ref: "#/definitions/CreateShelfRequest"
        x-returns:
          $ref: "#/definitions/Shelf"
      "GetShelf":
        x-accepts:
          $ref: "#/definitions/GetShelfRequest"
        x-returns:
          $ref: "#/definitions/Shelf"
      "DeleteShelf":
        x-accepts:
          $ref: "#/definitions/DeleteShelfRequest"
        x-returns:
          $ref: "google-protobuf.yaml/#Value"
      "ListBooks": 
        x-accepts:
          $ref: "#/definitions/ListBooksRequest"
        x-returns:
          $ref: "#/definitions/ListBooksResponse"
      "CreateBook": 
        x-accepts:
          $ref: "#/definitions/CreateBookRequest"
        x-returns:
          $ref: "#/definitions/Book"
      "GetBook": 
        x-accepts:
          allOf:
            - $ref: "#/definitions/GetBookRequest"
            - streaming: true
        x-returns:
          allOf:
            - $ref: "#/definitions/Book"
            - streaming: true
      "DeleteBook": 
        x-accepts:
          $ref: "#/definitions/DeleteBookRequest"
        x-returns:
          $ref: "#/definitions/Shelf"

Discussion

Our intent is to make it possible to write tools that convert back-and-forth between .proto and OpenAPI representations and to build RPC flows that completely replace .proto inputs with OpenAPI RPC specifications.

OpenAPI representations are more verbose than .proto, but are more amenable to automated processing and custom editors like swagger-editor. Our hope is that this representation will lead more editors and other tools to support RPC APIs.

Types in OpenAPI and Protocol Buffers

Protocol Buffers contain many finely-differentiated scalar types, while the OpenAPI spec uses general types such as number and integer. In OpenAPI, an additional format field supplements the type field with additional representation detail, so our proposal uses this field to include the full name of the corresponding Protocol Buffer type.

.proto field type	OpenAPI type field value	OpenAPI format field value
double	number	double
float	number	float
int64	integer	int64
uint64	integer	uint64
int32	integer	int32
fixed64	integer	fixed64
fixed32	integer	fixed64
bool	boolean	-
string	string	-
group	-	-
message	-	-
bytes	binary	-
uint32	integer	uint32
enum	?	?
sfixed32	integer	sfixed32
sfixed64	integer	sfixed64
sint32	integer	sint32
sint64	integer	sint64

In the above table, the bool, string, and bytes types directly correspond to OpenAPI types and need no additional detail in the format field. The group type in .proto is deprecated and unsupported, and the message type corresponds to inclusion of another message, which is represented in OpenAPI with the $ref property.

The enum type is represented in .proto as an integer and in OpenAPI as a string. This difference is unresolved in this proposal.

Gaps between OpenAPI and Protocol Buffers

There are some Protocol Buffer features that aren’t yet covered by this proposal. Gaps that are significant and unresolved may be addressed in future proposals.

Default values

Default field values could be represented with the existing OpenAPI default property, but in some message representations (such as proto3, default values are specified for each type and are not modifiable.

Enumerations

Enumerations are represented with strings in OpenAPI and integers in .proto. Resolution of this is a high priority future proposal.

Maps

The .proto format allows fields to have map types which include type specifications for map keys and values. When serialized, these maps are represented with special automatically-created messages.

Here we omit further discussion of maps, leaving it as a more general question about the OpenAPI Specification.

Extensions

Extensions are defined in proto2 and allow additional fields to be added to messages and for ranges of field numbers to be reserved for third-party extensions. Extensions are not supported in proto3 and are omitted from this proposal.

Nested messages

The .proto language allows messages to be defined inside other messages. Nested types can’t be written directly in OpenAPI, but we can define messages with hierarchical names similar to the ones that would be implied for nested .proto messages.

Options

In .proto, options are predefined annotations that can be added in various places in a .proto file. Common options configure code generators (by specifying package names or class name prefixes) or map RPC procedures to REST methods. Options are not addressed here but are a priority for inclusion in a future proposal.

Oneof

In .proto, the oneof statement groups fields to indicate that only one of the grouped fields can be included in a message. There is no corresponding concept in OpenAPI 2.0, but this appears to be addressed by a pending pull request.

Packages and API Versions

.proto descriptions optionally include a package name. This has a similar purpose as the basePath field in the OpenAPI root object and we suggest that either a new field named package or the existing basePath be used for this.

API Versions are commonly indicated by the last segment in a package name (segments are separated by periods). When the last segment looks like a version name (beginning with a 'v' and following with a possibly-dotted number), it is used as the version name. We omit this convention from this proposal and leave the representation of API versions to the existing version field of the info object.

[fehguy]

RPC semantics is quite impossible for 3.0 but protobufs as payload may be doable and would certainly help out. Will look at your comments about the missing fields in our extended json schema for supporting protos.

[cfineman]

@fehguy "quite impossible" because it goes beyond the standard CRUD REST verbs or is there some other reason it's untenable?

[fehguy]

@cfineman because we're closed on major features for 3.0.

[cfineman]

acknowledged

[fmvilas]

It might be a good fit for the AsyncAPI spec, given it's message-driven instead of request/response.

[smyrman]

If we look at RPC APIs over JSON/YAML, I feel the "discriminator" feature in OpenAPI should allow describing the requests side of things pretty accurately as a single path item. However, there is lacking a way to associate the right requests with the right response when doing this.

E.g. if we look at JSON RPC 2.0 for instance, requests have two main formats.

• {"jsonrpc": "2.0", "id": 1, "method": "service.functionCall", "params: {...}} named parameters.
• {"jsonrpc": "2.0", "id": 1, "method": "service.functionCall", "params: [...]} positional parameters.

Where a large collection of different methods exist.

The responses in JSON RPC are of format:

• {"jsonrpc": "2.0", "id":"1", "result": ..} // On success
• {"jsonrpc": "2.0", "id":1, "error": ...} // On error

Where the schema of the result (and in principal error) are generally determined by the request method.

At least for the first variant, there is no problem describing this as a set of "<functionCall>Request" body definitions and have a single Open API path item for them (describing the HTTP protocol specific RPC endpoint) that uses a discriminator object to select between them. If we want to describe e.g. JSON batch requests as well (an array of the described request bodies), we could probably do it with an outer $anyOf.

The only missing feature for a correct implementation with this approach, is to be able to provide a similar discriminator object for the response type that instead of switching on a field in the response payload, switches on a field at a particular path in the request (A discriminator object for batch requests is probably not something we could pull of though).

I am not stating that this is the best way of documenting a RPC API.

But I think it's important that the scope should be to describe the HTTP Protocol for the RPC API, and not the RPC API in general (which could be protocol agnostic).

[philsturgeon]

Hey @timburks, the goal of OpenAPI Specification has swapped from REST to HTTP so that's one roadblock gone. #1946

I know protobuf support is wanted by @darrelmiller and it's come up several times at Stoplight. Maybe it's time to progress. 😎

[Cahl-Dee]

Hey team. I come from a web2 background where "REST is the way" and have always leaned on OpenAPI for defining our API specifications. I now work at a blockchain infrastructure provider where our APIs are mostly JSON-RPC with some REST here and there. I'd love to leverage OAS for our documentation and may even be able to contribute some development resources to the effort if we could come together and agree on a path forward.

[smyrman]

@Cahl-Dee, OpenAPI isn't particularly suitable for describing RPC APIs, and for JSON RPC 2.0 in particular, where all requests generally happen against the same endpoint, you are probably better off looking at https://open-rpc.org/ and associated tools.

[smyrman]

I mean all requests for JSON RPC 2.0 generally (or often) happen against the same URL / URL path. E.g. all method calls might be done against a single URL https://example.com/rpc. I.e. same endpoint from an HTTP (protocol) perspective.

[Cahl-Dee]

@Cahl-Dee, OpenAPI isn't particularly suitable for describing RPC APIs, and for JSON RPC 2.0 in particular, where all requests generally happen against the same endpoint, you are probably better off looking at https://open-rpc.org/ and associated tools.

@smyrman I'm not familiar with the inner workings of OAS so I have no clue how big of a change this would be, but are methods and endpoints really that different of concepts? Has there been, or is there openness to discussing how the primitive leveraged by OAS could be abstracted to cover both?

[Oct17th]

@smyrman Thats because they are sending 'method' in the passed data (which is unusual for RPC); look at how JSON-RPC passes data.

gRPC encodes the controller/method in the URI just like all other RPC. JSON-RPC seems to be odd in this respect.

ex. http:// localhost/controller/method ie http://localhost/user/getByName

The reason it is done in the URI is because request uri will ALWAYS be passed with redirects in your network and cannot be overwritten. The request variables can be removed/overwritten.

ALSO... would point out ticket stated 'RPC' so is not specific to any one RPC implementation so as long as you stick with a controller/method approach, this will work with JSON-RPC, gRPC and pretty much all RPC

+1 use http:// localhost/controller/method can mapped to a unique server implementation just like http endpoint.
btw i have another question, rpc methods do not have http verbs and http status, so we seems like need a new tag like paths and webhooks to describe the rpc methods, otherwise it seems to be odd when use paths to describe the rpc methods, it always use 'post' verb and always defined request in 'requestBody' and always defined response in '200' status

  /controller/method:
    post:
      requestBody:
        - name: name
          xxx
        - name: key
          xxx
      responses:
        '200':
          description: An array of rpc resp
          schema:
            type: array
            items:
              $ref: '#/definitions/RPCstruct'

[handrews]

OAS 4 Moonwalk will support synchronous HTTP RPC APIs under the big tent principle. Please join the discussions in that repository if you are still interested.

For asynchronous RPC, there is AsyncAPI.