New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: Expand OpenAPI to include RPC APIs #801

Open
timburks opened this Issue Sep 29, 2016 · 5 comments

Comments

Projects
None yet
5 participants
@timburks
Contributor

timburks commented Sep 29, 2016

The OpenAPI 2.0 Specification states that its goal is “to define a standard, language-agnostic interface to REST APIs”. REST is an extremely popular style for implementing APIs that run over HTTP and related protocols (HTTPS, HTTP/2, QUIC). But other transport mechanisms are common and APIs are often defined in ways that correspond to procedure calls. In some ways, procedure calls are more fundamental, as REST APIs are often implemented by systems that convert REST requests into procedure calls.

At times it is desirable to use remote procedure calls (RPCs) as the primary form of an API. RPC systems allow expression of semantics that go outside the range of the HTTP verbs used by REST. RPC systems map naturally to the function calls that are used to implement most software systems, and RPC provides a design pattern that can be use to guide system implementations. The RPC abstraction is also much older than REST, and new systems such as gRPC and Thrift show its enduring usefulness.

With this in mind, we think it would be beneficial to expand the scope of the OpenAPI specification to include remote procedure call semantics. Here we propose an expanded specification that builds on our experience using RPCs to connect very large distributed systems while being general enough to apply to a broad range of variations.

Background

Protocol Buffers

We give special consideration to the Protocol Buffer message representation. At Google, Protocol Buffers are used to send and receive tens of billions of messages each second. Protocol Buffers are also used for messages in the open-source gRPC framework and other more specialized RPC frameworks [1] [2] [3].

Protocol Buffers are typically represented in a binary form, and the latest version (proto3) also allows mapping to representations in JSON and YAML formats. Messages can be generally treated as collections of typed fields, just as they are in other serialization formats including Thrift, Avro, and Cap’n Proto. Thus message description can be treated similarly for all of these representations, including the JSON and YAML that OpenAPI uses to describe request and response objects.

Protocol Buffer interfaces are defined using a special description language (.proto) that describes messages and the services that use them. This language is different from the JSON and YAML used for OpenAPI specifications. To keep the specification and tooling simple, this proposal replaces this language with extensions to OpenAPI that carry the same meanings. It seeks to express as much as possible of the .proto language within OpenAPI so that if desired, .proto files can be converted into OpenAPI RPC descriptions and vice versa.

RPC Elements

Messages

Remote procedure calls are described in terms of the messages that they send and receive. Messages contain fields that have names, types, and other attributes, often including an integer field number that is used in message serialization.

Services

In RPC terminology, "Services" are collections of remote procedure calls that send and receive messages. Each call has a single input and output message, and some RPC implementations will allow input and output messages to be streamed in a single call. Thus each remote procedure call is described by its name, input type, output type, and whether or not the input and output messages are streamed.

Example

Here we show an example .proto description of an API. For illustration, we've specified the GetBook RPC as a streaming call that accepts a stream of book requests and returns a stream of books. This is indicated with the stream keyword that appears before the request and response types.

syntax = "proto3";

package examples.bookstore;

import "google/protobuf/empty.proto";
import "google/protobuf/struct.proto";

service Bookstore {
  rpc ListShelves(google.protobuf.Empty) returns (ListShelvesResponse) {}
  rpc CreateShelf(CreateShelfRequest) returns (Shelf) {}
  rpc GetShelf(GetShelfRequest) returns (Shelf) {}
  rpc DeleteShelf(DeleteShelfRequest) returns (google.protobuf.Value) {}
  rpc ListBooks(ListBooksRequest) returns (ListBooksResponse) {}
  rpc CreateBook(CreateBookRequest) returns (Book) {}
  rpc GetBook(stream GetBookRequest) returns (stream Book) {}
  rpc DeleteBook(DeleteBookRequest) returns (google.protobuf.Value) {}
}

message Shelf {
  string name = 2;
  string theme = 3;
}

message Book {
  string author = 2;
  string name = 3;
  string title = 4;
}

message ListShelvesResponse {
  repeated Shelf shelves = 1;
}

message CreateShelfRequest {
  Shelf shelf = 1;
}

message GetShelfRequest {
  int64 shelf = 1;
}

message DeleteShelfRequest {
  int64 shelf = 1;
}

message ListBooksRequest {
  int64 shelf = 1;
}

message ListBooksResponse {
  repeated Book books = 1;
}

message CreateBookRequest {
  int64 shelf = 1;
  Book book = 2;
}

message GetBookRequest {
  int64 shelf = 1;
  int64 book = 2;
}

message DeleteBookRequest {
  int64 shelf = 1;
  int64 book = 2;
}

Open API Representation

Messages

RPC messages are described in the OpenAPI definitions section. To these we add a few new fields to provide protocol buffer semantics. New fields are prefixed x- but in the accepted proposal, we would expect all x- prefixes to be deleted.

Properties of a message correspond to fields in a protocol buffer message. x-field-number (or fieldNumber) is a required integer property that associates a unique field number with each property. x-repeated is an optional boolean property (by default false) that, when true, indicates that a field may occur more than once.

Here we show an example OpenAPI representation of the messages in the Bookstore API.

 definitions:
  "Shelf":
    type: object
    properties:
      name: 
        type: string
        x-field-number: 2
      theme:
        type: string
        x-field-number: 3
  "Book":
    type: object
    properties:
      author: 
        type: string
        x-field-number: 2
      name:
        type: string
        x-field-number: 3
      title:
        type: string
        x-field-number: 4
  "ListShelvesResponse":
    type: object
    properties:
      shelves:
        allOf:
          - $ref: "#/definitions/Shelf"
          - x-repeated: true
          - x-field-number: 1
  "CreateShelfRequest":
    type: object
    properties:
      shelf:
        allOf:
          - $ref: "#/definitions/Shelf"
          - x-field-number: 1
  "GetShelfRequest":
    type: object
    properties:
      shelf:
        type: integer
        format: int64
        x-field-number: 1
  "DeleteShelfRequest":
    type: object
    properties:
      shelf:
        type: integer
        format: int64
        x-field-number: 1
  "ListBooksRequest":
    type: object
    properties:
      shelf:
        type: integer
        format: int64
        x-field-number: 1
  "ListBooksResponse":
    type: object
    properties:
      books:
        allOf:
          - $ref: "#/definitions/Book"
          - x-repeated: true
          - x-field-number: 1
  "CreateBookRequest":
    type: object
    properties:
      shelf:
        type: integer
        format: int64
        x-field-number: 1
      book:
        allOf:
          - $ref: "#/definitions/Book"
          - x-field-number: 2
  "GetBookRequest": 
    type: object
    properties:
      shelf:
        type: integer
        format: int64
        x-field-number: 1
      book:
        type: integer
        format: int64
        x-field-number: 2
  "DeleteBookRequest": 
    type: object
    properties:
      shelf:
        type: integer
        format: int64
        x-field-number: 1
      book:
        type: integer
        format: int64
        x-field-number: 2

Services

Services are a distinct new entity that we represent using the x-services key at the top level of the OpenAPI description. This allows an API to include both RPC and REST representations side-by-side.

Services are described by their name and the procedures they contain. Procedures are described by the objects they accept and return. Streamed values are indicated with x-streaming, an optional boolean property (by default false).

Here is an example OpenAPI representation of the services in the Bookstore API (in this case, a single service named "Bookstore").

x-services:
  "Bookstore":
    x-procedures:
      "ListShelves":
        x-accepts: 
          $ref: "google-protobuf.yaml#/Empty"
        x-returns:
          $ref: "#/definitions/ListShelvesResponse"
      "CreateShelf":
        x-accepts:
          $ref: "#/definitions/CreateShelfRequest"
        x-returns:
          $ref: "#/definitions/Shelf"
      "GetShelf":
        x-accepts:
          $ref: "#/definitions/GetShelfRequest"
        x-returns:
          $ref: "#/definitions/Shelf"
      "DeleteShelf":
        x-accepts:
          $ref: "#/definitions/DeleteShelfRequest"
        x-returns:
          $ref: "google-protobuf.yaml/#Value"
      "ListBooks": 
        x-accepts:
          $ref: "#/definitions/ListBooksRequest"
        x-returns:
          $ref: "#/definitions/ListBooksResponse"
      "CreateBook": 
        x-accepts:
          $ref: "#/definitions/CreateBookRequest"
        x-returns:
          $ref: "#/definitions/Book"
      "GetBook": 
        x-accepts:
          allOf:
            - $ref: "#/definitions/GetBookRequest"
            - streaming: true
        x-returns:
          allOf:
            - $ref: "#/definitions/Book"
            - streaming: true
      "DeleteBook": 
        x-accepts:
          $ref: "#/definitions/DeleteBookRequest"
        x-returns:
          $ref: "#/definitions/Shelf"

Discussion

Our intent is to make it possible to write tools that convert back-and-forth between .proto and OpenAPI representations and to build RPC flows that completely replace .proto inputs with OpenAPI RPC specifications.

OpenAPI representations are more verbose than .proto, but are more amenable to automated processing and custom editors like swagger-editor. Our hope is that this representation will lead more editors and other tools to support RPC APIs.

Types in OpenAPI and Protocol Buffers

Protocol Buffers contain many finely-differentiated scalar types, while the OpenAPI spec uses general types such as number and integer. In OpenAPI, an additional format field supplements the type field with additional representation detail, so our proposal uses this field to include the full name of the corresponding Protocol Buffer type.

.proto field type OpenAPI type field value OpenAPI format field value
double number double
float number float
int64 integer int64
uint64 integer uint64
int32 integer int32
fixed64 integer fixed64
fixed32 integer fixed64
bool boolean -
string string -
group - -
message - -
bytes binary -
uint32 integer uint32
enum ? ?
sfixed32 integer sfixed32
sfixed64 integer sfixed64
sint32 integer sint32
sint64 integer sint64

In the above table, the bool, string, and bytes types directly correspond to OpenAPI types and need no additional detail in the format field. The group type in .proto is deprecated and unsupported, and the message type corresponds to inclusion of another message, which is represented in OpenAPI with the $ref property.

The enum type is represented in .proto as an integer and in OpenAPI as a string. This difference is unresolved in this proposal.

Gaps between OpenAPI and Protocol Buffers

There are some Protocol Buffer features that aren’t yet covered by this proposal. Gaps that are significant and unresolved may be addressed in future proposals.

Default values

Default field values could be represented with the existing OpenAPI default property, but in some message representations (such as proto3, default values are specified for each type and are not modifiable.

Enumerations

Enumerations are represented with strings in OpenAPI and integers in .proto. Resolution of this is a high priority future proposal.

Maps

The .proto format allows fields to have map types which include type specifications for map keys and values. When serialized, these maps are represented with special automatically-created messages.

Here we omit further discussion of maps, leaving it as a more general question about the OpenAPI Specification.

Extensions

Extensions are defined in proto2 and allow additional fields to be added to messages and for ranges of field numbers to be reserved for third-party extensions. Extensions are not supported in proto3 and are omitted from this proposal.

Nested messages

The .proto language allows messages to be defined inside other messages. Nested types can’t be written directly in OpenAPI, but we can define messages with hierarchical names similar to the ones that would be implied for nested .proto messages.

Options

In .proto, options are predefined annotations that can be added in various places in a .proto file. Common options configure code generators (by specifying package names or class name prefixes) or map RPC procedures to REST methods. Options are not addressed here but are a priority for inclusion in a future proposal.

Oneof

In .proto, the oneof statement groups fields to indicate that only one of the grouped fields can be included in a message. There is no corresponding concept in OpenAPI 2.0, but this appears to be addressed by a pending pull request.

Packages and API Versions

.proto descriptions optionally include a package name. This has a similar purpose as the basePath field in the OpenAPI root object and we suggest that either a new field named package or the existing basePath be used for this.

API Versions are commonly indicated by the last segment in a package name (segments are separated by periods). When the last segment looks like a version name (beginning with a 'v' and following with a possibly-dotted number), it is used as the version name. We omit this convention from this proposal and leave the representation of API versions to the existing version field of the info object.

@fehguy

This comment has been minimized.

Show comment
Hide comment
@fehguy

fehguy Sep 29, 2016

Contributor

RPC semantics is quite impossible for 3.0 but protobufs as payload may be doable and would certainly help out. Will look at your comments about the missing fields in our extended json schema for supporting protos.

Contributor

fehguy commented Sep 29, 2016

RPC semantics is quite impossible for 3.0 but protobufs as payload may be doable and would certainly help out. Will look at your comments about the missing fields in our extended json schema for supporting protos.

@cfineman

This comment has been minimized.

Show comment
Hide comment
@cfineman

cfineman Oct 6, 2016

@fehguy "quite impossible" because it goes beyond the standard CRUD REST verbs or is there some other reason it's untenable?

cfineman commented Oct 6, 2016

@fehguy "quite impossible" because it goes beyond the standard CRUD REST verbs or is there some other reason it's untenable?

@fehguy

This comment has been minimized.

Show comment
Hide comment
@fehguy

fehguy Oct 7, 2016

Contributor

@cfineman because we're closed on major features for 3.0.

Contributor

fehguy commented Oct 7, 2016

@cfineman because we're closed on major features for 3.0.

@cfineman

This comment has been minimized.

Show comment
Hide comment
@cfineman

cfineman Oct 7, 2016

acknowledged

cfineman commented Oct 7, 2016

acknowledged

@fmvilas

This comment has been minimized.

Show comment
Hide comment
@fmvilas

fmvilas May 24, 2017

It might be a good fit for the AsyncAPI spec, given it's message-driven instead of request/response.

fmvilas commented May 24, 2017

It might be a good fit for the AsyncAPI spec, given it's message-driven instead of request/response.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment