registry upload protos

The registry tool includes an opinionated uploader that imports Protocol Buffer API descriptions into the registry. It is designed to import API descriptions from github.com/googleapis/googleapis but will work with other API descriptions stored in a similar structure.

The uploader makes these assumptions:

• All proto files are under a common top-level directory. For Google APIs, this is the root of the https://github.com/googleapis/googleapis repo.

• Each API spec is in a subdirectory of this top-level directory that is named with the corresponding API version. Version names begin with a lower-case v which is typically followed by a small integer. Optionally the words alpha or beta might appear between v and the integer. As an example, the description of the first version of the Apigee Registry API is in google/cloud/apigeeregistry/v1.

• Each API spec consists of protos and an additional YAML file that contains metadata in the google.api.Service format, also known as “Service Configuration”. This includes the name of the API, which is not directly represented in the Protocol Buffer source files. The service configuration for the Apigee Registry API is in apigeeregistry_v1.yaml and begins as follows:

  type: google.api.Service
  config_version: 3
  name: apigeeregistry.googleapis.com
  title: Apigee Registry API
  

  For the id of the API in the registry, the .googleapis.com suffix is removed.

• The protoc compiler can be used to compile the protos in the API spec directory. It is invoked from the common top-level directory and might import files from other directories under that common top-level.

For each API, the registry tool compiles the protos in the API spec directory, reads a list of dependencies from the compilation output (a FileDescriptorSet), and adds all of these files to a zip archive along with any YAML or JSON metadata files in the API spec directory. This zip archive is the API spec that is uploaded to the registry. It is “fully resolved” in that any client can download this zip archive, unpack it, and rerun protoc to compile the protos for use in any downstream tooling.

The upload command is as follows:

registry upload protos $DIRECTORY --project-id $PROJECTID

Here $DIRECTORY is the top-level directory of proto files. This is a "bulk" uploader; it will upload every API spec that it finds meeting the above criteria. With the optional --base-uri argument, we can specify the root of a path to be stored in each spec's source_uri field. Here we include the commit id from the cloned Git repository:

# Get the commit hash of the checked-out protos directory
export COMMIT=`(cd $DIRECTORY; git rev-parse HEAD)`
registry upload protos $DIRECTORY \
	--project-id $PROJECT \
	--base-uri https://github.com/googleapis/googleapis/blob/$COMMIT 

With these protos stored in the registry, we can run linters and other tools by reading API specs from the registry, analyzing them, and putting the results back in the registry. Since each API spec is independent and fully-resolved, these tools can run concurrently and in a distributed way.