Skip to content

Latest commit

 

History

History
266 lines (191 loc) · 8.88 KB

README.md

File metadata and controls

266 lines (191 loc) · 8.88 KB
Raw

New client generation (GitHub Action)

This new generation workflow enables generation of new libraries by

  1. Appending a new library to our generation_config.yaml.
  2. Creating a PR with the changes.

The new client will be generated by Hermetic library generation workflow.

Components

new_library_hermetic_build/add-new-client-config.py

This script takes 10 arguments that map to items in the newly added library that goes in generation_config.yaml. A new entry will be added to libraries with the necessary generation configuration.

.github/workflows/generate_new_client_hermetic_build.yaml

This workflow orchestrates the add-new-client-config.py script and creates a pull request.

Execute the GitHub Action

In order to run the GitHub Action, you need to specify a few parameters. These parameters will be available in the Cloud Drop link (a YAML file) included in the Buganizer request. The example in this README uses AlloyDB's Cloud Drop file as an example.

⚠️ IMPORTANT: Not all the new-client.py arguments are available in the Github Action. Please refer to this section

API short name (api_shortname)

As a convenience for the subsequent commands, we need an identifier for the library, called api_shortname. This identifier will be used by default to generate the following:

  • --distribution-name
  • --library_name

The corresponding value in the Cloud Drop page is api_short_name.

Example: alloydb

Important

api_short_name is not always unique across client libraries. In the instance that the api_short_name is already in use by an existing client library, you will need to determine a unique name OR to pass a unique library_name. See Advanced Options.

Proto path (proto_path)

This is the path from the internal google3/third_party/googleapis/stable root to the directory that contains the proto definitions for a specific version. For example: google/datastore/v2. Root-level proto paths like google/datastore are not supported. Note that the internal google3/third_party/googleapis/stable directory is mirrored externally in https://github.com/googleapis/googleapis/blob/master/.

For example, if the Buganizer ticket includes:

Link to protos: http://...(omit).../google/cloud/alloydb/v1alpha/alloydb_v1alpha.yaml.

then the corresponding external mirrored proto is here: https://github.com/googleapis/googleapis/blob/master/google/cloud/alloydb/v1alpha/alloydb_v1alpha.yaml.

Therefore, the "proto path" value we supply to the command is google/cloud/alloydb/v1alpha.

We will publish a single module for a service that includes the specified version (in the example, v1alpha). Any future version must be manually added to the configuration yaml (google-cloud-java/generation_config.yaml)

More than one proto_path

If you need another proto_path in the library, you must manually add it to google-cloud-java/generation_config.yaml after generating the new client.

Name pretty (name_pretty)

The corresponding value in the Cloud Drop page is title.

Example: AlloyDB API

Product Docs (product_docs)

The corresponding value in the Cloud Drop page is documentation_uri. The value must starts with "https://".

Example: https://cloud.google.com/alloydb/docs

REST Docs (rest_docs)

The corresponding value in the Cloud Drop page is rest_reference_documentation_uri. The value must starts with "https://".

Example: https://cloud.google.com/alloydb/docs/reference/rest

If the value exists, add it as a flag to the python command below (see Advanced Options: --rest-docs="https://cloud.google.com/alloydb/docs/reference/rest" \

RPC Docs (rpc_docs)

The corresponding value in the Cloud Drop page is proto_reference_documentation_uri. The value must starts with "https://".

Example: https://cloud.google.com/speech-to-text/docs/reference/rpc

If the value exists, add it as a flag to the python command below (see Advanced Options: --rpc-docs="https://cloud.google.com/speech-to-text/docs/reference/rpc" \

API description (api_description)

The corresponding value in the Cloud Drop page is documentation.summary or documentation.overview. If both of those fields are missing, take the description from the product page above. Use the first sentence to keep it concise.

Example:

    AlloyDB for PostgreSQL is an open source-compatible database service that
    provides a powerful option for migrating, modernizing, or building
    commercial-grade applications.

Distribution Name (distribution_name)

This variable determines the Maven coordinates of the generated library. It defaults to com.google.cloud:google-cloud-{api_shortname}. This mainly affect the values in the generated pom.xml files.

Library Name (library_name)

This variable indicates the output folder of the library. For example, you can have two libraries with alloydb (AlloyDB and AlloyDB Connectors) as api_shortname. In order to avoid both libraries going to the default java-alloydb folder, we can override this behavior by specifying a value like alloydb-connectors so the AlloyDB Connectors goes to java-alloydb-connectors.

Prerequisites (for local environment)

This section is only needed for the first local run of this script.

Checkout google-cloud-java repository

$ git clone https://github.com/googleapis/google-cloud-java
$ git checkout main
$ git pull

Install pyenv

Install pyenv

curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer \
| bash

Append the following lines to $HOME/.bashrc.

export PYENV_ROOT="$HOME/.pyenv"
command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

Logout the shell and login again. You should be at the home directory.

Assuming you have the following folder structure:

~ (Home)
    -> IdeaProjects/
        -> google-cloud-java
    -> ...

You can run these next commands in the home directory (or IdeaProjects). Otherwise, run it at google-cloud-java's parent directory.

Confirm pyenv installation succeeded:

~$ pyenv
pyenv 2.3.4
Usage: pyenv <command> [<args>]

Some useful pyenv commands are:
   activate    Activate virtual environment
   commands    List all available pyenv commands
   deactivate   Deactivate virtual environment
...

Install Python 3.9 via pyenv

~$ pyenv install 3.9.13
Downloading Python-3.9.13.tar.xz...
-> https://www.python.org/ftp/python/3.9.13/Python-3.9.13.tar.xz
Installing Python-3.9.13...
WARNING: The Python sqlite3 extension was not compiled. Missing the SQLite3 lib?
Installed Python-3.9.13 to /usr/local/google/home/suztomo/.pyenv/versions/3.9.13

Install Python v3.9.13 locally

Run this command

$ pyenv local 3.9.13

Confirm python3.9 is installed:

$ which python3.9
/usr/local/google/home/suztomo/.pyenv/shims/python3.9

Install Python packages

At the root of google-cloud-java repository clone, run:

$ python3.9 -m pip install -r generation/new_client_hermetic_build/requirements.txt

Advanced Options

In case the steps above don't show you how to specify the desired options, you can run the add-new-client-config.py script in your local environment. The advanced options not shown in the section above cannot be specified in the GitHub Action, hence the need for a local run (refer to the "Prerequisites (for local environment)" section). For the explanation of the available parameters, run:

python3.9 generation/new_client_hermetic_build/add-new-client-config.py generate  --help

After you run the script, you will see that the generation_config.yaml file was modified (or the script exited because the library already existed).

Please create a PR explaining what commands you used and make sure the add-new-client-config.py arguments were listed).

Special case example: Google Maps

Sometimes, a library generation requires special handling for Maven coordinates or API ID, especially when the library is not specific to Google Cloud. The table below is the summary of the special cases:

API paths --api_shortname --distribution-name
google/shopping/* shopping-<API short name> com.google.shipping:google-shopping-<API short name>
google/maps/* maps-<API short name> com.google.maps:google-maps-<API short name>

where <API short name> is the value from Cloud Drop file.