Skip to content
Permalink
Browse files

feat: Add support for using static discovery documents (#1109)

* feat: Add support for static discovery documents

* Auto generated docs should use static artifacts
  • Loading branch information
parthea committed Jan 14, 2021
1 parent b7b9986 commit 32d1c597b364e2641eca33ccf6df802bb218eea1
Showing with 160 additions and 81 deletions.
  1. +2 −6 describe.py
  2. +9 −7 docs/start.md
  3. +30 −3 googleapiclient/discovery.py
  4. +28 −1 googleapiclient/discovery_cache/__init__.py
  5. +81 −54 tests/test_discovery.py
  6. +1 −1 tests/test_http.py
  7. +9 −9 tests/test_mocks.py
@@ -37,6 +37,7 @@
from googleapiclient.discovery import build
from googleapiclient.discovery import build_from_document
from googleapiclient.discovery import UnknownApiNameOrVersion
from googleapiclient.discovery_cache import get_static_doc
from googleapiclient.http import build_http
from googleapiclient.errors import HttpError

@@ -395,19 +396,14 @@ def document_api(name, version, uri):
"""
try:
service = build(name, version)
content = get_static_doc(name, version)
except UnknownApiNameOrVersion as e:
print("Warning: {} {} found but could not be built.".format(name, version))
return
except HttpError as e:
print("Warning: {} {} returned {}.".format(name, version, e))
return

http = build_http()
response, content = http.request(
uri or uritemplate.expand(
FLAGS.discovery_uri_template, {"api": name, "apiVersion": version}
)
)
discovery = json.loads(content)

version = safe_version(version)
@@ -19,24 +19,24 @@ It is important to understand the basics of how API authentication and authoriza
These API calls do not access any private user data. Your application must authenticate itself as an application belonging to your Google Cloud project. This is needed to measure project usage for accounting purposes.

**API key**: To authenticate your application, use an [API key](https://cloud.google.com/docs/authentication/api-keys) for your Google Cloud Console project. Every simple access call your application makes must include this key.

> **Warning**: Keep your API key private. If someone obtains your key, they could use it to consume your quota or incur charges against your Google Cloud project.
### 2. Authorized API access (OAuth 2.0)

These API calls access private user data. Before you can call them, the user that has access to the private data must grant your application access. Therefore, your application must be authenticated, the user must grant access for your application, and the user must be authenticated in order to grant that access. All of this is accomplished with [OAuth 2.0](https://developers.google.com/identity/protocols/OAuth2) and libraries written for it.

* **Scope**: Each API defines one or more scopes that declare a set of operations permitted. For example, an API might have read-only and read-write scopes. When your application requests access to user data, the request must include one or more scopes. The user needs to approve the scope of access your application is requesting. A list of accessible OAuth 2.0 scopes can be [found here](https://developers.google.com/identity/protocols/oauth2/scopes).
* **Refresh and access tokens**: When a user grants your application access, the OAuth 2.0 authorization server provides your application with refresh and access tokens. These tokens are only valid for the scope requested. Your application uses access tokens to authorize API calls. Access tokens expire, but refresh tokens do not. Your application can use a refresh token to acquire a new access token.

> **Warning**: Keep refresh and access tokens private. If someone obtains your tokens, they could use them to access private user data.
* **Client ID and client secret**: These strings uniquely identify your application and are used to acquire tokens. They are created for your Google Cloud project on the [API Access pane](https://console.developers.google.com/apis/credentials) of the Google Cloud. There are several types of client IDs, so be sure to get the correct type for your application:

* Web application client IDs
* Installed application client IDs
* [Service Account](https://developers.google.com/identity/protocols/OAuth2ServiceAccount) client IDs

> **Warning**: Keep your client secret private. If someone obtains your client secret, they could use it to consume your quota, incur charges against your Google Cloud project, and request access to user data.
## Building and calling a service
@@ -45,7 +45,7 @@ This section describes how to build an API-specific service object, make calls t

### Build the service object

Whether you are using simple or authorized API access, you use the [build()](http://googleapis.github.io/google-api-python-client/docs/epy/googleapiclient.discovery-module.html#build) function to create a service object. It takes an API name and API version as arguments. You can see the list of all API versions on the [Supported APIs](dyn/index.md) page. The service object is constructed with methods specific to the given API.
Whether you are using simple or authorized API access, you use the [build()](http://googleapis.github.io/google-api-python-client/docs/epy/googleapiclient.discovery-module.html#build) function to create a service object. It takes an API name and API version as arguments. You can see the list of all API versions on the [Supported APIs](dyn/index.md) page. When `build()` is called, a service object will attempt to be constructed with methods specific to the given API.

`httplib2`, the underlying transport library, makes all connections persistent by default. Use the service object with a context manager or call `close` to avoid leaving sockets open.

@@ -65,6 +65,8 @@ with build('drive', 'v3') as service:
# ...
```

**Note**: Under the hood, the `build()` function retrieves a discovery artifact in order to construct the service object. If the `cache_discovery` argument of `build()` is set to `True`, the library will attempt to retrieve the discovery artifact from the legacy cache which is only supported with `oauth2client<4.0`. If the artifact is not available in the legacy cache and the `static_discovery` argument of `build()` is set to `True`, which is the default, the library will use the service definition shipped in the library. If always using the latest version of a service definition is more important than reliability, users should set `static_discovery=False` to retrieve the service definition from the internet.

### Collections

Each API service provides access to one or more resources. A set of resources of the same type is called a collection. The names of these collections are specific to the API. The service object is constructed with a function for every collection defined by the API. If the given API has a collection named `stamps`, you create the collection object like this:
@@ -193,6 +193,7 @@ def build(
adc_cert_path=None,
adc_key_path=None,
num_retries=1,
static_discovery=True,
):
"""Construct a Resource for interacting with an API.
@@ -246,6 +247,8 @@ def build(
https://google.aip.dev/auth/4114
num_retries: Integer, number of times to retry discovery with
randomized exponential backoff in case of intermittent/connection issues.
static_discovery: Boolean, whether or not to use the static discovery docs
included in the library.
Returns:
A Resource object with methods for interacting with the service.
@@ -271,9 +274,12 @@ def build(
requested_url,
discovery_http,
cache_discovery,
serviceName,
version,
cache,
developerKey,
num_retries=num_retries,
static_discovery=static_discovery,
)
service = build_from_document(
content,
@@ -330,7 +336,15 @@ def _discovery_service_uri_options(discoveryServiceUrl, version):


def _retrieve_discovery_doc(
url, http, cache_discovery, cache=None, developerKey=None, num_retries=1
url,
http,
cache_discovery,
serviceName,
version,
cache=None,
developerKey=None,
num_retries=1,
static_discovery=True
):
"""Retrieves the discovery_doc from cache or the internet.
@@ -339,26 +353,39 @@ def _retrieve_discovery_doc(
http: httplib2.Http, An instance of httplib2.Http or something that acts
like it through which HTTP requests will be made.
cache_discovery: Boolean, whether or not to cache the discovery doc.
serviceName: string, name of the service.
version: string, the version of the service.
cache: googleapiclient.discovery_cache.base.Cache, an optional cache
object for the discovery documents.
developerKey: string, Key for controlling API usage, generated
from the API Console.
num_retries: Integer, number of times to retry discovery with
randomized exponential backoff in case of intermittent/connection issues.
static_discovery: Boolean, whether or not to use the static discovery docs
included in the library.
Returns:
A unicode string representation of the discovery document.
"""
if cache_discovery:
from . import discovery_cache
from . import discovery_cache

if cache_discovery:
if cache is None:
cache = discovery_cache.autodetect()
if cache:
content = cache.get(url)
if content:
return content

# When `static_discovery=True`, use static discovery artifacts included
# with the library
if static_discovery:
content = discovery_cache.get_static_doc(serviceName, version)
if content:
return content
else:
raise UnknownApiNameOrVersion("name: %s version: %s" % (serviceName, version))

actual_url = url
# REMOTE_ADDR is defined by the CGI spec [RFC3875] as the environment
# variable that contains the network address of the client sending the
@@ -23,7 +23,8 @@
LOGGER = logging.getLogger(__name__)

DISCOVERY_DOC_MAX_AGE = 60 * 60 * 24 # 1 day

DISCOVERY_DOC_DIR = os.path.join(os.path.dirname(
os.path.realpath(__file__)), 'documents')

def autodetect():
"""Detects an appropriate cache module and returns it.
@@ -48,3 +49,29 @@ def autodetect():
LOGGER.info("file_cache is only supported with oauth2client<4.0.0",
exc_info=False)
return None

def get_static_doc(serviceName, version):
"""Retrieves the discovery document from the directory defined in
DISCOVERY_DOC_DIR corresponding to the serviceName and version provided.
Args:
serviceName: string, name of the service.
version: string, the version of the service.
Returns:
A string containing the contents of the JSON discovery document,
otherwise None if the JSON discovery document was not found.
"""

content = None
doc_name = "{}.{}.json".format(serviceName, version)

try:
with open(os.path.join(DISCOVERY_DOC_DIR, doc_name), 'r') as f:
content = f.read()
except FileNotFoundError:
# File does not exist. Nothing to do here.
pass

return content

0 comments on commit 32d1c59

Please sign in to comment.