Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds Cloud Function API Reference #43

Merged
merged 3 commits into from
Oct 14, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
210 changes: 142 additions & 68 deletions docs/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -32,13 +32,12 @@ User Guide
There are two ways to use some functions with GCloud-utils: as CLI or as a python package.

CLI
----------
===
The are some functions that can be used as a CLI.

- **Saving query result in a BigQuery table**

.. code-block:: console
:linenos:

query_to_table dataset table json_key YYYYMMDD query_file -Aquery_arg1=arg -Aquery_arg2=arg"

Expand All @@ -57,7 +56,6 @@ The CLI allows put some fixed variables in queries:
- **Importing table from BigQuery to Cloud Storage**

.. code-block:: console
:linenos:

table_to_gcs dataset table bucket cloudstorage_filename json_key YYYYMMDD time_delta export_format compression_format

Expand All @@ -72,7 +70,6 @@ Where the parameters are:
- **Save table from BigQuery in GoogleStorage**

.. code-block:: console
:linenos:

gcs_to_table bucket cloudstorage_filename dataset table json_key YYYYMMDD

Expand All @@ -81,112 +78,189 @@ Where the parameters are:
- ``json_key``: Credentials to bigquery service

Python package
--------------
==============

- **Simple query**
BigQuery
--------

**Simple query**

.. code-block:: python
:linenos:

from google.cloud import bigquery
from gcloud_utils.bigquery.bigquery import Bigquery
from google.cloud import bigquery
from gcloud_utils.bigquery.bigquery import Bigquery

query = "select * from bq_table"
query = "SELECT * FROM bq_table"

client = bigquery.Client.from_service_account_json(args.gcs_key_json)
bq_client = Bigquery(client)
result = bq_client.query(self, query, **kwargs)
client = bigquery.Client.from_service_account_json(args.gcs_key_json)
bq_client = Bigquery(client)

- **Query with parameters**
result = bq_client.query(self, query, **kwargs)

**Query with parameters**

.. code-block:: python
:linenos:

from google.cloud import bigquery
from gcloud_utils.bigquery.bigquery import Bigquery
from gcloud_utils.bigquery.query_builder import QueryBuilder
from google.cloud import bigquery
from gcloud_utils.bigquery.bigquery import Bigquery
from gcloud_utils.bigquery.query_builder import QueryBuilder

query = QueryBuilder("select * from ${my_table}")
query = QueryBuilder("SELECT * FROM ${my_table}")
query.with_vars(my_table="bq_table")

query.with_vars(my_table="bq_table")
client = bigquery.Client.from_service_account_json(args.gcs_key_json)
bq_client = Bigquery(client)

client = bigquery.Client.from_service_account_json(args.gcs_key_json)
bq_client = Bigquery(client)
result = bq_client.query(self, query, **kwargs)
result = bq_client.query(self, query)

- **Saving Query in BigQuery**
**Saving Query in BigQuery**

.. code-block:: python
:linenos:

from google.cloud import bigquery
client = bigquery.Client.from_service_account_json(args.gcs_key_json)
bq_client = Bigquery(client)
bq_client.query_to_table(
query_or_object,
dataset_id,
table_id,
write_disposition="WRITE_TRUNCATE",
job_config=None,
**kwargs)

- **Saving BigQuery's table in Cloud Storage**

client = bigquery.Client.from_service_account_json(args.gcs_key_json)
bq_client = Bigquery(client)

bq_client.query_to_table(
query_or_object,
dataset_id,
table_id,
write_disposition="WRITE_TRUNCATE",
job_config=None,
)

**Saving BigQuery's table in Cloud Storage**

.. code-block:: python
:linenos:

from google.cloud import bigquery
client = bigquery.Client.from_service_account_json(args.gcs_key_json)
bq_client = Bigquery(client)
bq_client.table_to_cloud_storage(
dataset_id,
table_id,
bucket_name,
filename,
job_config=None,
export_format="csv",
compression_format="gz",
location="US",
**kwargs)

- **Salving Cloud Storage in BigQuery's table**

client = bigquery.Client.from_service_account_json(args.gcs_key_json)
bq_client = Bigquery(client)

bq_client.table_to_cloud_storage(
dataset_id,
table_id,
bucket_name,
filename,
job_config=None,
export_format="csv",
compression_format="gz",
location="US",
)

**Salving Cloud Storage in BigQuery's table**

.. code-block:: python
:linenos:

from google.cloud import bigquery
client = bigquery.Client.from_service_account_json(args.gcs_key_json)
bq_client = Bigquery(client)
bq_client.cloud_storage_to_table(
bucket_name,
filename,
dataset_id,
table_id,
job_config=None,
location="US",
**kwargs)
from google.cloud import bigquery

client = bigquery.Client.from_service_account_json(args.gcs_key_json)
bq_client = Bigquery(client)

bq_client.cloud_storage_to_table(
bucket_name,
filename,
dataset_id,
table_id,
job_config=None,
location="US",
)

Cloud Function
--------------

**Create a function**

.. code-block:: python

from gcloud_utils.functions import Functions

functions_handler = Functions('my-project', 'us-central1-a')

function_name = 'my-function-name'
function_path = '/path/to/function.py'
function_runtime = 'python37'

functions_handler.create_function(
function_name,
function_runtime,
function_path,
)

**List all functions**

.. code-block:: python

from gcloud_utils.functions import Functions

functions_handler = Functions('my-project', 'us-central1-a')

for function in functions_handler.list_functions():
print(function.name)

**Describe a specific function**

.. code-block:: python

from gcloud_utils.functions import Functions

functions_handler = Functions('my-project', 'us-central1-a')
function_detail = functions_handler.describe_function('my-function-name')

print('Status: {}'.format(function_detail.status))
print('Last update: {}'.format(function_detail.updateTime))

**Trigger a function**

.. code-block:: python

import json

from gcloud_utils.functions import Functions

functions_handler = Functions('my-project', 'us-central1-a')

data = json.dumps({'example': 'example'})
functions_handler.call_function('my-function-name', data)


API Reference
=============

MLEngine
=================
--------

.. automodule:: gcloud_utils.ml_engine
:members:

Compute
=================
-------

.. automodule:: gcloud_utils.compute
:members:

Cloud Function
--------------

.. automodule:: gcloud_utils.functions
:members:

BigQuery
=================
--------

.. automodule:: gcloud_utils.bigquery.bigquery
:members:

Dataproc
=================
--------

.. automodule:: gcloud_utils.dataproc
:members:

Storage
=================
-------

.. automodule:: gcloud_utils.storage
:members:
3 changes: 2 additions & 1 deletion requirements_dev.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,5 @@ pytest==3.8.1
freezegun==0.3.10
mock==2.0.0
coverage==4.5.1
twine==1.13.0
twine==1.13.0
Sphinx==3.2.1