Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[AIRFLOW-5199]Move GCP Spanner to core #5802

Merged
merged 1 commit into from
Aug 17, 2019
Merged

[AIRFLOW-5199]Move GCP Spanner to core #5802

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
397 changes: 8 additions & 389 deletions airflow/contrib/hooks/gcp_spanner_hook.py
View file
Open in desktop

Large diffs are not rendered by default.

441 changes: 18 additions & 423 deletions airflow/contrib/operators/gcp_spanner_operator.py
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion ...ntrib/example_dags/example_gcp_spanner.py → airflow/gcp/example_dags/example_spanner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@

import airflow
from airflow import models
from airflow.contrib.operators.gcp_spanner_operator import \
from airflow.gcp.operators.spanner import \
CloudSpannerInstanceDeployOperator, \
CloudSpannerInstanceDatabaseQueryOperator, \
CloudSpannerInstanceDeleteOperator, \
Expand Down
0 ...trib/example_dags/example_gcp_spanner.sql → airflow/gcp/example_dags/example_spanner.sql
View file
Open in desktop
File renamed without changes.
410 changes: 410 additions & 0 deletions airflow/gcp/hooks/spanner.py
View file
Open in desktop

Large diffs are not rendered by default.

441 changes: 441 additions & 0 deletions airflow/gcp/operators/spanner.py
View file
Open in desktop

Large diffs are not rendered by default.

42 changes: 21 additions & 21 deletions docs/howto/operator/gcp/spanner.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,14 +31,14 @@ Deletes a database from the specified Cloud Spanner instance. If the database do
exist, no action is taken, and the operator succeeds.

For parameter definition, take a look at
:class:`~airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDatabaseDeleteOperator`.
:class:`~airflow.gcp.operators.spanner.CloudSpannerInstanceDatabaseDeleteOperator`.

Arguments
"""""""""

Some arguments in the example DAG are taken from environment variables.

.. exampleinclude:: ../../../../airflow/contrib/example_dags/example_gcp_spanner.py
.. exampleinclude:: ../../../../airflow/gcp/example_dags/example_spanner.py
:language: python
:start-after: [START howto_operator_spanner_arguments]
:end-before: [END howto_operator_spanner_arguments]
Expand All @@ -49,7 +49,7 @@ Using the operator
You can create the operator with or without project id. If project id is missing
it will be retrieved from the GCP connection used. Both variants are shown:

.. exampleinclude:: ../../../../airflow/contrib/example_dags/example_gcp_spanner.py
.. exampleinclude:: ../../../../airflow/gcp/example_dags/example_spanner.py
:language: python
:dedent: 4
:start-after: [START howto_operator_spanner_database_delete]
Expand All @@ -58,7 +58,7 @@ it will be retrieved from the GCP connection used. Both variants are shown:
Templating
""""""""""

.. literalinclude:: ../../../../airflow/contrib/operators/gcp_spanner_operator.py
.. literalinclude:: ../../../../airflow/gcp/operators/spanner.py
:language: python
:dedent: 4
:start-after: [START gcp_spanner_delete_template_fields]
Expand All @@ -81,14 +81,14 @@ configuration. No structure of the database is verified - it's enough if the dat
with the same name.

For parameter definition, take a look at
:class:`~airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDatabaseDeployOperator`.
:class:`~airflow.gcp.operators.spanner.CloudSpannerInstanceDatabaseDeployOperator`.

Arguments
"""""""""

Some arguments in the example DAG are taken from environment variables.

.. exampleinclude:: ../../../../airflow/contrib/example_dags/example_gcp_spanner.py
.. exampleinclude:: ../../../../airflow/gcp/example_dags/example_spanner.py
:language: python
:start-after: [START howto_operator_spanner_arguments]
:end-before: [END howto_operator_spanner_arguments]
Expand All @@ -99,7 +99,7 @@ Using the operator
You can create the operator with or without project id. If project id is missing
it will be retrieved from the GCP connection used. Both variants are shown:

.. exampleinclude:: ../../../../airflow/contrib/example_dags/example_gcp_spanner.py
.. exampleinclude:: ../../../../airflow/gcp/example_dags/example_spanner.py
:language: python
:dedent: 4
:start-after: [START howto_operator_spanner_database_deploy]
Expand All @@ -108,7 +108,7 @@ it will be retrieved from the GCP connection used. Both variants are shown:
Templating
""""""""""

.. literalinclude:: ../../../../airflow/contrib/operators/gcp_spanner_operator.py
.. literalinclude:: ../../../../airflow/gcp/operators/spanner.py
:language: python
:dedent: 4
:start-after: [START gcp_spanner_database_deploy_template_fields]
Expand All @@ -135,14 +135,14 @@ a valid identifier: `[a-z][a-z0-9_]*`. More information can be found in
`the documentation of updateDdl API <https://cloud.google.com/spanner/docs/reference/rest/v1/projects.instances.databases/updateDdl>`_

For parameter definition take a look at
:class:`~airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDatabaseUpdateOperator`.
:class:`~airflow.gcp.operators.spanner.CloudSpannerInstanceDatabaseUpdateOperator`.

Arguments
"""""""""

Some arguments in the example DAG are taken from environment variables.

.. exampleinclude:: ../../../../airflow/contrib/example_dags/example_gcp_spanner.py
.. exampleinclude:: ../../../../airflow/gcp/example_dags/example_spanner.py
:language: python
:start-after: [START howto_operator_spanner_arguments]
:end-before: [END howto_operator_spanner_arguments]
Expand All @@ -153,13 +153,13 @@ Using the operator
You can create the operator with or without project id. If project id is missing
it will be retrieved from the GCP connection used. Both variants are shown:

.. exampleinclude:: ../../../../airflow/contrib/example_dags/example_gcp_spanner.py
.. exampleinclude:: ../../../../airflow/gcp/example_dags/example_spanner.py
:language: python
:dedent: 4
:start-after: [START howto_operator_spanner_database_update]
:end-before: [END howto_operator_spanner_database_update]

.. exampleinclude:: ../../../../airflow/contrib/example_dags/example_gcp_spanner.py
.. exampleinclude:: ../../../../airflow/gcp/example_dags/example_spanner.py
:language: python
:dedent: 4
:start-after: [START howto_operator_spanner_database_update_idempotent]
Expand All @@ -168,7 +168,7 @@ it will be retrieved from the GCP connection used. Both variants are shown:
Templating
""""""""""

.. literalinclude:: ../../../../airflow/contrib/operators/gcp_spanner_operator.py
.. literalinclude:: ../../../../airflow/gcp/operators/spanner.py
:language: python
:dedent: 4
:start-after: [START gcp_spanner_database_update_template_fields]
Expand All @@ -188,14 +188,14 @@ CloudSpannerInstanceDatabaseQueryOperator
Executes an arbitrary DML query (INSERT, UPDATE, DELETE).

For parameter definition take a look at
:class:`~airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDatabaseQueryOperator`.
:class:`~airflow.gcp.operators.spanner.CloudSpannerInstanceDatabaseQueryOperator`.

Arguments
"""""""""

Some arguments in the example DAG are taken from environment variables.

.. exampleinclude:: ../../../../airflow/contrib/example_dags/example_gcp_spanner.py
.. exampleinclude:: ../../../../airflow/gcp/example_dags/example_spanner.py
:language: python
:start-after: [START howto_operator_spanner_arguments]
:end-before: [END howto_operator_spanner_arguments]
Expand All @@ -206,7 +206,7 @@ Using the operator
You can create the operator with or without project id. If project id is missing
it will be retrieved from the GCP connection used. Both variants are shown:

.. exampleinclude:: ../../../../airflow/contrib/example_dags/example_gcp_spanner.py
.. exampleinclude:: ../../../../airflow/gcp/example_dags/example_spanner.py
:language: python
:dedent: 4
:start-after: [START howto_operator_spanner_query]
Expand All @@ -215,7 +215,7 @@ it will be retrieved from the GCP connection used. Both variants are shown:
Templating
""""""""""

.. literalinclude:: ../../../../airflow/contrib/operators/gcp_spanner_operator.py
.. literalinclude:: ../../../../airflow/gcp/operators/spanner.py
:language: python
:dedent: 4
:start-after: [START gcp_spanner_query_template_fields]
Expand All @@ -236,14 +236,14 @@ Deletes a Cloud Spanner instance. If an instance does not exist, no action is ta
and the operator succeeds.

For parameter definition take a look at
:class:`~airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDeleteOperator`.
:class:`~airflow.gcp.operators.spanner.CloudSpannerInstanceDeleteOperator`.

Arguments
"""""""""

Some arguments in the example DAG are taken from environment variables:

.. exampleinclude:: ../../../../airflow/contrib/example_dags/example_gcp_spanner.py
.. exampleinclude:: ../../../../airflow/gcp/example_dags/example_spanner.py
:language: python
:start-after: [START howto_operator_spanner_arguments]
:end-before: [END howto_operator_spanner_arguments]
Expand All @@ -254,7 +254,7 @@ Using the operator
You can create the operator with or without project id. If project id is missing
it will be retrieved from the GCP connection used. Both variants are shown:

.. exampleinclude:: ../../../../airflow/contrib/example_dags/example_gcp_spanner.py
.. exampleinclude:: ../../../../airflow/gcp/example_dags/example_spanner.py
:language: python
:dedent: 4
:start-after: [START howto_operator_spanner_delete]
Expand All @@ -263,7 +263,7 @@ it will be retrieved from the GCP connection used. Both variants are shown:
Templating
""""""""""

.. literalinclude:: ../../../../airflow/contrib/operators/gcp_spanner_operator.py
.. literalinclude:: ../../../../airflow/gcp/operators/spanner.py
:language: python
:dedent: 4
:start-after: [START gcp_spanner_delete_template_fields]
Expand Down
14 changes: 7 additions & 7 deletions docs/integration.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -364,26 +364,26 @@ They also use :class:`airflow.contrib.hooks.bigquery_hook.BigQueryHook` to commu
Cloud Spanner
'''''''''''''

:class:`airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDatabaseDeleteOperator`
:class:`airflow.gcp.operators.spanner.CloudSpannerInstanceDatabaseDeleteOperator`
deletes an existing database from a Google Cloud Spanner instance or returns success if the database is missing.

:class:`airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDatabaseDeployOperator`
:class:`airflow.gcp.operators.spanner.CloudSpannerInstanceDatabaseDeployOperator`
creates a new database in a Google Cloud instance or returns success if the database already exists.

:class:`airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDatabaseQueryOperator`
:class:`airflow.gcp.operators.spanner.CloudSpannerInstanceDatabaseQueryOperator`
executes an arbitrary DML query (INSERT, UPDATE, DELETE).

:class:`airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDatabaseUpdateOperator`
:class:`airflow.gcp.operators.spanner.CloudSpannerInstanceDatabaseUpdateOperator`
updates the structure of a Google Cloud Spanner database.

:class:`airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDeleteOperator`
:class:`airflow.gcp.operators.spanner.CloudSpannerInstanceDeleteOperator`
deletes a Google Cloud Spanner instance.

:class:`airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDeployOperator`
:class:`airflow.gcp.operators.spanner.CloudSpannerInstanceDeployOperator`
creates a new Google Cloud Spanner instance, or if an instance with the same name exists, updates the instance.


They also use :class:`airflow.contrib.hooks.gcp_spanner_hook.CloudSpannerHook` to communicate with Google Cloud Platform.
They also use :class:`airflow.gcp.hooks.spanner.CloudSpannerHook` to communicate with Google Cloud Platform.


Cloud SQL
Expand Down
Loading