[AIRFLOW-5199] Move GCP Spanner to core

This commit moves GCP Spanner from contrib to core. For more information check AIP-21.
apache · Aug 16, 2019 · 4e4f7d4 · 4e4f7d4
1 parent e6a20ac
commit 4e4f7d4
Show file tree

Hide file tree

Showing 12 changed files with 975 additions and 910 deletions.
diff --git a/airflow/contrib/hooks/gcp_spanner_hook.py b/airflow/contrib/hooks/gcp_spanner_hook.py
diff --git a/airflow/contrib/operators/gcp_spanner_operator.py b/airflow/contrib/operators/gcp_spanner_operator.py
diff --git a/...ntrib/example_dags/example_gcp_spanner.py → airflow/gcp/example_dags/example_spanner.py b/...ntrib/example_dags/example_gcp_spanner.py → airflow/gcp/example_dags/example_spanner.py
@@ -37,7 +37,7 @@
 
 import airflow
 from airflow import models
-from airflow.contrib.operators.gcp_spanner_operator import \
+from airflow.gcp.operators.spanner import \
     CloudSpannerInstanceDeployOperator, \
     CloudSpannerInstanceDatabaseQueryOperator, \
     CloudSpannerInstanceDeleteOperator, \

diff --git a/...trib/example_dags/example_gcp_spanner.sql → airflow/gcp/example_dags/example_spanner.sql b/...trib/example_dags/example_gcp_spanner.sql → airflow/gcp/example_dags/example_spanner.sql
diff --git a/airflow/gcp/hooks/spanner.py b/airflow/gcp/hooks/spanner.py
diff --git a/airflow/gcp/operators/spanner.py b/airflow/gcp/operators/spanner.py
diff --git a/docs/howto/operator/gcp/spanner.rst b/docs/howto/operator/gcp/spanner.rst
@@ -31,14 +31,14 @@ Deletes a database from the specified Cloud Spanner instance. If the database do
 exist, no action is taken, and the operator succeeds.
 
 For parameter definition, take a look at
-:class:`~airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDatabaseDeleteOperator`.
+:class:`~airflow.gcp.operators.spanner.CloudSpannerInstanceDatabaseDeleteOperator`.
 
 Arguments
 """""""""
 
 Some arguments in the example DAG are taken from environment variables.
 
-.. exampleinclude:: ../../../../airflow/contrib/example_dags/example_gcp_spanner.py
+.. exampleinclude:: ../../../../airflow/gcp/example_dags/example_spanner.py
     :language: python
     :start-after: [START howto_operator_spanner_arguments]
     :end-before: [END howto_operator_spanner_arguments]
@@ -49,7 +49,7 @@ Using the operator
 You can create the operator with or without project id. If project id is missing
 it will be retrieved from the GCP connection used. Both variants are shown:
 
-.. exampleinclude:: ../../../../airflow/contrib/example_dags/example_gcp_spanner.py
+.. exampleinclude:: ../../../../airflow/gcp/example_dags/example_spanner.py
     :language: python
     :dedent: 4
     :start-after: [START howto_operator_spanner_database_delete]
@@ -58,7 +58,7 @@ it will be retrieved from the GCP connection used. Both variants are shown:
 Templating
 """"""""""
 
-.. literalinclude:: ../../../../airflow/contrib/operators/gcp_spanner_operator.py
+.. literalinclude:: ../../../../airflow/gcp/operators/spanner.py
     :language: python
     :dedent: 4
     :start-after: [START gcp_spanner_delete_template_fields]
@@ -81,14 +81,14 @@ configuration. No structure of the database is verified - it's enough if the dat
 with the same name.
 
 For parameter definition, take a look at
-:class:`~airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDatabaseDeployOperator`.
+:class:`~airflow.gcp.operators.spanner.CloudSpannerInstanceDatabaseDeployOperator`.
 
 Arguments
 """""""""
 
 Some arguments in the example DAG are taken from environment variables.
 
-.. exampleinclude:: ../../../../airflow/contrib/example_dags/example_gcp_spanner.py
+.. exampleinclude:: ../../../../airflow/gcp/example_dags/example_spanner.py
     :language: python
     :start-after: [START howto_operator_spanner_arguments]
     :end-before: [END howto_operator_spanner_arguments]
@@ -99,7 +99,7 @@ Using the operator
 You can create the operator with or without project id. If project id is missing
 it will be retrieved from the GCP connection used. Both variants are shown:
 
-.. exampleinclude:: ../../../../airflow/contrib/example_dags/example_gcp_spanner.py
+.. exampleinclude:: ../../../../airflow/gcp/example_dags/example_spanner.py
     :language: python
     :dedent: 4
     :start-after: [START howto_operator_spanner_database_deploy]
@@ -108,7 +108,7 @@ it will be retrieved from the GCP connection used. Both variants are shown:
 Templating
 """"""""""
 
-.. literalinclude:: ../../../../airflow/contrib/operators/gcp_spanner_operator.py
+.. literalinclude:: ../../../../airflow/gcp/operators/spanner.py
     :language: python
     :dedent: 4
     :start-after: [START gcp_spanner_database_deploy_template_fields]
@@ -135,14 +135,14 @@ a valid identifier: `[a-z][a-z0-9_]*`. More information can be found in
 `the documentation of updateDdl API <https://cloud.google.com/spanner/docs/reference/rest/v1/projects.instances.databases/updateDdl>`_
 
 For parameter definition take a look at
-:class:`~airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDatabaseUpdateOperator`.
+:class:`~airflow.gcp.operators.spanner.CloudSpannerInstanceDatabaseUpdateOperator`.
 
 Arguments
 """""""""
 
 Some arguments in the example DAG are taken from environment variables.
 
-.. exampleinclude:: ../../../../airflow/contrib/example_dags/example_gcp_spanner.py
+.. exampleinclude:: ../../../../airflow/gcp/example_dags/example_spanner.py
     :language: python
     :start-after: [START howto_operator_spanner_arguments]
     :end-before: [END howto_operator_spanner_arguments]
@@ -153,13 +153,13 @@ Using the operator
 You can create the operator with or without project id. If project id is missing
 it will be retrieved from the GCP connection used. Both variants are shown:
 
-.. exampleinclude:: ../../../../airflow/contrib/example_dags/example_gcp_spanner.py
+.. exampleinclude:: ../../../../airflow/gcp/example_dags/example_spanner.py
     :language: python
     :dedent: 4
     :start-after: [START howto_operator_spanner_database_update]
     :end-before: [END howto_operator_spanner_database_update]
 
-.. exampleinclude:: ../../../../airflow/contrib/example_dags/example_gcp_spanner.py
+.. exampleinclude:: ../../../../airflow/gcp/example_dags/example_spanner.py
     :language: python
     :dedent: 4
     :start-after: [START howto_operator_spanner_database_update_idempotent]
@@ -168,7 +168,7 @@ it will be retrieved from the GCP connection used. Both variants are shown:
 Templating
 """"""""""
 
-.. literalinclude:: ../../../../airflow/contrib/operators/gcp_spanner_operator.py
+.. literalinclude:: ../../../../airflow/gcp/operators/spanner.py
     :language: python
     :dedent: 4
     :start-after: [START gcp_spanner_database_update_template_fields]
@@ -188,14 +188,14 @@ CloudSpannerInstanceDatabaseQueryOperator
 Executes an arbitrary DML query (INSERT, UPDATE, DELETE).
 
 For parameter definition take a look at
-:class:`~airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDatabaseQueryOperator`.
+:class:`~airflow.gcp.operators.spanner.CloudSpannerInstanceDatabaseQueryOperator`.
 
 Arguments
 """""""""
 
 Some arguments in the example DAG are taken from environment variables.
 
-.. exampleinclude:: ../../../../airflow/contrib/example_dags/example_gcp_spanner.py
+.. exampleinclude:: ../../../../airflow/gcp/example_dags/example_spanner.py
     :language: python
     :start-after: [START howto_operator_spanner_arguments]
     :end-before: [END howto_operator_spanner_arguments]
@@ -206,7 +206,7 @@ Using the operator
 You can create the operator with or without project id. If project id is missing
 it will be retrieved from the GCP connection used. Both variants are shown:
 
-.. exampleinclude:: ../../../../airflow/contrib/example_dags/example_gcp_spanner.py
+.. exampleinclude:: ../../../../airflow/gcp/example_dags/example_spanner.py
     :language: python
     :dedent: 4
     :start-after: [START howto_operator_spanner_query]
@@ -215,7 +215,7 @@ it will be retrieved from the GCP connection used. Both variants are shown:
 Templating
 """"""""""
 
-.. literalinclude:: ../../../../airflow/contrib/operators/gcp_spanner_operator.py
+.. literalinclude:: ../../../../airflow/gcp/operators/spanner.py
     :language: python
     :dedent: 4
     :start-after: [START gcp_spanner_query_template_fields]
@@ -236,14 +236,14 @@ Deletes a Cloud Spanner instance. If an instance does not exist, no action is ta
 and the operator succeeds.
 
 For parameter definition take a look at
-:class:`~airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDeleteOperator`.
+:class:`~airflow.gcp.operators.spanner.CloudSpannerInstanceDeleteOperator`.
 
 Arguments
 """""""""
 
 Some arguments in the example DAG are taken from environment variables:
 
-.. exampleinclude:: ../../../../airflow/contrib/example_dags/example_gcp_spanner.py
+.. exampleinclude:: ../../../../airflow/gcp/example_dags/example_spanner.py
     :language: python
     :start-after: [START howto_operator_spanner_arguments]
     :end-before: [END howto_operator_spanner_arguments]
@@ -254,7 +254,7 @@ Using the operator
 You can create the operator with or without project id. If project id is missing
 it will be retrieved from the GCP connection used. Both variants are shown:
 
-.. exampleinclude:: ../../../../airflow/contrib/example_dags/example_gcp_spanner.py
+.. exampleinclude:: ../../../../airflow/gcp/example_dags/example_spanner.py
     :language: python
     :dedent: 4
     :start-after: [START howto_operator_spanner_delete]
@@ -263,7 +263,7 @@ it will be retrieved from the GCP connection used. Both variants are shown:
 Templating
 """"""""""
 
-.. literalinclude:: ../../../../airflow/contrib/operators/gcp_spanner_operator.py
+.. literalinclude:: ../../../../airflow/gcp/operators/spanner.py
     :language: python
     :dedent: 4
     :start-after: [START gcp_spanner_delete_template_fields]

diff --git a/docs/integration.rst b/docs/integration.rst
@@ -364,26 +364,26 @@ They also use :class:`airflow.contrib.hooks.bigquery_hook.BigQueryHook` to commu
 Cloud Spanner
 '''''''''''''
 
-:class:`airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDatabaseDeleteOperator`
+:class:`airflow.gcp.operators.spanner.CloudSpannerInstanceDatabaseDeleteOperator`
     deletes an existing database from a Google Cloud Spanner instance or returns success if the database is missing.
 
-:class:`airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDatabaseDeployOperator`
+:class:`airflow.gcp.operators.spanner.CloudSpannerInstanceDatabaseDeployOperator`
     creates a new database in a Google Cloud instance or returns success if the database already exists.
 
-:class:`airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDatabaseQueryOperator`
+:class:`airflow.gcp.operators.spanner.CloudSpannerInstanceDatabaseQueryOperator`
     executes an arbitrary DML query (INSERT, UPDATE, DELETE).
 
-:class:`airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDatabaseUpdateOperator`
+:class:`airflow.gcp.operators.spanner.CloudSpannerInstanceDatabaseUpdateOperator`
     updates the structure of a Google Cloud Spanner database.
 
-:class:`airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDeleteOperator`
+:class:`airflow.gcp.operators.spanner.CloudSpannerInstanceDeleteOperator`
     deletes a Google Cloud Spanner instance.
 
-:class:`airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDeployOperator`
+:class:`airflow.gcp.operators.spanner.CloudSpannerInstanceDeployOperator`
     creates a new Google Cloud Spanner instance, or if an instance with the same name exists, updates the instance.
 
 
-They also use :class:`airflow.contrib.hooks.gcp_spanner_hook.CloudSpannerHook` to communicate with Google Cloud Platform.
+They also use :class:`airflow.gcp.hooks.spanner.CloudSpannerHook` to communicate with Google Cloud Platform.
 
 
 Cloud SQL