[AIRFLOW-6981] Move Google Cloud Build from Discovery API to Python Library

[ryanyuan]

Move Google Cloud Build from Discovery API to Python Library
Issue: #8568
Old Jira Ticket: AIRFLOW-6981

---

Make sure to mark the boxes below before creating PR: [x]

• Description above provides context of the change
• Unit tests coverage for changes (not needed for documentation changes)
• Target Github ISSUE in description if exists
• Commits follow "How to write a good git commit message"
• Relevant documentation is updated including usage instructions.
• I will engage committers as explained in Contribution Workflow Example.

---

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

[ryanyuan]

@mik-laj Any ideas why the builds failed on Requirements 3.7 and 3.6?

[potiuk]

You modified setup.py but when you do this, you need to generate requirements (both 3.6, 3.7), Just copy-pasted the error printed in the logs:

ERROR! Requirements need to be updated!

     Please generate requirements with:

           breeze generate-requirements --python 3.6

[potiuk]

You can read more about it in contributing docs: https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#generating-requirement-files

[ryanyuan]

Cheers @potiuk. This time it failed at Quarantined:Pg9.6,Py3.6. But the error log didn't say much. Any suggestions?

[mik-laj]

Please ignore tests in quarantine. They are flaky.

[potiuk]

Yep. It's also described here: https://github.com/apache/airflow/blob/master/TESTING.rst#quarantined-tests

[potiuk]

We plan to review and fix the quarantined tests in the coming weeks and make them stable.

[ryanyuan]

thanks @potiuk

[turbaszek]

For the exception you had, it seems you ran the example dag but it failed in the midway and you re-ran it so that it said the trigger already existed. I tried to add a try catch to return the trigger if it already exists but I am unable to find a good way to get the trigger id, which will be used in CloudBuildHook.get_build_trigger().

Should we then add note that the operator is not idempotent?

[ryanyuan]

For the exception you had, it seems you ran the example dag but it failed in the midway and you re-ran it so that it said the trigger already existed. I tried to add a try catch to return the trigger if it already exists but I am unable to find a good way to get the trigger id, which will be used in CloudBuildHook.get_build_trigger().

Should we then add note that the operator is not idempotent?

Good idea. I've put a message in cloud_build.rst to remind the users the build ID and the trigger ID are not idempotent.

[turbaszek]

I tried running a system test test_run_example_gcp_cloud_build_trigger_dag and I got:

[2020-09-02 14:36:56,913] {taskinstance.py:1169} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=example_gcp_cloud_build_trigger
AIRFLOW_CTX_TASK_ID=run_build_trigger
AIRFLOW_CTX_EXECUTION_DATE=2020-09-01T00:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-09-01T00:00:00+00:00
[2020-09-02 14:36:56,926] {cloud_build.py:556} INFO - Start running build trigger: d4b43269-6ab6-46e3-950e-7541fc213a6d.
[2020-09-02 14:36:56,927] {taskinstance.py:1310} ERROR - Protocol message RepoSource has no "repo_source" field.
Traceback (most recent call last):
  File "/opt/airflow/airflow/models/taskinstance.py", line 1076, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/opt/airflow/airflow/models/taskinstance.py", line 1181, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/opt/airflow/airflow/models/taskinstance.py", line 1216, in _execute_task
    result = task_copy.execute(context=context)
  File "/opt/airflow/airflow/providers/google/cloud/operators/cloud_build.py", line 778, in execute
    metadata=self.metadata,
  File "/opt/airflow/airflow/providers/google/common/hooks/base_google.py", line 373, in inner_wrapper
    return func(self, *args, **kwargs)
  File "/opt/airflow/airflow/providers/google/cloud/hooks/cloud_build.py", line 564, in run_build_trigger
    metadata=metadata,
  File "/usr/local/lib/python3.7/site-packages/google/cloud/devtools/cloudbuild_v1/gapic/cloud_build_client.py", line 946, in run_build_trigger
    project_id=project_id, trigger_id=trigger_id, source=source
ValueError: Protocol message RepoSource has no "repo_source" field.

[ryanyuan]

I tried running a system test test_run_example_gcp_cloud_build_trigger_dag and I got:

[2020-09-02 14:36:56,913] {taskinstance.py:1169} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=example_gcp_cloud_build_trigger
AIRFLOW_CTX_TASK_ID=run_build_trigger
AIRFLOW_CTX_EXECUTION_DATE=2020-09-01T00:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-09-01T00:00:00+00:00
[2020-09-02 14:36:56,926] {cloud_build.py:556} INFO - Start running build trigger: d4b43269-6ab6-46e3-950e-7541fc213a6d.
[2020-09-02 14:36:56,927] {taskinstance.py:1310} ERROR - Protocol message RepoSource has no "repo_source" field.
Traceback (most recent call last):
  File "/opt/airflow/airflow/models/taskinstance.py", line 1076, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/opt/airflow/airflow/models/taskinstance.py", line 1181, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/opt/airflow/airflow/models/taskinstance.py", line 1216, in _execute_task
    result = task_copy.execute(context=context)
  File "/opt/airflow/airflow/providers/google/cloud/operators/cloud_build.py", line 778, in execute
    metadata=self.metadata,
  File "/opt/airflow/airflow/providers/google/common/hooks/base_google.py", line 373, in inner_wrapper
    return func(self, *args, **kwargs)
  File "/opt/airflow/airflow/providers/google/cloud/hooks/cloud_build.py", line 564, in run_build_trigger
    metadata=metadata,
  File "/usr/local/lib/python3.7/site-packages/google/cloud/devtools/cloudbuild_v1/gapic/cloud_build_client.py", line 946, in run_build_trigger
    project_id=project_id, trigger_id=trigger_id, source=source
ValueError: Protocol message RepoSource has no "repo_source" field.

Should be okay now.

[mik-laj]

@ryanyuan Everything looks fine to me. Can I ask for the last rebase, please? This week I will want to merge this PR. I already have it on my to-do list and I will never miss it. I apologize for the long waiting time for reviews.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[ryanyuan]

@mik-laj no worries. PTAL.

[ryanyuan]

@mik-laj Please let me know when you are going to merge this PR. I will do the rebase again.

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

[ryanyuan]

Hi @turbaszek @mik-laj, any updates on this?