Add CloudRunExecuteJobOperator #28525

VinceLegendre · 2022-12-21T17:21:56Z

This operator aims at executing existing Cloud Run jobs, with the same design philosophy as the DataflowStartFlexTemplateOperator.

CloudRunExecuteJobOperator extends the BaseOperator
CloudRunJobHook extends the GoogleBaseHook, to easily manage authentication and project_id fallback
CloudRunJobLink andCloudRunJobExecutionLink extend the BaseGoogleLink

Closes: #24730

boring-cyborg · 2022-12-21T17:22:00Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

tests/providers/google/cloud/hooks/test_cloud_run.py

lwyszomi · 2023-01-12T12:18:04Z

airflow/providers/google/cloud/hooks/cloud_run.py

+    def get_conn(self) -> build:
+        """Returns a Google Cloud Run service object."""
+        http_authorized = self._authorize()
+
+        # Use a regional endpoint since job execution is not possible from the global endpoint
+        client_options = ClientOptions(api_endpoint=f'https://{self.region}-run.googleapis.com')
+        return build('run', 'v1', client_options=client_options,
+                     http=http_authorized, cache_discovery=False)


Why we are not using JobsClient from python SDK? https://cloud.google.com/python/docs/reference/run/latest/google.cloud.run_v2.services.jobs.JobsClient

+1, the v2 API and its Python client libraries should be much easier to use.

Hi @lwyszomi @steren

Cannot install the python client library (google-cloud-run>=0.7.0) as it breaks other dependencies requirements.
Happy to use the JobsClient if we manage to install google-cloud-run library

@Taragolis would you have an idea of how to fix this?

If using the official Python Client and implementing all the CRUD operations on jobs are completion requirements for this PR, we should mark this PR as draft for the time being.

FYI Completion requirements were discussed in this issue : #24730.

Maybe we could consider all the Jobs CRUD operations in a later PR, when the official JobsClient can be installed properly?

Opened to suggestions here 😉

tests/providers/google/cloud/hooks/test_cloud_run.py

airflow/providers/google/cloud/hooks/cloud_run.py

Taragolis

Approve in advance (need to resolve doc spellcheck)

I'm not familiar with GCP, for me looks nice. So would be nice if someone else review it

airflow/providers/google/cloud/hooks/cloud_run.py

eladkal · 2023-03-09T07:11:00Z

tests are failing :(

eladkal · 2023-03-09T07:13:35Z

I'm assuming that by merging this PR we won't need #27638 ?

alexshires · 2023-03-15T17:22:31Z

I'm assuming that by merging this PR we won't need #27638 ?

CloudRun is a container hosting service and CloudRunJobs are a job-based container execution version of that - IMO, we would need both sets of operators in Airflow to support both functionalities

VinceLegendre · 2023-03-15T22:13:10Z

I'll try to fix tests in the coming days.

VinceLegendre · 2023-04-06T21:05:42Z

Tests should be fixed now @eladkal .
Waiting for approvals to run all tests

potiuk · 2023-04-07T10:49:44Z

Tests should be fixed now @eladkal . Waiting for approvals to run all tests

approved

VinceLegendre · 2023-04-07T12:19:25Z

Looks like the failing test is not related to my implementation. I believe I should wait for a fix and rebase my branch from time to time?

VinceLegendre · 2023-04-13T14:37:04Z

Checks seem ok (appart from 9 skipped) @eladkal 🙂
Is there a further step to approve these changes? Should I look for additional reviewers in the community?

eladkal · 2023-04-17T20:48:37Z

Is there a further step to approve these changes? Should I look for additional reviewers in the community?

waiting for your comment on #28525 (comment)

eladkal · 2023-05-21T13:25:34Z

@VinceLegendre We bumped many google packages #30067
Can you please recheck and comment on #28525 (comment) if there is still a reason not to use V2?

- New hook : CloudRunJobHook - New links : CloudRunJobLink and CloudRunJobExecutionLinks - New operator : CloudRunExecuteJobOperator - According tests - Documentation and examples

potiuk · 2023-05-21T15:04:46Z

I just rebased it to see if it works after recent upgrade of deps. I don't think there will be a big problem, It does not use any of the google client specific libraries - but the discovery client, so it should be fine as-is.

potiuk · 2023-05-22T08:33:59Z

I just rebased it to see if it works after recent upgrade of deps. I don't think there will be a big problem, It does not use any of the google client specific libraries - but the discovery client, so it should be fine as-is.

However. I think indeed. it would be better to use the actual cloud run client rather than discovery one now. It should be possible now, after all the dependencies are updated, So the reason why discovery client was used here is gone. @VinceLegendre ?

VinceLegendre · 2023-05-22T08:50:36Z

@potiuk @eladkal I think we should switch to the official client too.

Currently away from work for two weeks, I can try to work on it when I get back.

If this is urgent matter, please feel free to initiate new developments and I'll be happy to contribute 🙂

potiuk · 2023-05-22T11:04:45Z

No worries. It can wait

eladkal · 2023-06-02T16:06:03Z

@potiuk @eladkal I think we should switch to the official client too.

Cool! please ping me when ready for review

eladkal · 2023-06-30T09:56:27Z

Hi @VinceLegendre checking if you still plan to make the changes?

github-actions · 2023-08-15T00:10:25Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

VinceLegendre requested review from turbaszek and mik-laj as code owners December 21, 2022 17:21

boring-cyborg bot added area:providers area:system-tests kind:documentation provider:google Google (including GCP) related issues labels Dec 21, 2022

VinceLegendre changed the title ~~Add CloudRunExecuteJobOperator and required dependencies~~ Add CloudRunExecuteJobOperator Dec 21, 2022

VinceLegendre mentioned this pull request Dec 22, 2022

Google CloudRun job operator #24730

Closed

2 tasks

Taragolis reviewed Dec 22, 2022

View reviewed changes

tests/providers/google/cloud/hooks/test_cloud_run.py Outdated Show resolved Hide resolved

VinceLegendre force-pushed the add_google_cloud_run_execute_job_operator branch 2 times, most recently from 2c8fce5 to 20bfa7b Compare December 29, 2022 11:16

VinceLegendre requested review from Taragolis and removed request for turbaszek and mik-laj December 29, 2022 16:49

VinceLegendre force-pushed the add_google_cloud_run_execute_job_operator branch from 20bfa7b to 48eed2e Compare January 2, 2023 09:12

lwyszomi reviewed Jan 12, 2023

View reviewed changes

matthieuvilatte mentioned this pull request Feb 8, 2023

Add Cloud Run Operator #27638

Closed

Taragolis reviewed Feb 15, 2023

View reviewed changes

tests/providers/google/cloud/hooks/test_cloud_run.py Outdated Show resolved Hide resolved

VinceLegendre force-pushed the add_google_cloud_run_execute_job_operator branch 3 times, most recently from 104d1ef to 420c506 Compare February 16, 2023 15:45

Taragolis reviewed Feb 16, 2023

View reviewed changes

tests/providers/google/cloud/hooks/test_cloud_run.py Outdated Show resolved Hide resolved

VinceLegendre force-pushed the add_google_cloud_run_execute_job_operator branch from 420c506 to fabda1d Compare February 17, 2023 17:35

Taragolis reviewed Feb 17, 2023

View reviewed changes

airflow/providers/google/cloud/hooks/cloud_run.py Outdated Show resolved Hide resolved

Taragolis approved these changes Feb 17, 2023

View reviewed changes

VinceLegendre force-pushed the add_google_cloud_run_execute_job_operator branch from fabda1d to 2a95a05 Compare February 18, 2023 11:38

VinceLegendre requested a review from steren February 18, 2023 11:51

VinceLegendre force-pushed the add_google_cloud_run_execute_job_operator branch from 2a95a05 to d2a90cc Compare February 18, 2023 13:58

Taragolis reviewed Feb 18, 2023

View reviewed changes

airflow/providers/google/cloud/hooks/cloud_run.py Outdated Show resolved Hide resolved

VinceLegendre force-pushed the add_google_cloud_run_execute_job_operator branch 2 times, most recently from ab3e06e to c58f4a0 Compare February 23, 2023 14:29

VinceLegendre force-pushed the add_google_cloud_run_execute_job_operator branch 2 times, most recently from 7265c97 to 334efe5 Compare April 4, 2023 14:15

VinceLegendre requested a review from potiuk April 5, 2023 07:53

VinceLegendre force-pushed the add_google_cloud_run_execute_job_operator branch from 334efe5 to 7f19415 Compare April 11, 2023 08:16

VinceLegendre requested a review from r-richmond April 13, 2023 07:51

eladkal mentioned this pull request May 21, 2023

GCP Cloud Run Operator? #31447

Closed

2 tasks

Add CloudRunExecuteJobOperator

587504e

- New hook : CloudRunJobHook - New links : CloudRunJobLink and CloudRunJobExecutionLinks - New operator : CloudRunExecuteJobOperator - According tests - Documentation and examples

potiuk force-pushed the add_google_cloud_run_execute_job_operator branch from 7f19415 to 587504e Compare May 21, 2023 15:01

github-actions bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Aug 15, 2023

eladkal mentioned this pull request Aug 16, 2023

Add CloudRunHook and operators #33067

Merged

github-actions bot closed this Aug 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add CloudRunExecuteJobOperator #28525

Add CloudRunExecuteJobOperator #28525

VinceLegendre commented Dec 21, 2022 •

edited by josh-fell

boring-cyborg bot commented Dec 21, 2022

lwyszomi Jan 12, 2023

steren Feb 8, 2023

VinceLegendre Feb 16, 2023

VinceLegendre Feb 17, 2023 •

edited

VinceLegendre Feb 18, 2023 •

edited

Taragolis left a comment

eladkal commented Mar 9, 2023

eladkal commented Mar 9, 2023

alexshires commented Mar 15, 2023

VinceLegendre commented Mar 15, 2023

VinceLegendre commented Apr 6, 2023

potiuk commented Apr 7, 2023

VinceLegendre commented Apr 7, 2023

VinceLegendre commented Apr 13, 2023

eladkal commented Apr 17, 2023

eladkal commented May 21, 2023

potiuk commented May 21, 2023

potiuk commented May 22, 2023

VinceLegendre commented May 22, 2023

potiuk commented May 22, 2023

eladkal commented Jun 2, 2023

eladkal commented Jun 30, 2023

github-actions bot commented Aug 15, 2023

Add CloudRunExecuteJobOperator #28525

Add CloudRunExecuteJobOperator #28525

Conversation

VinceLegendre commented Dec 21, 2022 • edited by josh-fell

boring-cyborg bot commented Dec 21, 2022

lwyszomi Jan 12, 2023

Choose a reason for hiding this comment

steren Feb 8, 2023

Choose a reason for hiding this comment

VinceLegendre Feb 16, 2023

Choose a reason for hiding this comment

VinceLegendre Feb 17, 2023 • edited

Choose a reason for hiding this comment

VinceLegendre Feb 18, 2023 • edited

Choose a reason for hiding this comment

Taragolis left a comment

Choose a reason for hiding this comment

eladkal commented Mar 9, 2023

eladkal commented Mar 9, 2023

alexshires commented Mar 15, 2023

VinceLegendre commented Mar 15, 2023

VinceLegendre commented Apr 6, 2023

potiuk commented Apr 7, 2023

VinceLegendre commented Apr 7, 2023

VinceLegendre commented Apr 13, 2023

eladkal commented Apr 17, 2023

eladkal commented May 21, 2023

potiuk commented May 21, 2023

potiuk commented May 22, 2023

VinceLegendre commented May 22, 2023

potiuk commented May 22, 2023

eladkal commented Jun 2, 2023

eladkal commented Jun 30, 2023

github-actions bot commented Aug 15, 2023

VinceLegendre commented Dec 21, 2022 •

edited by josh-fell

VinceLegendre Feb 17, 2023 •

edited

VinceLegendre Feb 18, 2023 •

edited