Provider Databricks add jobs create operator. #29790

kyle-winkelman · 2023-02-27T20:28:26Z

Add the DatabricksJobsCreateOperator for use cases where the DatabricksSubmitRunOperator is insufficient.
closes: #29733

boring-cyborg · 2023-02-27T20:28:30Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

potiuk · 2023-03-04T18:00:26Z

cc: @alexott ?

alexott · 2023-03-04T20:01:13Z

@kyle-winkelman what are the use cases for that operator?

kyle-winkelman · 2023-03-04T20:50:18Z

My specific use case is that I want to run multiple tasks in the same job and on a single job_clusters. This is not supported by the DatabricksSubmitRunOperator because it doesn't support the parameter job_clusters, so the only way to support it is to create a new_cluster per task I want to run.

The other approach is to use the DatabricksRunNowOperator which has the limitation that you have to define your Databricks Job somewhere else and in some different manner (i.e. manually in Databricks UI, custom CI/CD pipeline, etc.). My team doesn't like having the definition of the job be separated from the use of it from Airflow. In my opinion the DatabricksRunNowOperator with just a single job_id lacks information about what is actually happening.

So to sum up, it is useful to be paired with the DatabricksRunNowOperator to define a job and run it in the same DAG.

alexott

I'll discuss this further with the Databricks Workflows team

alexott · 2023-03-05T14:07:19Z

airflow/providers/databricks/operators/databricks.py

+        tasks: list[object] | None = None,
+        job_clusters: list[object] | None = None,
+        email_notifications: object | None = None,
+        webhook_notifications: object | None = None,
+        timeout_seconds: int | None = None,
+        schedule: dict[str, str] | None = None,
+        max_concurrent_runs: int | None = None,
+        git_source: dict[str, str] | None = None,
+        access_control_list: list[dict[str, str]] | None = None,


For new operators I would really like to use real data structures (for example, data classes), not the simple objects as it doesn't provide users with auto-completion, etc. - it's easy to make mistake in the untyped JSON definition

Do you have an example of a different operator that follows this pattern? It would be helpful to see an example.

Any thoughts on a tool to generate such data structures from the Databricks OpenAPI Spec? There are a lot of data structures that would need to be created and I don't want to do so manually.

My initial thought was to rely on Databricks Python API, but it doesn't have these data structures or validations either.

I don’t have existing example, it was just thoughts for the new operators. New Python API will be available soon, that will provide access to the latest APIs. I need to ask dev team when it’s coming

Do you think this new operator should wait for the new Python API that you expect to be available soon?

I’m waiting for answer from product management - maybe we won’t need this operator…

Have you heard anything from the Databricks product management @alexott?

no decision yet.

@alexott lets please get to a decision in the upcoming days (before next wave of providers)... if not i will accept the PR as is.
Should in the future Databricks decide against these operators and present arguments why we shouldn't have them we can always remove with a major release.

alexott · 2023-03-05T14:11:16Z

airflow/providers/databricks/operators/databricks.py

+            include_prior_dates=True,
+        )
+        if self.job_id:
+            self._hook.reset(self.job_id, self.json)


Instead of I would recommend to get the current job definition and compare with the new definition, and reset only if this definition changes (that should happen relatively rare).

Also, you need to handle following edge cases:

Job is deleted via UI - the reset will fail because job ID doesn't exist

We can lose XCom information, so we'll create a duplicate for the job.

potiuk

@alexott convinced me that it needs changes.

potiuk · 2023-04-23T10:50:26Z

@alexott convinced me that it needs changes.

@alexott - are there any news from Databricks on that one? We are considering merging it regardless, but it seems you wanted to have some input from the team first?

alexott · 2023-04-23T11:52:13Z

@potiuk Unfortunately no news. The decision takes longer than planned. If we can deprecate it in the later versions, let merge it.

potiuk · 2023-04-23T12:50:39Z

@potiuk Unfortunately no news. The decision takes longer than planned. If we can deprecate it in the later versions, let merge it.

OK. Cool. I am a big fan of having tactical solutions that solve part of the probem or a problem for smaller group of people and releasing them early, as long as they are not preventing strategic long term solutions that needs a longer debate and far more work to be implemented later. This one looks like one of those.

I re-reviewed it and it looks cool. @kyle-winkelman -> can you please rebase and fix the static check problem, then I am happy to merge it (@eladkal - WDYT?).

lennartkats-db · 2023-05-30T14:23:01Z

@potiuk @kyle-winkelman This seems like a useful addition from the Databricks perspective. Thanks for contributing, Kyle! 🙏

I do think it would be good to look at the XCOM issue that Alex called out. Wouldn't it be better to just use the API to find an existing job for the cases that Alex called out where the current approach doesn't work? To avoid creating duplicate jobs or having the operator fail? The DatabricksRunNowOperator already applies logic like this, as it allows users to run a given job based on the name of that job (rather than the job id).

stikkireddy · 2023-06-06T18:38:19Z

hey @kyle-winkelman do you have bandwidth to move forward with this PR?

stikkireddy · 2023-06-23T19:46:55Z

@potiuk is it okay if someone else forks Kyle's repo and fix the static check to not lose @kyle-winkelman's credit/commits and also address the previous comments made by @alexott? I am also okay to wait a bit for Kyle to respond.

potiuk · 2023-06-23T20:11:38Z

@potiuk is it okay if someone else forks Kyle's repo and fix the static check to not lose @kyle-winkelman's credit/commits and also address the previous comments made by @alexott? I am also okay to wait a bit for Kyle to respond.

Fine for me.

github-actions · 2023-08-08T00:12:16Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

boring-cyborg bot added area:providers provider:databricks labels Feb 27, 2023

Provider Databricks add jobs create operator.

e8ad958

potiuk approved these changes Mar 4, 2023

View reviewed changes

alexott reviewed Mar 5, 2023

View reviewed changes

Merge branch 'main' into databricks-jobs-create

91568b2

potiuk requested changes Apr 22, 2023

View reviewed changes

potiuk approved these changes Apr 23, 2023

View reviewed changes

stikkireddy mentioned this pull request Jun 28, 2023

Add DatabricksJobsCreateOperator #32221

Closed

github-actions bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Aug 8, 2023

github-actions bot closed this Aug 14, 2023

tatiana mentioned this pull request Oct 24, 2023

Add operator to create jobs in Databricks #35156

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Provider Databricks add jobs create operator. #29790

Provider Databricks add jobs create operator. #29790

kyle-winkelman commented Feb 27, 2023 •

edited by eladkal

boring-cyborg bot commented Feb 27, 2023

potiuk commented Mar 4, 2023

alexott commented Mar 4, 2023

kyle-winkelman commented Mar 4, 2023

alexott left a comment

alexott Mar 5, 2023

kyle-winkelman Mar 6, 2023

alexott Mar 6, 2023 •

edited

kyle-winkelman Mar 7, 2023

alexott Mar 7, 2023

kyle-winkelman Apr 4, 2023

alexott Apr 4, 2023

eladkal Apr 17, 2023

alexott Mar 5, 2023

potiuk left a comment

potiuk commented Apr 23, 2023

alexott commented Apr 23, 2023

potiuk commented Apr 23, 2023

lennartkats-db commented May 30, 2023

stikkireddy commented Jun 6, 2023

stikkireddy commented Jun 23, 2023

potiuk commented Jun 23, 2023

github-actions bot commented Aug 8, 2023

Provider Databricks add jobs create operator. #29790

Provider Databricks add jobs create operator. #29790

Conversation

kyle-winkelman commented Feb 27, 2023 • edited by eladkal

boring-cyborg bot commented Feb 27, 2023

potiuk commented Mar 4, 2023

alexott commented Mar 4, 2023

kyle-winkelman commented Mar 4, 2023

alexott left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexott Mar 6, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

potiuk left a comment

Choose a reason for hiding this comment

potiuk commented Apr 23, 2023

alexott commented Apr 23, 2023

potiuk commented Apr 23, 2023

lennartkats-db commented May 30, 2023

stikkireddy commented Jun 6, 2023

stikkireddy commented Jun 23, 2023

potiuk commented Jun 23, 2023

github-actions bot commented Aug 8, 2023

kyle-winkelman commented Feb 27, 2023 •

edited by eladkal

alexott Mar 6, 2023 •

edited