Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] SQL Job Task #119

Closed
1 of 4 tasks
RajuGujjalapati opened this issue May 23, 2024 · 0 comments
Closed
1 of 4 tasks

[FEATURE] SQL Job Task #119

RajuGujjalapati opened this issue May 23, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@RajuGujjalapati
Copy link
Contributor

Is your feature request related to a problem? Please describe.
Databricks now supports creating a "SQL" task, which can allowing you to create, schedule, operate, and monitor workflows that include Databricks SQL objects such as queries, legacy dashboards, sql files and alerts.. It would be nice to have this feature in BrickFlow.

Cloud Information

  • AWS
  • Azure
  • GCP
  • Other

Describe the solution you'd like
SQL task has 4 other sub tasks will be implementing all of them, also will add a new sql_type type. It would be nice to have something like this:

# For SQL query
@wf.sql_task
def sample_sql_task_query() -> any:
    return SqlTask(
        query_id="4e16dc24-e30d-4683-96d7-cf7da4e263ad", warehouse_id="044a6f42ad7d914a"
    )

# For File
@wf.sql_task
def sample_sql_task_file() -> any:
    return SqlTask(file_path="products/brickflow_test/src/sql/sql_task_file_test.sql", warehouse_id="044a6f42ad7d914a")

# For alerts
@wf.sql_task
def sample_sql_alert() -> any:
    return SqlTask(
        alert_id="41ca5e33-21c2-40a2-8f77-183351d2b566",
        pause_subscriptions=False,
        subscriptions={
            "usernames": ["raju.gujjalapati@nike.com", "Mohanasilpa.Palla@nike.com"]
        },
        warehouse_id="044a6f42ad7d914a",
    )

# For Dashboards
@wf.sql_task
def sample_sql_dashboard():
    return SqlTask(
        dashboard_id="f57447ca-e8d4-4dad-a66c-f524464e52a8",
        dashboard_custom_subject="Raju Legacy Dashboard Test",
        pause_subscriptions=True,
        subscriptions={
            "usernames": ["rabc@abc.com", "abc.cd@abc.com"],
            "destination_id": ["434354545"],
        },
        warehouse_id="044a6fad7d914a",
    )

Describe alternatives you've considered
I've looked at invoking dependent jobs with the Databricks API

Additional context
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants