Skip to content

WIP: addition of trino airflow scheduler#2188

Merged
eakmanrq merged 5 commits intoSQLMesh:mainfrom
SiddiqueAhmad:main
Feb 29, 2024
Merged

WIP: addition of trino airflow scheduler#2188
eakmanrq merged 5 commits intoSQLMesh:mainfrom
SiddiqueAhmad:main

Conversation

@SiddiqueAhmad
Copy link
Contributor

No description provided.

@CLAassistant
Copy link

CLAassistant commented Feb 28, 2024

CLA assistant check
All committers have signed the CLA.

## Airflow Scheduler
**Engine Name:** `trino`

The SQLMesh Trino Operator is similar to the [TrinoOperator](https://airflow.apache.org/docs/apache-airflow-providers-postgres/stable/_api/airflow/providers/postgres/operators/postgres/index.html), and relies on the same [TrinoHook](https://airflow.apache.org/docs/apache-airflow-providers-postgres/stable/_api/airflow/providers/postgres/hooks/postgres/index.html) implementation.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This section doc section has postgres links throughout instead of Trino (probably copy/pasted from postgres).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh you got the source too :), yes copied let me correct it

"trino",
default_catalog="<database name>",
engine_operator_args={
"tromp_conn_id": "<Connection ID>"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo: trino_conn_id

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sending fix

Copy link
Contributor

@eakmanrq eakmanrq left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for contributing this @SiddiqueAhmad! Were you able to test this and verify it works?

@SiddiqueAhmad
Copy link
Contributor Author

Thanks for contributing this @SiddiqueAhmad! Were you able to test this and verify it works?

Most welcome, and yes it is working and running in my local setup.

pip install "sqlmesh[trino]"
```

The operator requires an [Airflow connection](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html) to determine the target Trino account. Refer to [Trino connection](https://airflow.apache.org/docs/apache-airflow-providers-trino/stable/connections.html) for more details.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see you fixed this link but you still have the links above this that are linking to Postgres.

Copy link
Contributor

@eakmanrq eakmanrq left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @SiddiqueAhmad! Will fix minor issues with links in docs in follow up PR.

@eakmanrq eakmanrq merged commit b0a85d0 into SQLMesh:main Feb 29, 2024
@SiddiqueAhmad
Copy link
Contributor Author

I updated docs anyway thanks for merging and fixing too

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants