Add Databricks sensor check partition and delta table events#23567
Add Databricks sensor check partition and delta table events#23567thcidale0808 wants to merge 5 commits intoapache:mainfrom
Conversation
|
Failing static checks, please run the pre-commit tests and fix those. |
|
@ferruzzi your comments make sense. I've pushed the fixes. |
|
Nice. You'll need a committer's approval to merge it, but looks good to me. 👍 |
alexott
left a comment
alexott
left a comment
There was a problem hiding this comment.
You also need to update provider.yaml to register sensors, plus add documentation
| databricks_conn_id: str, | ||
| table_name: str, | ||
| partition_name: str, | ||
| database_name: Optional[str] = 'default', |
There was a problem hiding this comment.
In the Base Sensor you use catalog & schema, and here you use only database (schema). Please unify that
| self.timestamp = timestamp | ||
| self.database_name = 'default' if not database_name else database_name | ||
|
|
||
| def poke(self, context: Context) -> bool: |
There was a problem hiding this comment.
I would generalize this into the DatabricksSqlSensor and just pass different conditions
| return self.partition_name in record | ||
|
|
||
|
|
||
| class DatabricksDeltaTableChangeSensor(DatabricksBaseSensor): |
There was a problem hiding this comment.
Also, It would be useful to keep last "known" version, and compare it with the current - like I did for Delta sharing: https://github.com/apache/airflow/pull/22692/files#diff-219de7033c7aa02bc63b2bb0b8f6441f2d076bc48f52af129ed70d2e132b6944R98
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. |
github-actions
bot
added
the
stale
3 participants
This PR propose the creation of two new sensors within the Databricks provider package:
Both sensors leverage the existing DatabricksSQLHook to query the information from Databricks.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragement file, named
{pr_number}.significant.rst, in newsfragments.