-
Notifications
You must be signed in to change notification settings - Fork 14.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
New Tableau operator: TableauRefreshDatasourceOperator #16937
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks cool. Let's see the test pass but it looks good to me :)
The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease. |
|
||
job_id = self._refresh_datasource(tableau_hook, datasource.id) | ||
if self.blocking: | ||
from airflow.providers.tableau.sensors.tableau_job_status import TableauJobStatusSensor |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One small comment here though. It looks a bit strange to use sensor in operator. While I understand where it comes from, I would like to propose a small refactor. Why don't you extract the "poke()" internals as a separate Hook method wait_until_succeeded
and use that method in both Sensor and refresh_datasource Operator?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry commented with my work account rather than my personal one :)
so removed the previous comments
One small comment here though. It looks a bit strange to use sensor in operator
This is how it's done now in TableauRefreshWorkbookOperator
TableauJobStatusSensor( |
Also #16916 will replace the usage of the TableauJobStatusSensor
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I didn't notice that! The most generic implementation of #16915 is cooler than mine. I will go with that!
Anyway, I had already implemented the requested changes, in a similar way of #16916 , but modified both the sensor and the hook. I pushed the implementation in case you want to use it to replace the TableauJobStatusSensor
in the operators.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I looked closer and I proposed a small refactor to move waiting logic to hook.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I think though that #16915 going to solve this in a more generic manner as it will handle all options of resource methods. WDYT?
b08b6a3
to
0b8cc9a
Compare
This PR address several points:
Item 1-2 are independent from 3. Item 3 may not be needed if #16915 is merged. |
I agree with you, let's split 1-2 and then wait for #16915. If @potiuk agrees, I will split and create a new PR. |
Sure ! |
@pmalafosse I think that that issue is solved (or partially) with PR #16916 with the deprecation of the access token authentication. |
I think #16915 covers the functionality of this PR. The current PR is not needed anymore. |
Hello all,
I propose you a new Airflow Operator for Tableau:
TableauRefreshDatasourceOperator
. This new operator uses the Tableau API to refresh an existing Datasource published in a Tableau Server.The operator is very similar to
TableauRefreshWorkbookOperator
, but I thought it was more clear and clean to create a new one instead to modify the existing one.The operator is very useful to refresh data sources, not in time set way but in an asynchronous way, in my case, it is very useful to add the refresh after my ETL pipelines. I'm currently using this operator in my Airflow installation.
Changes:
Thank you.