Feature description
Add an dbt task to check until job completed given a jobId.
See discussion here
This would help reduce risks of failures / task resubmit generating an extra job when running long syncs.
Details:
Currently if Kestra worker fails at fetching the status when wait=True on this current plugin resubmitted task will trigger a new job instead of using the existing one
-
Flow will first trigger in a dedicated task, then “wait for status” will be done in a dedicated task.
If “wait for status” task fails (worker is dead / killed / pod restarted, this task can last for 10h+ so kind of likely) => task is resubmitted with the jobId to fetch for
-
No need for pod restart (due to k8s node upgrades for example) supervision
I am willing to do a PR 👍