DominoSparkOperator #80
Conversation
|
@abhijeet2096 For my personal development environment I've been using poetry to manage package dependencies (also to pull in test dependencies like pytest), I can also push those files if you all are interested. |
change the template field to be a proper tuple
Add RunFailedException as a possible throwable from the polling function.
| @@ -377,6 +377,8 @@ def job_start_blocking(self, poll_freq: int = 5, max_poll_time: int = 6000, **kw | |||
| def get_job_status(job_identifier): | |||
| status = self.job_status(job_identifier) | |||
| self.log.info(f"Polling Job: {job_identifier} status is completed: {status['statuses']['isCompleted']}") | |||
| if status['statuses']['executionStatus'] == "Failed": | |||
There was a problem hiding this comment.
@abhijeet2096 throwing an exception here if the status is failed so it can be handled downstream by other functions, like the airflow operator. I noticed failures were happening silenty since we just check for job completion and
job_complete != job_success
| dag=dag, | ||
| task_id="foo", | ||
| project=TEST_PROJECT, | ||
| command="test_spark.py", |
There was a problem hiding this comment.
Hey @wseaton can we have python/bash commands instead of name of the test files test_spark.py, test_spark_fail.sh?
There was a problem hiding this comment.
Not here, since I can't provide a direct command via the v4 api, before I could do this via isDirect. I'd also like to not have to use wrapper scripts 🙂
There was a problem hiding this comment.
Hey @wseaton can you add these sample test files (test_spark.py, test_spark_fail.sh) here, in this folder?
There was a problem hiding this comment.
@abhijeet2096 this has been done
|
@abhijeet2096 readme has been updated |
Add new
DominoSparkOperatorthat supports the v4 job start function signature along with associated unit tests.