-
Notifications
You must be signed in to change notification settings - Fork 92
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Task communication failures should not cause a task to fail #115
Comments
Note 2: One way to do this is to have each submitted task to keep a status file at, say,
If any previous communications between the submitted task and the suite failed, the suite would be able to recover the above information from the submitted task's status file. |
Dave - agreed entirely. The current situation basically conforms to my default first-cut treatment for any feature - if something goes wrong then abort, and put in more nuanced behaviour later as required. |
I guess we'll need some kind of polling mechanism (perhaps checking for Matt's .status file) in case the "cylc task succeeded" call fails to get through at all. And if that fails, assume the task failed or use a new "unknown" task state? Note ideas on polling in Issue #67 |
The full solution, including status files and polling, could be quite involved. But the intermediate solution - don't abort tasks if messaging calls fail, and assume all internal outputs complete if a task reports success without first reporting the internal outputs complete - is relatively easy. |
See also #86. |
See also #114. |
Job status put in place by #282. |
status update
To simplify things, I suggest we close this issue once #364 is merged and put use of task status files by cylc in a new issue: #381 |
Closing this as per previous comment. |
Currently tasks fail if any "cylc task" commands fail. However communication failures in a task should only ever be a warning. After all, they don't mean that the task itself has failed, only that it hasn't managed to report its status back to the server. Consider if your suite has died for some reason. In most situations it's preferable that any tasks that have been submitted should run
through to completion. For example, if you've got some large job submitted which has to queue for ages in order to get the required resources then you're not going to want it to fail if the "cylc task started" command fails.
The text was updated successfully, but these errors were encountered: