[batch] Add always_copy_output option - #11884
Conversation
|
I think the proposed new default and the option to change it is much more intuitive than the current behavior and worth a change. Though, I think it would be most polite to announce it on zulip/email list a week or two in advance of the release (which you may already planned to do). |
|
Another thing I just thought of, I think we should add raise some form of error if a job that is |
daniel-goldstein
left a comment
daniel-goldstein
left a comment
There was a problem hiding this comment.
Needs rebase fix
|
I'm not sure how I feel about the warning / error suggestion when always run jobs have inputs that aren't always copied out. Cleanup jobs shouldn't care whether the outputs don't get copied out on failure. |
|
Do cleanup jobs take input files from the parent job? IIUC, if I have an always_run job that takes an input from a job that failed, and whose outputs were not written, the input container for the always_run job would fail. |
|
Hopefully the new warning gets at the case you were thinking of as well as a more general problem beyond |
|
|
||
| if self._always_run: | ||
| warnings.warn('A job marked as always run has a resource file dependency on another job. If the dependent job fails, ' | ||
| f'the always run job with the following command will not run:\n{command}') |
There was a problem hiding this comment.
I think this should be the always run job with the following command may not succeed. It will run, right?
|
Let's leave the WIP tag on, but this should be all set. |
| **Version 0.2.96** | ||
|
|
||
| - Added ``Job.always_copy_output`` when using the ``ServiceBackend``. The default behavior is `False` which | ||
| is a breaking change from the previous behavior to always copy output files irregardless of the job's |
50ae84a to
9d917ee
Compare
|
We cannot take off WIP and merge this until November 1st. |
danking
left a comment
danking
left a comment
There was a problem hiding this comment.
Why are we making the server side change? If we only change the client and also always send the setting then we preserve the behavior of old versions of the hailtop.batch library while also switching to a different behavior.
|
Agreed I think this properly preserves behavior for old clients |
|
@danking This is assigned to me but since you're the last person to leave |
3 participants
CHANGE_LOG:
Job.always_copy_outputwhen using theServiceBackend. The default behavior isFalsewhichis a breaking change from the previous behavior to always copy output files irregardless of the job's
completion state.
Thoughts on introducing this "breaking" change? I think it's okay and we will announce the change on Zulip. I think the benefits of getting rid of this not so intuitive behavior where we copy files despite the main container's completion state outweighs possibly breaking someone's pipeline that relied on files being there when the copy step could have failed as well.