-
Notifications
You must be signed in to change notification settings - Fork 248
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[batch] Add always_copy_output option #11884
Conversation
I think the proposed new default and the option to change it is much more intuitive than the current behavior and worth a change. Though, I think it would be most polite to announce it on zulip/email list a week or two in advance of the release (which you may already planned to do). |
Another thing I just thought of, I think we should add raise some form of error if a job that is |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Needs rebase fix
I'm not sure how I feel about the warning / error suggestion when always run jobs have inputs that aren't always copied out. Cleanup jobs shouldn't care whether the outputs don't get copied out on failure. |
Do cleanup jobs take input files from the parent job? IIUC, if I have an always_run job that takes an input from a job that failed, and whose outputs were not written, the input container for the always_run job would fail. |
Hopefully the new warning gets at the case you were thinking of as well as a more general problem beyond |
hail/python/hailtop/batch/job.py
Outdated
|
||
if self._always_run: | ||
warnings.warn('A job marked as always run has a resource file dependency on another job. If the dependent job fails, ' | ||
f'the always run job with the following command will not run:\n{command}') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this should be the always run job with the following command may not succeed
. It will run, right?
Let's leave the WIP tag on, but this should be all set. |
**Version 0.2.96** | ||
|
||
- Added ``Job.always_copy_output`` when using the ``ServiceBackend``. The default behavior is `False` which | ||
is a breaking change from the previous behavior to always copy output files irregardless of the job's |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
irregardless => regardless
50ae84a
to
9d917ee
Compare
We cannot take off WIP and merge this until November 1st. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why are we making the server side change? If we only change the client and also always send the setting then we preserve the behavior of old versions of the hailtop.batch library while also switching to a different behavior.
Agreed I think this properly preserves behavior for old clients |
@danking This is assigned to me but since you're the last person to leave |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
CHANGE_LOG:
Job.always_copy_output
when using theServiceBackend
. The default behavior isFalse
whichis a breaking change from the previous behavior to always copy output files irregardless of the job's
completion state.
Thoughts on introducing this "breaking" change? I think it's okay and we will announce the change on Zulip. I think the benefits of getting rid of this not so intuitive behavior where we copy files despite the main container's completion state outweighs possibly breaking someone's pipeline that relied on files being there when the copy step could have failed as well.