-
Notifications
You must be signed in to change notification settings - Fork 187
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Process
: properly cleanup when exception in state transition
#5697
Process
: properly cleanup when exception in state transition
#5697
Conversation
Note this is blocked because it requires a fix in |
2668c0c
to
c9e9240
Compare
193dded
to
d837c97
Compare
3086fb9
to
b5cda38
Compare
b5cda38
to
c9d6f0e
Compare
Anyone would like to review this? Otherwise I will merge this soon |
@sphuber I'll give it a look now. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@sphuber thanks! Only a minor request, but not very important.
self.node.set_process_state(state.LABEL) | ||
self.update_outputs() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not important, might be more clear move update_outputs
to on_entered
? I am quite surprised to realize that every state transfer will call update_outputs
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we can see if we can move this to something like on_terminate
so as to only do it once. Not sure if this would break anything but it should be more performant. Will have a look tomorrow.
If a process would hit an exception during a state transition, the node would not always be cleaned up properly. For example, when a process registers invalid outputs, the process would except in `update_outputs` which is called by `on_entered` which is called once the process has entered a new state. This exception would not be dealt with, skipping the code that is responsible for updating the process state on the node. This would lead to the node of the excepted node to still show the previous incorrect state. Typically, users would see processes as `Running` in the output of `verdi process list`. This is because that value is taken from the `process_state` attribute, but this simply contained the incorrect state. The solution requires a fix in `plumpy`, which was released with `v0.21.1` but it also requires an update in `update_node_state` of the `Process` class to make sure the `set_process_state` call is made before the `update_outputs` which is prone to excepting in the case of invalid outputs having been registered by the process implementation.
c9d6f0e
to
5b00a7d
Compare
Fixes #2861
Fixes #3334
If a process would hit an exception during a state transition, the node would not always be cleaned up properly. For example, when a process registers invalid outputs, the process would except in
update_outputs
which is called byon_entered
which is called once the process has entered a new state.This exception would not be dealt with, skipping the code that is responsible for updating the process state on the node. This would lead to the node of the excepted node to still show the previous incorrect state. Typically, users would see processes as
Running
in the output ofverdi process list
. This is because that value is taken from theprocess_state
attribute, but this simply contained the incorrect state.The solution requires a fix in
plumpy
, which was released withv0.21.
but it also requires an update inupdate_node_state
of theProcess
class to make sure theset_process_state
call is made before theupdate_outputs
which is prone to excepting in the case of invalid outputs having been registered by the process implementation.