Ensure aborted/force-complete workflows leave no elements behind #11113

amaltaro · 2022-04-25T18:32:34Z

Status

not-tested

Description

First commit contains a script to be used to cancel WQEs at the global workqueue level. It can be used like:

python3 kill-workflow-in-global <your_workflow_name>

Second commit contains changes to how aborted/force-complete workflows are dealt with in the ReqMgr2 CherryPy thread responsible for status transition. Including:

if such workflows contains available/active WQEs, then a cancelWorkflow request is triggered once again
global workqueue state can now return an aborted status, which in this context means that all the WQEs are sitting in CancelRequested state
don't (re-)set WQEs in CancelRequested to the same state

NOTE: the logic to avoid pulling work down for aborted workflows would demand a substantial amount of changes, so I will skip it and rely only on the cherrypy thread mechanism.

Is it backward compatible (if not, which system it affects?)

YES

Related PRs

None

External dependencies / deployment changes

None

cmsdmwmbot · 2022-04-25T18:48:26Z

Jenkins results:

Python3 Unit tests: succeeded
- 7 tests added
Python3 Pylint check: succeeded
Pylint py3k check: succeeded
Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13072/artifact/artifacts/PullRequestReport.html

cmsdmwmbot · 2022-04-25T20:12:43Z

Jenkins results:

Python3 Unit tests: failed
- 2 new failures
- 7 tests added
- 2 changes in unstable tests
Python3 Pylint check: succeeded
- 1 warnings
- 25 comments to review
Pylint py3k check: succeeded
Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13073/artifact/artifacts/PullRequestReport.html

cmsdmwmbot · 2022-04-25T21:04:19Z

Jenkins results:

Python3 Unit tests: succeeded
- 7 tests added
- 1 changes in unstable tests
Python3 Pylint check: succeeded
- 2 warnings
- 38 comments to review
Pylint py3k check: succeeded
Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13074/artifact/artifacts/PullRequestReport.html

cmsdmwmbot · 2022-04-25T21:20:01Z

Jenkins results:

Python3 Unit tests: succeeded
- 7 tests added
Python3 Pylint check: succeeded
- 2 warnings
- 38 comments to review
Pylint py3k check: succeeded
Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13076/artifact/artifacts/PullRequestReport.html

vkuznet

I find very confusing which naming convention style should be used for states. On one side you have all lower case states like, aborted, running-open, on another within the code you use Camel case, like Done, Running. I think I understand that lower case used in WQ DB records, while Camel case within the code, but such convention can lead to simple mistake when somewhere in a code someone will use different style.

Apart from that the changes are fine.

todor-ivanov

Thanks @amaltaro for providing those changes. I have left two minor comments inline. You may want to take a look. Other than that the code looks good to me.

todor-ivanov · 2022-04-26T20:32:44Z

src/python/WMCore/ReqMgr/DataStructs/RequestStatus.py


+# Workflow state transition automatically controlled by ReqMgr2
+# Specific to workflows either aborted or force-completed
+CANCEL_AUTO_TRANSITION = {"aborted": "aborted-completed",


Shouldn't those "allowed" status transitions (here aborted-completed) be enclosed in a list instead of pure strings. I'd say a list here would be better at least for staying consistent with the rest.

It could, but there is no real need to have a list because it's a single transition (instead of chained transition for some of the previous ones). This makes the source code slightly more clear as well, so I think it's a reasonable tradeoff.

todor-ivanov · 2022-04-26T20:48:55Z

src/python/WMCore/Services/WorkQueue/WorkQueue.py

@@ -37,11 +37,15 @@ def convertWQElementsStatusToWFStatus(elementsStatusSet):
    running = set(["Running"])
    runningOpen = set(["Available", "Negotiating", "Acquired"])
    runningClosed = set(["Running", "Done", "Canceled"])
+    aborted = set(["CancelRequested", "Done", "Canceled", "Failed"])


I am wondering if putting the CancelRequested status under the set of statuses covered by the aborted does not make any difference in the note from above:

NOTE: CancelRequested status is a transient status and it should not trigger any request status transition (thus, None gets returned).

I mean it could now trigger a request status transition (IINM) because of line 48 from bellow, which says:

if elementsStatusSet == forceCompleted: # all WQEs in CancelRequested return "aborted"

Good point! I definitely need to update the docstring.

This aborted state here should not trigger any status transition, as implemented:
https://github.com/dmwm/WMCore/pull/11113/files#diff-b42637227c688d15bfc68058c1005999390006897e0abc07b1a8b6989039afb1R18
and
https://github.com/dmwm/WMCore/pull/11113/files#diff-b42637227c688d15bfc68058c1005999390006897e0abc07b1a8b6989039afb1R62

but I had to implement it to be able to differentiate options that were returning a None value.

todor-ivanov · 2022-04-26T21:00:04Z

bin/kill-workflow-in-global

+
+    # get configuration file path
+    if "WMAGENT_CONFIG" not in os.environ:
+        os.environ['WMAGENT_CONFIG'] = '/data/srv/wmagent/current/config/wmagent/config.py'


Well this is just a comment here, no need to be changed with this PR (because it is in bin and supposedly the script should be used for manual workflow cancellation). But in any case, I think we should avoid hard coding a global path inside those scripts. We should rather stick to relative paths starting from the root of the deployment destination, which may change in the future (e.g if/once we move the wmagent deployment procedure under a virtual environment with the freedom to deploy anywhere but not just at /data/srv)

The WMAgent ROOT directory would have to be defined within the WMAgent/WMCore setup as well, and I think if WMAGENT_CONFIG isn't set, changes that the ROOT_DIR (or whatever name that is) would not be set as well.

This is a fallback though, so it should be good enough until we come back and update all of these bin/ scripts (I am pretty sure we have a few with such hard-coded path). Please keep it in mind as we get closer to the virtual environment setup ;)

Please keep it in mind as we get closer to the virtual environment setup ;)

Indeed, this was the reason I mentioned it. No change needed right now.

amaltaro · 2022-04-26T23:22:00Z

@vkuznet yes, I completely second your comment. As you noted, we have many structs/objects in WMCore and they have a different set of status (sometimes called states) and some of those were implemented following a different convention.

Yes, WorkQueue element properties and status are all upper CamelCase. This code is basically deciding what status should the request be set, according to its aggregation of workqueue element statuses.

Todor, Valentin, the aborted word there is probably not the best choice to be honest, since it covers both aborted and force-completed workflows. I'm going to rename it to canceled instead (even though that will require changes everywhere in my changes, sorry!).

cmsdmwmbot · 2022-04-27T00:02:27Z

Jenkins results:

Python3 Unit tests: succeeded
- 7 tests added
- 3 changes in unstable tests
Python3 Pylint check: succeeded
- 2 warnings
- 38 comments to review
Pylint py3k check: succeeded
Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13084/artifact/artifacts/PullRequestReport.html

todor-ivanov

Thanks for the follow up @amaltaro
the PR looks good to me.

… workflows CancelRequested plus final state should be considered as aborted renamed constant for work cancelation rename workqueue aggregate status from aborted to canceled

update unit tests

amaltaro · 2022-04-27T17:24:16Z

Thanks for the re-review Todor. This needs to be further tested, but instead of running the whole deployment/workflow in my VMs, I will hand this over to Imran for a testbed deployment tomorrow.

cmsdmwmbot · 2022-04-27T17:34:19Z

Jenkins results:

Python3 Unit tests: succeeded
- 7 tests added
- 1 changes in unstable tests
Python3 Pylint check: succeeded
- 2 warnings
- 38 comments to review
Pylint py3k check: succeeded
Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13090/artifact/artifacts/PullRequestReport.html

amaltaro added PR: Do not merge yet PR: Work in progress labels Apr 25, 2022

amaltaro changed the title ~~Create script to cancel GQEs~~ Ensure aborted/force-complete workflows leave no elements behind Apr 25, 2022

amaltaro force-pushed the fix-11112 branch from 4217d1b to 887029f Compare April 25, 2022 20:53

amaltaro added PR: Further tests needed and removed PR: Work in progress labels Apr 25, 2022

amaltaro requested review from todor-ivanov and vkuznet April 25, 2022 21:07

amaltaro force-pushed the fix-11112 branch from 887029f to 45bef51 Compare April 25, 2022 21:10

vkuznet approved these changes Apr 26, 2022

View reviewed changes

todor-ivanov approved these changes Apr 26, 2022

View reviewed changes

amaltaro added the PR: squashing needed label Apr 27, 2022

todor-ivanov approved these changes Apr 27, 2022

View reviewed changes

amaltaro removed PR: Do not merge yet PR: squashing needed labels Apr 27, 2022

amaltaro added 3 commits April 27, 2022 13:21

Create script to cancel GQEs

766d9f5

Update ReqMgr2 CP thread to properly deal with aborted/force-complete…

6ef58dc

… workflows CancelRequested plus final state should be considered as aborted renamed constant for work cancelation rename workqueue aggregate status from aborted to canceled

fix unit tests

ebaf3eb

update unit tests

amaltaro force-pushed the fix-11112 branch from 69c78ae to ebaf3eb Compare April 27, 2022 17:21

amaltaro self-assigned this Apr 27, 2022

amaltaro merged commit f8478de into dmwm:master Apr 27, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ensure aborted/force-complete workflows leave no elements behind #11113

Ensure aborted/force-complete workflows leave no elements behind #11113

amaltaro commented Apr 25, 2022 •

edited

Loading

cmsdmwmbot commented Apr 25, 2022

cmsdmwmbot commented Apr 25, 2022

cmsdmwmbot commented Apr 25, 2022

cmsdmwmbot commented Apr 25, 2022

vkuznet left a comment

todor-ivanov left a comment

todor-ivanov Apr 26, 2022

amaltaro Apr 26, 2022

todor-ivanov Apr 26, 2022

amaltaro Apr 26, 2022

todor-ivanov Apr 26, 2022

amaltaro Apr 26, 2022

todor-ivanov Apr 27, 2022 •

edited

Loading

amaltaro commented Apr 26, 2022

cmsdmwmbot commented Apr 27, 2022

todor-ivanov left a comment

amaltaro commented Apr 27, 2022

cmsdmwmbot commented Apr 27, 2022

Ensure aborted/force-complete workflows leave no elements behind #11113

Ensure aborted/force-complete workflows leave no elements behind #11113

Conversation

amaltaro commented Apr 25, 2022 • edited Loading

Status

Description

Is it backward compatible (if not, which system it affects?)

Related PRs

External dependencies / deployment changes

cmsdmwmbot commented Apr 25, 2022

cmsdmwmbot commented Apr 25, 2022

cmsdmwmbot commented Apr 25, 2022

cmsdmwmbot commented Apr 25, 2022

vkuznet left a comment

Choose a reason for hiding this comment

todor-ivanov left a comment

Choose a reason for hiding this comment

todor-ivanov Apr 26, 2022

Choose a reason for hiding this comment

amaltaro Apr 26, 2022

Choose a reason for hiding this comment

todor-ivanov Apr 26, 2022

Choose a reason for hiding this comment

amaltaro Apr 26, 2022

Choose a reason for hiding this comment

todor-ivanov Apr 26, 2022

Choose a reason for hiding this comment

amaltaro Apr 26, 2022

Choose a reason for hiding this comment

todor-ivanov Apr 27, 2022 • edited Loading

Choose a reason for hiding this comment

amaltaro commented Apr 26, 2022

cmsdmwmbot commented Apr 27, 2022

todor-ivanov left a comment

Choose a reason for hiding this comment

amaltaro commented Apr 27, 2022

cmsdmwmbot commented Apr 27, 2022

amaltaro commented Apr 25, 2022 •

edited

Loading

todor-ivanov Apr 27, 2022 •

edited

Loading