Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add campaign names support for stepchain workflows #11738

Merged
merged 2 commits into from
Sep 25, 2023
Merged

Conversation

khurtado
Copy link
Contributor

@khurtado khurtado commented Sep 22, 2023

Fixes #11706

Status

Ready

Description

For each step, look for the campaign attribute in the workflow description and append this information in the campaign name at the task level. All step campaign names are appended to the task level campaign name with a comma. When there is no campaign attribute for a campaign, this defaults to the workload base campaign name. At the end, all unique campaigns are reported, this means if all steps have the same campaign name, this will be reported only once.

Is it backward compatible (if not, which system it affects?)

YES

Related PRs

<If it's a follow up work; or porting a fix from a different branch, please mention them here.>

External dependencies / deployment changes

<Does it require deployment changes? Does it rely on third-party libraries?>

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 23 new failures
    • 2 tests no longer failing
    • 1 changes in unstable tests
  • Python3 Pylint check: failed
    • 12 warnings and errors that must be fixed
    • 2 warnings
    • 34 comments to review
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded
    • 5 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14497/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 1 new failures
    • 1 tests no longer failing
    • 1 changes in unstable tests
  • Python3 Pylint check: failed
    • 10 warnings and errors that must be fixed
    • 2 warnings
    • 34 comments to review
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded
    • 5 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14498/artifact/artifacts/PullRequestReport.html

@khurtado
Copy link
Contributor Author

Here are my test cases:

Structure:
workload Campaign = DMWM_Test
Step1 Campaign = CMSSW_12_4_0_pre2__fullsim_noPU_2021_14TeV-1649399635Step1
Step2 Campaign = CMSSW_12_4_0_pre2__fullsim_noPU_2021_14TeV-1649399635Step2
Step3 Campaign = CMSSW_12_4_0_pre2__fullsim_noPU_2021_14TeV-1649399635Step3

Workload campaign = yes
Step 1 campaign= yes
Step 2 campaign= no
Step 3 campaign= yes

Result:
campaignName = 'CMSSW_12_4_0_pre2__fullsim_noPU_2021_14TeV-1649399635Step1,DMWM_Test,CMSSW_12_4_0_pre2__fullsim_noPU_2021_14TeV-1649399635Step3'

Workload campaign = yes
Step 1 campaign= no
Step 2 campaign= no
Step 3 campaign= no

Result:
campaignName = 'DMWM_Test'

Workload campaign = no
Step 1 campaign= no
Step 2 campaign= no
Step 3 campaign= no

Result:
campaignName = 'DEFAULT_AcqEra'

Result:
campaignName = 'CMSSW_12_4_0_pre2__fullsim_noPU_2021_14TeV-1649399635Step1,CMSSW_12_4_0_pre2__fullsim_noPU_2021_14TeV-1649399635Step2,CMSSW_12_4_0_pre2__fullsim_noPU_2021_14TeV-1649399635Step3'

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 1 new failures
    • 2 tests no longer failing
    • 1 changes in unstable tests
  • Python3 Pylint check: failed
    • 10 warnings and errors that must be fixed
    • 2 warnings
    • 34 comments to review
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded
    • 5 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14499/artifact/artifacts/PullRequestReport.html

@khurtado
Copy link
Contributor Author

@amaltaro This is ready for review.

Copy link
Contributor

@amaltaro amaltaro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@khurtado these changes are looking good to me.
Can you please cover this feature in a couple StepChain unit tests as well:
https://github.com/dmwm/WMCore/blob/master/test/python/WMCore_t/WMSpec_t/StdSpecs_t/StepChain_t.py
?

Side note, when creating a pull request (or issue), please update the fields/placeholders accordingly. The initial PR description needs to be revisited.

@khurtado
Copy link
Contributor Author

@amaltaro Yes! I will add the unit tests

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 2 new failures
    • 1 tests no longer failing
    • 1 changes in unstable tests
  • Python3 Pylint check: failed
    • 10 warnings and errors that must be fixed
    • 2 warnings
    • 35 comments to review
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded
    • 6 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14500/artifact/artifacts/PullRequestReport.html

@khurtado
Copy link
Contributor Author

@amaltaro The unit tests have been added.

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: succeeded
    • 1 tests no longer failing
    • 4 tests added
    • 3 changes in unstable tests
  • Python3 Pylint check: failed
    • 29 warnings and errors that must be fixed
    • 5 warnings
    • 114 comments to review
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded
    • 52 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14501/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 1 new failures
    • 1 tests no longer failing
    • 4 tests added
    • 1 changes in unstable tests
  • Python3 Pylint check: failed
    • 29 warnings and errors that must be fixed
    • 5 warnings
    • 114 comments to review
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded
    • 52 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14502/artifact/artifacts/PullRequestReport.html

Copy link
Contributor

@amaltaro amaltaro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And now this feature is well covered. Thank you for providing those unit tests, Kenyi.

@amaltaro amaltaro merged commit b937b1e into dmwm:master Sep 25, 2023
2 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support campaign name for StepChain workflows
3 participants