Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shorten upgrade workflow names, reorganize premixing workflow setup and add to PR tests for phase 2 #30988

Merged
merged 8 commits into from Aug 4, 2020

Conversation

kpedro88
Copy link
Contributor

@kpedro88 kpedro88 commented Jul 30, 2020

PR description:

The upgrade premixing workflow setup is reimplemented to use the special workflow approach. This reduces the number of hacks necessary (though a few are still needed). To facilitate this, PU steps are first added to the list and then specialized (this allows changing PU-related parameters).

The names of the premix workflows are adjusted slightly to try to keep the length down.

A new workflow is added for phase2 premixing using PU50 (rather than PU200, to reduce CPU/mem/disk usage). This will automatically generate PR comparison plots, which helps to test and validate PRs that impact Phase 2 premixing. The PU50 scenario used in this workflow only considers bunch crossings -3,3, as with the other phase 2 PU scenarios.

The upgrade workflow names are shortened by eliminating unnecessary words from step names (mostly 'Full'), and preventing the key and special workflow suffix (if any) from being repeated in the names. This also entails a small amount of reorganization in the order of information in the names (key and suffix now come first). The directory and display names were originally:

23461.999_NuGun+SingleNuE10_2026D49PU_GenSimHLBeamSpotFull_PMXS1S2PR+PREMIX_2026D49PU_PremixHLBeamSpotFullPU_PMXS1S2PR+DigiFullTriggerPU_PMXS1S2PR_2026D49PU+RecoFullGlobalPU_PMXS1S2PR_2026D49PU+HARVESTFullGlobalPU_PMXS1S2PR_2026D49PU
23461.999 SingleNuE10_2026D49PU_GenSimHLBeamSpotFull_PMXS1S2PR+PREMIX_2026D49PU_PremixHLBeamSpotFullPU_PMXS1S2PR+DigiFullTriggerPU_PMXS1S2PR_2026D49PU+RecoFullGlobalPU_PMXS1S2PR_2026D49PU+HARVESTFullGlobalPU_PMXS1S2PR_2026D49PU

and are now reduced to:

23461.999_NuGun+2026D49PU_PMXS1S2PR+SingleNuE10_GenSimHLBeamSpot+PREMIX_PremixHLBeamSpotPU+DigiTriggerPU+RecoGlobalPU+HARVESTGlobalPU
23461.999 2026D49PU_PMXS1S2PR+SingleNuE10_GenSimHLBeamSpot+PREMIX_PremixHLBeamSpotPU+DigiTriggerPU+RecoGlobalPU+HARVESTGlobalPU 

PR validation:

Compared the commands generated by runTheMatrix.py -w upgrade -l 23461.97,23461.98,23461.99 --dryRun before and after this PR to see that they contain the same flags. Also tested the commands for 23461.999.

@cmsbuild
Copy link
Contributor

The code-checks are being triggered in jenkins.

@cmsbuild
Copy link
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30988/17439

  • This PR adds an extra 16KB to repository

@cmsbuild
Copy link
Contributor

A new Pull Request was created by @kpedro88 (Kevin Pedro) for master.

It involves the following packages:

Configuration/PyReleaseValidation

@chayanit, @cmsbuild, @wajidalikhan, @pgunnell, @kpedro88 can you please review it and eventually sign? Thanks.
@makortel, @Martin-Grunewald, @fabiocos, @slomeo this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

@kpedro88
Copy link
Contributor Author

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Jul 30, 2020

The tests are being triggered in jenkins.

@cmsbuild
Copy link
Contributor

+1
Tested at: c3ef67e
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a15850/8440/summary.html
CMSSW: CMSSW_11_2_X_2020-07-30-1100
SCRAM_ARCH: slc7_amd64_gcc820

@cmsbuild
Copy link
Contributor

Comparison job queued.

@cmsbuild
Copy link
Contributor

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a15850/8440/summary.html

@slava77 comparisons for the following workflows were not done due to missing matrix map:

  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-a15850/23434.99_TTbar_14TeV+TTbar_14TeV_TuneCP5_2026D49PU_GenSimHLBeamSpotFull14+PREMIX_2026D49PU_PremixHLBeamSpotFull14PU_Premix+DigiFullTriggerPUPRMXCombined_2026D49PU+RecoFullGlobalPUPRMX_2026D49PU+HARVESTFullGlobalPU_2026D49PU

Comparison Summary:

  • No significant changes to the logs found
  • Reco comparison results: 4 differences found in the comparisons
  • DQMHistoTests: Total files compared: 34
  • DQMHistoTests: Total histograms compared: 2525444
  • DQMHistoTests: Total failures: 7
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 2525390
  • DQMHistoTests: Total skipped: 47
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 33 files compared)
  • Checked 144 log files, 17 edm output root files, 34 DQM output files

@kpedro88
Copy link
Contributor Author

+upgrade
@slava77 please add the matrix map

@chayanit
Copy link

+1

@cmsbuild
Copy link
Contributor

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

@cmsbuild
Copy link
Contributor

cmsbuild commented Aug 4, 2020

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-78e1d8/8534/summary.html

Comparison Summary:

  • No significant changes to the logs found
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 34
  • DQMHistoTests: Total histograms compared: 1387074
  • DQMHistoTests: Total failures: 1
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 1387034
  • DQMHistoTests: Total skipped: 39
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 25 files compared)
  • Checked 103 log files, 12 edm output root files, 34 DQM output files

@kpedro88
Copy link
Contributor Author

kpedro88 commented Aug 4, 2020

+upgrade

@slava77
Copy link
Contributor

slava77 commented Aug 4, 2020

Reco comparison results: 0 differences found in the comparisons

this is a bit misleading though: IIUC, all comparison jobs in the upgrade workflows, the ones that were changed by this PR (not just in reco, but also in relmon/DQM), have failed due to different directory names.

@kpedro88
Copy link
Contributor Author

kpedro88 commented Aug 4, 2020

Yes, maybe we need to add a line about "missing workflows"? In this case it's intentional, but in other cases it might not be...

@kpedro88
Copy link
Contributor Author

kpedro88 commented Aug 4, 2020

@chayanit, @wajidalikhan, @pgunnell I think this has finally converged, please take a look

@chayanit
Copy link

chayanit commented Aug 4, 2020

+1

@silviodonato
Copy link
Contributor

+operations

@cmsbuild
Copy link
Contributor

cmsbuild commented Aug 4, 2020

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

@silviodonato
Copy link
Contributor

+1

@cmsbuild cmsbuild merged commit 6fb37cd into cms-sw:master Aug 4, 2020
@Dr15Jones
Copy link
Contributor

I think this broke the nanoAOD steps in workflows 11834.99, 12634.99, and 13034.99.

https://cmssdt.cern.ch/SDT/html/cmssdt-ib/#/relVal/CMSSW_11_2/2020-08-04-2300?selectedArchs=cc8_amd64_gcc8&selectedFlavors=X&selectedStatus=failed

@kpedro88 kpedro88 mentioned this pull request Aug 5, 2020
@kpedro88
Copy link
Contributor Author

kpedro88 commented Aug 5, 2020

see #31060

mmusich added a commit to mmusich/cmssw that referenced this pull request Aug 7, 2020
mmusich added a commit to mmusich/cmssw that referenced this pull request Aug 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants