Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nano test in runthematrix #39472

Merged
merged 21 commits into from Oct 14, 2022
Merged

Conversation

vlimant
Copy link
Contributor

@vlimant vlimant commented Sep 21, 2022

PR description:

as wished in #39452 the first step is to integrate some workflows in runTheMatrix. Here is a first stab at this, knowing that it's missing a couple of workflows, and that some will go away soon too.

PR validation:

runTheMatrix.py --sites '' --das-options '--limit 5' --what nano

did not run to full completion because some input datasets are deprecated, and some are not present on disk anymore.

EDIT:
post-facto, the behaviour of the MatrixReader had to be slightly changed to take properly into account run:lmi ranges. Thus far only fist three were considered and inconsistently if more than 3 were provided.
we found out existing nano workflows 136.7722, 136.7952, 136.8521, 136.8522, 136.8523, 1325.6, 1325.7, 1325.8, 1325.61, 1325.81, 1329.1 that are superseded by those put in place by this PR. we removed them and swapped 1325.81 for 2500.601

@cmsbuild
Copy link
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39472/32202

  • This PR adds an extra 20KB to repository

@cmsbuild
Copy link
Contributor

A new Pull Request was created by @vlimant (vlimant) for master.

It involves the following packages:

  • Configuration/PyReleaseValidation (pdmv, upgrade)

@bbilin, @cmsbuild, @AdrianoDee, @srimanob, @kskovpen, @sunilUIET can you please review it and eventually sign? Thanks.
@makortel, @kpedro88, @fabiocos, @Martin-Grunewald, @missirol, @trtomei, @beaucero, @slomeo this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

@@ -87,7 +89,8 @@ def reset(self, what='all'):
'relval_2026':True,
'relval_identity':False,
'relval_machine':True,
'relval_premix':True
'relval_premix':True,
'relval_nano':False
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vlimant , I would suggest to enable it for default runTheMatrix . This will allow to run these workflows for IBs. This way we make sure that relvals are working and automatically cache the input used by these wfs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is partially heavy to run all these, are you sure @smuzaffar that we want to enable them by default ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vlimant , for IBs, yes we should enable these all to make sure that they all work.
For PR tests, we can select a few of these for which bot should run the comparisons.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in 9806eb4

@cmsbuild
Copy link
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39472/32219

  • This PR adds an extra 12KB to repository

@cmsbuild
Copy link
Contributor

Pull request #39472 was updated. @bbilin, @cmsbuild, @AdrianoDee, @srimanob, @kskovpen, @sunilUIET can you please check and sign again.

@vlimant
Copy link
Contributor Author

vlimant commented Sep 22, 2022

to be noted : 91a8801

@cmsbuild
Copy link
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39472/32220

  • This PR adds an extra 16KB to repository

@cmsbuild
Copy link
Contributor

Pull request #39472 was updated. @bbilin, @cmsbuild, @AdrianoDee, @srimanob, @kskovpen, @sunilUIET can you please check and sign again.

@smuzaffar
Copy link
Contributor

smuzaffar commented Sep 26, 2022

test parameters:

  • workflows = 2500.0,2500.001,2500.1,2500.101,2500.11,2500.12,2500.2,2500.21,2500.3,2500.301,2500.302,2500.31,2500.311,2500.312,2500.32,2500.321,2500.322,2500.33,2500.331,2500.332,2500.4,2500.401,2500.6,2500.601
  • relvals_opt = -w nano

@smuzaffar
Copy link
Contributor

please test

@cmsbuild
Copy link
Contributor

-1

Failed Tests: RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1debec/27768/summary.html
COMMIT: 612849c
CMSSW: CMSSW_12_6_X_2022-09-25-2300/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/39472/27768/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

  • 2500.0012500.001_data80X+MuonEG2016HMINIAOD8.0+NANO_data8.0/step2_data80X+MuonEG2016HMINIAOD8.0+NANO_data8.0.log
  • 2500.122500.12_data94X2016+MuonEG2016MINIAOD9.4v2+NANO_2016data9.4v2/step2_data94X2016+MuonEG2016MINIAOD9.4v2+NANO_2016data9.4v2.log
  • 2500.52500.5_mc124Xrun3_v10+TTbarMINIAOD12.4+NANO_mc12.4_v10/step2_mc124Xrun3_v10+TTbarMINIAOD12.4+NANO_mc12.4_v10.log
Expand to see more relval errors ...

@vlimant
Copy link
Contributor Author

vlimant commented Oct 10, 2022

thanks @kskovpen and apologies, please re-sign

@kskovpen
Copy link
Contributor

+pdmv

@cmsbuild
Copy link
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1debec/28156/summary.html
COMMIT: 7bf5fad
CMSSW: CMSSW_12_6_X_2022-10-09-2300/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/39472/28156/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-1debec/2500.001_data80X+MuonEG2016HMINIAOD8.0+NANO_data8.0+HRV_NANO_data
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-1debec/2500.0_mc80X+TTbarMINIAOD8.0+NANO_mc8.0+HRV_NANO_mc
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-1debec/2500.101_mc94Xv2+TTbarMINIAOD9.4v2+NANO_mc9.4v2+HRV_NANO_mc
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-1debec/2500.11_data94X2016+MuonEG2016MINIAOD9.4v2+NANO_2016data9.4v2+HRV_NANO_data
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-1debec/2500.12_data94Xv2+MuonEG2017MINIAOD9.4v2+NANO_data9.4v2+HRV_NANO_data
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-1debec/2500.1_mc94X2016+TTbar2016MINIAOD9.4v2+NANO_2016mc9.4v2+HRV_NANO_mc
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-1debec/2500.21_data102X+MuonEG2018MINIAOD10.2+NANO_data10.2+HRV_NANO_data
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-1debec/2500.2_mc102X+TTbarMINIAOD10.2+NANO_mc10.2+HRV_NANO_mc
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-1debec/2500.301_mc106Xul17+TTbarMINIAOD10.6_UL17v1+NANO_mc10.6ul17v1+HRV_NANO_mc
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-1debec/2500.302_mc106Xul18+TTbarMINIAOD10.6_UL18v1+NANO_mc10.6ul18v1+HRV_NANO_mc
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-1debec/2500.311_mc106Xul17v2+TTbarMINIAOD10.6_UL17v2+NANO_mc10.6ul17v2+HRV_NANO_mc
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-1debec/2500.312_mc106Xul17v2+TTbarMINIAOD10.6_UL18v2+NANO_mc10.6ul18v2+HRV_NANO_mc
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-1debec/2500.31_mc106Xul17v2+TTbarMINIAOD10.6_UL16v2+NANO_mc10.6ul16v2+HRV_NANO_mc
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-1debec/2500.321_data106Xul17+MuonEG2017MINIAOD10.6v1+NANO_data10.6ul17v1+HRV_NANO_data
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-1debec/2500.322_data106Xul18+MuonEG2018MINIAOD10.6v1+NANO_data10.6ul18v1+HRV_NANO_data
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-1debec/2500.32_data106Xul16+MuonEG2016MINIAOD10.6v1+NANO_data10.6ul16v1+HRV_NANO_data
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-1debec/2500.331_data106Xul17v2+MuonEG2017MINIAOD10.6v2+NANO_data10.6ul17v2+HRV_NANO_data
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-1debec/2500.332_data106Xul18v2+MuonEG2018MINIAOD10.6v2+NANO_data10.6ul18v2+HRV_NANO_data
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-1debec/2500.33_data106Xul16v2+MuonEG2016MINIAOD10.6v2+NANO_data10.6ul16v2+HRV_NANO_data
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-1debec/2500.3_mc106Xul16+TTbarMINIAOD10.6_UL16v1+NANO_mc10.6ul16v1+HRV_NANO_mc
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-1debec/2500.401_mc122Xrun3+TTbarMINIAOD12.2+NANO_mc12.2+HRV_NANO_mc
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-1debec/2500.4_mc122Xrun3_v10+TTbarMINIAOD12.2+NANO_mc12.2_v10+HRV_NANO_mc
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-1debec/2500.601_mc126X+TTBarMINIAOD12.6+NANO_mc12.6+HRV_NANO_mc
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-1debec/2500.6_mc126X_v10+TTBarMINIAOD12.6+NANO_mc12.6_v10+HRV_NANO_mc
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-1debec/41834.0_TTbar_14TeV+2026D94+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 1 differences found in the comparisons
  • DQMHistoTests: Total files compared: 47
  • DQMHistoTests: Total histograms compared: 3301534
  • DQMHistoTests: Total failures: 3
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3301511
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 45 files compared)
  • Checked 194 log files, 43 edm output root files, 47 DQM output files
  • TriggerResults: no differences found

@vlimant
Copy link
Contributor Author

vlimant commented Oct 11, 2022

@cms-sw/upgrade-l2 please sign or comment further.

@srimanob
Copy link
Contributor

+Upgrade

@cmsbuild
Copy link
Contributor

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

@perrotta
Copy link
Contributor

@vlimant scrolling through the thread I understand that this is not a Work In Progress any more, and it can be merged in master. If so, could you please update the title and remove "[WIP]"? (Or I can do if you confirm so)

@vlimant vlimant changed the title Nano test in runthematrix [WIP] Nano test in runthematrix Oct 14, 2022
@vlimant
Copy link
Contributor Author

vlimant commented Oct 14, 2022

I edited the title accordingly, thanks

@perrotta
Copy link
Contributor

+1

@cmsbuild cmsbuild merged commit 6ba5904 into cms-sw:master Oct 14, 2022
cramonal pushed a commit to cramonal/cmssw that referenced this pull request Nov 14, 2022
cramonal pushed a commit to cramonal/cmssw that referenced this pull request Nov 14, 2022
@vlimant vlimant deleted the nano_test_runthematrix branch November 22, 2022 10:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants