Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using explicit lazy parsing & concrete data types in SimpleFlatTableProducer #44782

Merged
merged 3 commits into from
Apr 24, 2024

Conversation

hqucms
Copy link
Contributor

@hqucms hqucms commented Apr 19, 2024

PR description:

Building on the implementation by @Dr15Jones in #44575 and some discussions there, I modified the usage of SimpleFlatTableProducer to read concrete data types instead of the base reco::Candidate. This avoids the large number of lazyEval = True previously needed in #44575.

PR validation:

Tested for all NANO workflows:

runTheMatrix.py --ibeos --nEvents 100 -w nano -i all -l 2500.0,2500.001,2500.002,2500.01,2500.011,2500.012,2500.1,2500.2,2500.21,2500.211,2500.3,2500.301,2500.31,2500.311,2500.312,2500.313,2500.32,2500.321,2500.322,2500.323,2500.324,2500.325,2500.326,2500.327,2500.4,2500.401,2500.402,2500.403,2500.404,2500.405,2500.5,2500.51

Dr15Jones and others added 2 commits April 16, 2024 10:06
The T0 had a failure from the string parser which we have previously
attributed to using the lazy option of the string parser.
Added option to turn on lazy parsing from the configuration. This
is an 'opt' in since lazy parsing can be problematic so best to
explicitly state it must be used.
@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 19, 2024

cms-bot internal usage

@cmsbuild
Copy link
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44782/40016

@cmsbuild
Copy link
Contributor

A new Pull Request was created by @hqucms for master.

It involves the following packages:

  • CalibTracker/SiStripCommon (alca)
  • DPGAnalysis/MuonTools (xpog)
  • DataFormats/Candidate (reconstruction)
  • PhysicsTools/NanoAOD (xpog)

@vlimant, @cmsbuild, @hqucms, @saumyaphor4252, @perrotta, @jfernan2, @mandrenguyen, @consuegs can you please review it and eventually sign? Thanks.
@rsreds, @battibass, @robervalwalsh, @tocheng, @missirol, @AnnikaStein, @gpetruc, @yuanchao, @gbenelli, @rovere, @mmusich, @echabert this is something you requested to watch as well.
@rappoccio, @antoniovilela, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

@hqucms
Copy link
Contributor Author

hqucms commented Apr 19, 2024

enable nano

@hqucms
Copy link
Contributor Author

hqucms commented Apr 19, 2024

please test

@cmsbuild
Copy link
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2ec77b/38946/summary.html
COMMIT: 66624e6
CMSSW: CMSSW_14_1_X_2024-04-18-2300/el8_amd64_gcc12
Additional Tests: NANO
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44782/38946/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

NANO Comparison Summary

Summary:

  • You potentially added 4 lines to the logs
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 15
  • DQMHistoTests: Total histograms compared: 16792
  • DQMHistoTests: Total failures: 0
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 16792
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 14 files compared)
  • Checked 55 log files, 32 edm output root files, 15 DQM output files

Nano size comparison Summary:

Sample kb/ev ref kb/ev diff kb/ev ev/s/thd ref ev/s/thd diff rate mem/thd ref mem/thd
2500.0 2.774 2.774 0.000 ( +0.0% ) 3.57 3.51 +1.7% 2.259 2.310
2500.001 2.887 2.887 0.000 ( +0.0% ) 3.20 3.13 +2.4% 2.688 2.625
2500.002 2.834 2.834 0.000 ( +0.0% ) 3.34 3.27 +1.9% 2.685 2.627
2500.01 1.440 1.440 0.000 ( +0.0% ) 6.26 6.03 +4.0% 2.329 2.297
2500.011 1.898 1.898 0.000 ( +0.0% ) 3.37 3.27 +2.9% 2.465 2.445
2500.012 1.754 1.754 0.000 ( +0.0% ) 4.92 4.75 +3.6% 2.431 2.360
2500.1 2.345 2.345 0.000 ( +0.0% ) 4.44 4.49 -0.9% 2.070 2.074
2500.2 2.449 2.449 0.000 ( +0.0% ) 5.25 5.14 +2.3% 1.986 1.922
2500.21 1.279 1.279 0.000 ( +0.0% ) 3.51 3.49 +0.5% 2.270 2.343
2500.211 1.659 1.659 0.000 ( +0.0% ) 3.28 3.13 +4.9% 2.345 2.240
2500.3 2.219 2.219 0.000 ( +0.0% ) 10.01 9.65 +3.8% 1.966 2.000
2500.301 2.822 2.822 0.000 ( +0.0% ) 8.59 8.08 +6.3% 1.956 1.951
2500.31 7.164 7.164 0.000 ( +0.0% ) 1.53 1.30 +18.0% 1.709 1.704
2500.311 1.568 1.568 0.000 ( +0.0% ) 8.88 4.78 +85.7% 1.061 1.055
2500.312 540.457 540.457 0.000 ( +0.0% ) 0.54 0.53 +1.2% 1.605 1.602
2500.313 817.694 817.694 0.000 ( +0.0% ) 0.76 0.74 +3.1% 1.592 1.614
2500.32 1.341 1.341 0.000 ( +0.0% ) 13.16 12.64 +4.1% 2.364 2.333
2500.321 1.748 1.748 0.000 ( +0.0% ) 10.05 8.99 +11.8% 2.453 2.447
2500.322 1.240 1.240 0.000 ( +0.0% ) 8.96 9.23 -2.8% 2.231 2.210
2500.323 7.772 7.772 0.000 ( +0.0% ) 3.73 3.21 +16.4% 1.817 1.928
2500.324 1.870 1.870 0.000 ( +0.0% ) 9.70 9.02 +7.5% 2.193 2.188
2500.325 4.156 4.156 0.000 ( +0.0% ) 4.51 4.22 +6.8% 2.145 2.147
2500.326 3.309 3.309 0.000 ( +0.0% ) 1.66 1.65 +0.7% 2.153 2.119
2500.327 1.804 1.804 0.000 ( +0.0% ) 10.03 9.29 +8.0% 2.341 2.327
2500.4 2.363 2.363 0.000 ( +0.0% ) 9.43 9.21 +2.4% 1.830 1.812
2500.401 1.891 1.891 0.000 ( +0.0% ) 8.69 8.12 +7.0% 1.689 1.707
2500.402 2.937 2.937 0.000 ( +0.0% ) 8.86 8.05 +10.1% 1.870 1.782
2500.403 8.687 8.687 0.000 ( +0.0% ) 3.17 2.96 +7.0% 1.928 1.786
2500.404 5.438 5.438 0.000 ( +0.0% ) 1.22 1.21 +1.2% 1.833 1.810
2500.405 2.847 2.847 0.000 ( +0.0% ) 8.92 8.22 +8.4% 1.909 1.910
2500.5 5.194 5.194 0.000 ( +0.0% ) 16.37 15.77 +3.8% 1.535 1.564
2500.51 9.120 9.120 0.000 ( +0.0% ) 9.84 9.59 +2.6% 1.473 1.512

@hqucms
Copy link
Contributor Author

hqucms commented Apr 19, 2024

+1

@perrotta
Copy link
Contributor

+alca

  • Updates in CalibTracker/SiStripCommon/plugins/SealModules.cc look correct

@cmsbuild
Copy link
Contributor

Pull request #44782 was updated. @hqucms, @vlimant can you please check and sign again.

@cmsbuild
Copy link
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2ec77b/39012/summary.html
COMMIT: 782b734
CMSSW: CMSSW_14_1_X_2024-04-22-1100/el8_amd64_gcc12
Additional Tests: NANO
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/44782/39012/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

NANO Comparison Summary

Summary:

  • You potentially added 1 lines to the logs
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 15
  • DQMHistoTests: Total histograms compared: 17023
  • DQMHistoTests: Total failures: 0
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 17023
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 14 files compared)
  • Checked 55 log files, 32 edm output root files, 15 DQM output files

Nano size comparison Summary:

Sample kb/ev ref kb/ev diff kb/ev ev/s/thd ref ev/s/thd diff rate mem/thd ref mem/thd
2500.0 2.783 2.783 0.000 ( +0.0% ) 3.41 3.51 -2.9% 2.189 2.260
2500.001 2.897 2.897 0.000 ( +0.0% ) 3.08 3.14 -1.7% 2.586 2.688
2500.002 2.843 2.843 0.000 ( +0.0% ) 3.21 3.24 -1.1% 2.570 2.661
2500.01 1.446 1.446 0.000 ( +0.0% ) 5.78 6.03 -4.1% 2.300 2.282
2500.011 1.906 1.906 0.000 ( +0.0% ) 3.12 3.28 -5.1% 2.456 2.467
2500.012 1.761 1.761 0.000 ( +0.0% ) 4.59 4.77 -3.7% 2.375 2.442
2500.1 2.354 2.354 0.000 ( +0.0% ) 4.37 4.43 -1.4% 2.073 2.075
2500.2 2.459 2.459 0.000 ( +0.0% ) 4.96 5.08 -2.4% 1.983 1.982
2500.21 1.286 1.286 0.000 ( +0.0% ) 3.40 3.47 -1.9% 2.269 2.269
2500.211 1.668 1.668 0.000 ( +0.0% ) 3.03 3.06 -0.9% 2.344 2.350
2500.3 2.229 2.229 0.000 ( +0.0% ) 9.12 9.24 -1.4% 1.973 1.969
2500.301 2.833 2.833 0.000 ( +0.0% ) 8.24 7.92 +4.0% 1.961 1.948
2500.31 7.164 7.164 0.000 ( +0.0% ) 1.41 1.38 +2.4% 1.672 1.702
2500.311 1.568 1.568 0.000 ( +0.0% ) 7.71 6.57 +17.3% 1.019 1.047
2500.312 540.457 540.457 0.000 ( +0.0% ) 0.48 0.50 -2.8% 1.599 1.589
2500.313 817.694 817.694 0.000 ( +0.0% ) 0.67 0.68 -1.9% 1.578 1.575
2500.32 1.348 1.348 0.000 ( +0.0% ) 12.00 12.15 -1.3% 2.339 2.335
2500.321 1.757 1.757 0.000 ( +0.0% ) 10.05 9.14 +10.0% 2.412 2.405
2500.322 1.236 1.236 0.000 ( +0.0% ) 9.67 9.58 +1.0% 2.103 2.129
2500.323 7.772 7.772 0.000 ( +0.0% ) 4.04 3.32 +21.6% 1.923 1.923
2500.324 1.873 1.873 0.000 ( +0.0% ) 9.78 9.37 +4.3% 2.167 2.170
2500.325 4.136 4.136 0.000 ( +0.0% ) 4.33 4.29 +0.8% 2.183 2.187
2500.326 3.342 3.342 0.000 ( +0.0% ) 1.56 1.60 -2.7% 2.231 1.914
2500.327 1.811 1.811 0.000 ( +0.0% ) 9.51 9.61 -1.0% 2.331 2.299
2500.4 2.374 2.374 0.000 ( +0.0% ) 8.75 8.26 +5.9% 1.812 1.784
2500.401 1.891 1.891 0.000 ( +0.0% ) 7.88 7.81 +0.9% 1.685 1.686
2500.402 2.950 2.950 0.000 ( +0.0% ) 8.00 7.57 +5.7% 1.762 1.779
2500.403 8.700 8.700 0.000 ( +0.0% ) 2.66 2.84 -6.2% 1.796 1.792
2500.404 5.474 5.474 0.000 ( +0.0% ) 1.16 1.18 -2.2% 1.808 1.767
2500.405 2.860 2.860 0.000 ( +0.0% ) 7.98 7.64 +4.5% 1.863 1.616
2500.5 5.194 5.194 0.000 ( +0.0% ) 15.32 15.36 -0.3% 1.530 1.499
2500.51 9.120 9.120 0.000 ( +0.0% ) 9.40 9.53 -1.4% 1.473 1.481

@hqucms
Copy link
Contributor Author

hqucms commented Apr 23, 2024

+1

@cmsbuild
Copy link
Contributor

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @rappoccio, @sextonkennedy, @antoniovilela (and backports should be raised in the release meeting by the corresponding L2)

@antoniovilela
Copy link
Contributor

+1

@cmsbuild cmsbuild merged commit 21cb3af into cms-sw:master Apr 24, 2024
13 checks passed
@antoniovilela
Copy link
Contributor

@vlimant
Copy link
Contributor

vlimant commented Apr 25, 2024

looks like unrelated to this PR ; what else has gone in that IB ?

@Dr15Jones
Copy link
Contributor

The list of PRs added to that build can be found here
https://cmssdt.cern.ch/SDT/html/cmssdt-ib/#/ib/CMSSW_14_1_X

@Dr15Jones
Copy link
Contributor

@vlimant it could be this PR if the type of the data product now being asked for by the module does not actually match the type in the edm::Event.

@vlimant
Copy link
Contributor

vlimant commented Apr 25, 2024

indeed, the type is reco::GenMETCollection in the NANOGEN, and std::vector<pat::MET> in standard nano

@hqucms
Copy link
Contributor Author

hqucms commented Apr 25, 2024

@Dr15Jones
Copy link
Contributor

Given the values being used in the cut string, just using SimpleCandidateFlatTableProducer would still work without having to use lazy evaluation. Alternatively, switching to SimpleMETFlatTableProducer.

@hqucms
Copy link
Contributor Author

hqucms commented Apr 25, 2024

just using SimpleCandidateFlatTableProducer would still work without having to use lazy evaluation

yes that's what I did :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants