Remove deprecated JetCorrector and related files #39953

wddgit · 2022-11-01T19:00:30Z

PR description:

Remove the deprecated ::JetCorrector class from CMSSW and other related files with it. The migration from ::JetCorrector to reco::JetCorrector started in 2014 and it is time to finally complete it. For the most part this PR removes unused dead code and simplifies things. It makes it impossible for someone to use the deprecated code and obvious failures will occur if they try. It reduces the amount of code we need to maintain and support.

Almost this entire PR consists of deletions. There is one exception to that. The exception is in RecoHI/HiJetAlgos/python/hiCaloJetsForTrk_cff.py. I migrated that file from using the old JetCorrector type to using the new reco::JetCorrector. Somehow that one was missed earlier.

There has been a long effort to migrate from the old JetCorrectors to the new JetCorrectors that has been in progress since 2014. Volker Adler did most of this migration long ago. I've recently migrated all the remaining C++ code. I've also migrated all the Python configuration code that is used in existing tests that I can find. And I will help if merging this PR reveals more cases of code in CMSSW that needs migration. There are 4 PRs that are submitted but not merged yet. These need to be merged before this PR or there will be compilation errors. These are #39875 #39864 #39847 and #39804.

I also deleted the obsolete central cff configuration files commonly imported to use the old JetCorrector class.

There are dozens of other configuration files that still reference the old cff files deleted by this PR. Those will be broken by this PR, but they are all old and look to be old obsolete code that is already broken. It is not worth the effort to deal with them at this point. If someone tries to run them they will fail with an obvious exception and then they can migrated to use the new JetCorrector class. There is a good chance none of them will ever be used again.

PR validation:

Relies on existing tests. Errors caused by this should result in obvious exceptions.

The one exception is the change in hiCaloJetsForTrk_cff.py. That is tested by runTheMatrix.py test 140.53. That is how I noticed it still needed to be migrated.

If other failures show up, I will be available to help migrate the affected code.

cmsbuild · 2022-11-01T19:08:36Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39953/32883

This PR adds an extra 20KB to repository

cmsbuild · 2022-11-01T19:09:02Z

A new Pull Request was created by @wddgit (W. David Dagenhart) for master.

It involves the following packages:

DQM/Physics (dqm)
JetMETCorrections/Algorithms (analysis)
JetMETCorrections/Configuration (reconstruction)
JetMETCorrections/Modules (reconstruction)
JetMETCorrections/Objects (reconstruction)
PhysicsTools/PatAlgos (xpog, reconstruction)
RecoHI/HiJetAlgos (reconstruction)

@swertz, @vlimant, @micsucmed, @emanueleusai, @ahmad3213, @cmsbuild, @jfernan2, @clacaputo, @syuvivida, @pmandrik, @mandrenguyen, @rvenditti can you please review it and eventually sign? Thanks.
@rappoccio, @gouskos, @yslai, @schoef, @emilbols, @mbluj, @demuller, @seemasharmafnal, @mmarionncern, @jdolen, @ahinzmann, @dgulhan, @missirol, @azotz, @mandrenguyen, @yetkinyilmaz, @jdamgov, @nhanvtran, @gkasieczka, @hatakeyamak, @mariadalfonso, @clelange, @AlexDeMoor, @AnnikaStein, @jazzitup, @JyothsnaKomaragiri, @yenjie, @kurtejung, @gpetruc, @andrzejnovak this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

wddgit · 2022-11-01T19:22:58Z

please test with #39875,#39864,#39847,#39804

FYI @Dr15Jones @makortel

makortel · 2022-11-01T19:26:44Z

JetMETCorrections/Objects/interface/JetCorrector.h

-inline bool JetCorrector::vectorialCorrection() const { return false; }
-
-#endif
+#error, obsolete header, use reco::JetCorrector in JetMETCorrections/JetCorrector/interface/JetCorrector.h instead


Should this be

Suggested change

#error, obsolete header, use reco::JetCorrector in JetMETCorrections/JetCorrector/interface/JetCorrector.h instead

#error "obsolete header, use reco::JetCorrector in JetMETCorrections/JetCorrector/interface/JetCorrector.h instead"

?

I will fix that as soon as the tests finish. Thanks. No idea why I put a comma there...

cmsbuild · 2022-11-01T22:50:23Z

-1

Failed Tests: HeaderConsistency
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c6983d/28715/summary.html
COMMIT: 2ad4017
CMSSW: CMSSW_12_6_X_2022-11-01-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/39953/28715/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

There are some workflows for which there are errors in the baseline:
534.0 step 1
536.0 step 1
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

No significant changes to the logs found
Reco comparison results: 10 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3416398
DQMHistoTests: Total failures: 12
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3416364
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 206 log files, 48 edm output root files, 48 DQM output files
TriggerResults: no differences found

makortel · 2022-11-02T13:22:56Z

-1

Failed Tests: HeaderConsistency

Note that the header consistency check failure is caused by the #error, i.e. it is expected.

cmsbuild · 2022-11-02T14:41:35Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39953/32889

This PR adds an extra 20KB to repository

makortel · 2022-11-28T14:58:05Z

hold

(seems that holding needs more power)

cmsbuild · 2022-11-28T14:58:25Z

Pull request has been put on hold by @makortel
They need to issue an unhold command to remove the hold state or L1 can unhold it for all

wddgit · 2022-12-05T14:40:22Z

please test

@makortel I think we can remove the hold. I'm rerunning the tests to see if they will pass with no other PR's included. I think it should now that all the other PRs were merged.

makortel · 2022-12-05T14:44:18Z

unhold

Thanks for reminding

cmsbuild · 2022-12-05T19:12:55Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c6983d/29464/summary.html
COMMIT: 44852c6
CMSSW: CMSSW_13_0_X_2022-12-05-1100/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/39953/29464/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 25 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3421337
DQMHistoTests: Total failures: 1170
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3420145
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 206 log files, 158 edm output root files, 48 DQM output files
TriggerResults: no differences found

makortel · 2022-12-05T19:36:43Z

The comparison differences are in 11634.7, so this looks like another symptom of #39803 (comment)

rappoccio · 2022-12-07T14:25:31Z

+1

Comparison differences are spurious.

perrotta · 2022-12-12T09:50:24Z

merge

perrotta · 2022-12-12T12:56:28Z

@wddgit after the merge of this PR there are unit test errors in Configuration/DataProcessing, apparently related to the relics JetCorrectionServicesAllAlgos_cff.py which is still imported in

HeavyIonsAnalysis/Configuration/python/HI_DiJetSkim_cff.py
HeavyIonsAnalysis/Configuration/python/analysisFilters_cff.py

In reality I see many more places in CMSSW where that config is imported, see e.g. https://cmssdt.cern.ch/lxr/search?%21v=CMSSW_13_0_X_2022-12-12-1100&_filestring=&_string=rrectionServicesAllAlgos_cff

Could you please have a look?

wddgit · 2022-12-12T14:51:42Z

I'll take a look this morning.

wddgit · 2022-12-12T16:08:17Z

I missed HI_DiJetSkim_cff.py. I'll work on a PR migrating that to use the new JetCorrector class as soon as possible and submit a PR. Hopefully, I can do that today. It will be my top priority. If there are any others I need to migrate to get the unit tests in Configuration/DataProcessing to succeed, then I will take care of those also. Thanks for identifying this problem.

This PR deleted several central cff files. Many dozens of existing cff or cfg files import those and are also broken by this PR. Your search found some of them, but many more are broken and will show up if one searches for use of other cff files deleted by this PR. There is a paragraph about this in the comment at the top of the PR. My proposal for dealing with this was to migrate the ones needed in existing tests, but not to migrate the rest. Many of them are obsolete and broken already. In most cases, running a configuration actually trying to use the old JetCorrectors would have already resulted in an exception being thrown by the Framework. The Framework has been behaving that way for about a year.

If someone identifies specific cases of broken cff or cfg files that need migrating and that are still in use and they need help with this, then I am willing to help migrate them. I don't mind doing the migration itself, but it's hard for me to fix problems unrelated JetCorrectors or write new tests in domains where I don't have any experience.

perrotta · 2022-12-12T16:37:53Z

Thank you @wddgit
Please try to fix at least the files which are tested in the IB, as we wouldn't like to have a broke pre-release to be built in the next few days. Otherwise we will revert this one, and move it to the next pre-release.
Then, if you can prepare a list of all those "dozens of other configuration files that still reference the old cff files deleted by this PR" we can make a github issue with it, and ask people either to remove them, if not needed any more, or fix them.

wddgit · 2022-12-12T21:42:05Z

The PR fixing the unit test is #40301, recently submitted.

I also submitted a new Issue with the list of files importing the deleted cff files. Issue #40304. Note that none of these are used in tests. As far as I know they are not used at all.

wddgit · 2022-12-13T14:58:36Z

Starting with the IB after this was merged, CMSSW_13_0_X_2022-12-12-1100, the Static Analyzer test in the IB named "SA EventSetupRecord::get called" is now reporting this:

"no-package: {}"

I presume this means it found no problems. Clearing that was the main goal of this PR.

Also after the #40301 was merged last night, the unit tests all passed.

perrotta · 2022-12-13T15:20:14Z

Good. Thank you @wddgit , also for the github issue

cmsbuild added this to the CMSSW_12_6_X milestone Nov 1, 2022

cmsbuild added analysis-pending code-checks-pending dqm-pending orp-pending pending-signatures reconstruction-pending tests-pending xpog-pending labels Nov 1, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 1, 2022

cmsbuild added requires-external tests-started and removed tests-pending labels Nov 1, 2022

makortel reviewed Nov 1, 2022

View reviewed changes

cmsbuild added tests-rejected and removed tests-started labels Nov 1, 2022

Remove deprecated JetCorrector and related files

f3fd3e1

wddgit force-pushed the removeDeprecatedJetCorrector branch from 2ad4017 to f3fd3e1 Compare November 2, 2022 14:33

cmsbuild added code-checks-pending tests-pending and removed tests-rejected requires-external code-checks-approved labels Nov 2, 2022

cmsbuild added the hold label Nov 28, 2022

cmsbuild added tests-started and removed requires-external tests-approved labels Dec 5, 2022

cmsbuild removed the hold label Dec 5, 2022

cmsbuild added tests-approved and removed tests-started labels Dec 5, 2022

makortel mentioned this pull request Dec 5, 2022

spurious differences in outputs of Run-3 wfs *.7 #39803

Open

cmsbuild added orp-approved and removed orp-pending labels Dec 7, 2022

cmsbuild merged commit d2b40c9 into cms-sw:master Dec 12, 2022

This was referenced Dec 12, 2022

Complete Migration from Old JetCorrectors cms-sw/framework-team#452

Closed

Fix test failures caused by removing JetCorrectionServicesAllAlgos_cff.py cms-sw/framework-team#470

Closed

wddgit mentioned this pull request Dec 16, 2022

Delete commented out imports of deprecated JetCorrector cffs #40348

Merged

wddgit deleted the removeDeprecatedJetCorrector branch January 13, 2023 17:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove deprecated JetCorrector and related files #39953

Remove deprecated JetCorrector and related files #39953

wddgit commented Nov 1, 2022 •

edited

cmsbuild commented Nov 1, 2022

cmsbuild commented Nov 1, 2022

wddgit commented Nov 1, 2022

makortel Nov 1, 2022

wddgit Nov 1, 2022

cmsbuild commented Nov 1, 2022

makortel commented Nov 2, 2022

cmsbuild commented Nov 2, 2022

makortel commented Nov 28, 2022

cmsbuild commented Nov 28, 2022

wddgit commented Dec 5, 2022

makortel commented Dec 5, 2022

cmsbuild commented Dec 5, 2022

makortel commented Dec 5, 2022

rappoccio commented Dec 7, 2022

perrotta commented Dec 12, 2022 •

edited

perrotta commented Dec 12, 2022

wddgit commented Dec 12, 2022

wddgit commented Dec 12, 2022

perrotta commented Dec 12, 2022

wddgit commented Dec 12, 2022

wddgit commented Dec 13, 2022

perrotta commented Dec 13, 2022

	#error, obsolete header, use reco::JetCorrector in JetMETCorrections/JetCorrector/interface/JetCorrector.h instead
	#error "obsolete header, use reco::JetCorrector in JetMETCorrections/JetCorrector/interface/JetCorrector.h instead"

Remove deprecated JetCorrector and related files #39953

Remove deprecated JetCorrector and related files #39953

Conversation

wddgit commented Nov 1, 2022 • edited

PR description:

PR validation:

cmsbuild commented Nov 1, 2022

cmsbuild commented Nov 1, 2022

wddgit commented Nov 1, 2022

makortel Nov 1, 2022

Choose a reason for hiding this comment

wddgit Nov 1, 2022

Choose a reason for hiding this comment

cmsbuild commented Nov 1, 2022

Comparison Summary

makortel commented Nov 2, 2022

cmsbuild commented Nov 2, 2022

makortel commented Nov 28, 2022

cmsbuild commented Nov 28, 2022

wddgit commented Dec 5, 2022

makortel commented Dec 5, 2022

cmsbuild commented Dec 5, 2022

Comparison Summary

makortel commented Dec 5, 2022

rappoccio commented Dec 7, 2022

perrotta commented Dec 12, 2022 • edited

perrotta commented Dec 12, 2022

wddgit commented Dec 12, 2022

wddgit commented Dec 12, 2022

perrotta commented Dec 12, 2022

wddgit commented Dec 12, 2022

wddgit commented Dec 13, 2022

perrotta commented Dec 13, 2022

wddgit commented Nov 1, 2022 •

edited

perrotta commented Dec 12, 2022 •

edited