Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[91X] Unify alignment producers #18138

Merged
merged 24 commits into from Apr 26, 2017

Conversation

ghellwig
Copy link

@ghellwig ghellwig commented Mar 31, 2017

Previously two parallel code infrastructures in place for offline and PCL alignment

  • Offline: AlignmentProduceredm::ESProducerLooper
  • PCL: PCLTrackerAlProduceredm::EDAnalyzer (legacy!)

Most of the code is just duplication with more than 1000 lines of redundant code, i.e. it is hard to maintain.
In addition, only the PCL code is routinely tested during CMSSW software integration in workflow 1001.

This PR unifies the two alignment producers by means of a common base class AlignmentProducerBase which provides the common implementation.
AlignmentProducer now inherits from AlignmentProducerBase and edm::ESProducerLooper.
PCLTrackerAlProducer is renamed to the more generic AlignmentProducerAsAnalyzer which now inherits from AlignmentProducerBase and edm::one::EDAnalyzer.

This means easier maintenance and multi-threading friendliness. The latter applies not only to the PCL workflow, but also the offline MillePede workflow which is migrated the edm::one::EDAnalyzer version of the alignment producer.
As a bonus, the main part of the offline alignment code is now routinely exercised during integration tests.

This changes has also been presented at the last Tracker Alignment meeting.

This PR includes also some minor fixes regarding the transition to phase-1.

@cmsbuild
Copy link
Contributor

A new Pull Request was created by @ghellwig (Gregor Mittag) for master.

It involves the following packages:

Alignment/CommonAlignment
Alignment/CommonAlignmentMonitor
Alignment/CommonAlignmentProducer
Alignment/MillePedeAlignmentAlgorithm
Alignment/OfflineValidation
Alignment/TrackerAlignment

@ghellwig, @arunhep, @cerminar, @cmsbuild, @franzoni, @mmusich, @davidlange6 can you please review it and eventually sign? Thanks.
@ghellwig, @pakhotin, @tocheng, @tlampen, @mschrode, @mmusich this is something you requested to watch as well.
@Muzaffar, @davidlange6, @smuzaffar you are the release manager for this.

cms-bot commands are listed here #13028

@ghellwig
Copy link
Author

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Mar 31, 2017

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/18824/console Started: 2017/03/31 11:27

@cmsbuild
Copy link
Contributor

-1

Tested at: 75138fa

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
8898a82
40dcb28
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-18138/18824/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-18138/18824/git-merge-result

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-18138/18824/summary.html

I found follow errors while testing this PR

Failed tests: Build

  • Build:

I found an error when building:

>> Compiling edm plugin /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/CMSSW_9_1_X_2017-03-30-2300/src/CondFormats/PhysicsToolsObjects/test/TGraphWriter.cc 
>> Compiling edm plugin /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/CMSSW_9_1_X_2017-03-30-2300/src/CondFormats/PhysicsToolsObjects/test/SiStripDeDxMipReader.cc 
>> Compiling edm plugin /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/CMSSW_9_1_X_2017-03-30-2300/src/CondFormats/PhysicsToolsObjects/test/SiStripDeDx2DReader.cc 
>> Compiling edm plugin /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/CMSSW_9_1_X_2017-03-30-2300/src/CondFormats/PhysicsToolsObjects/test/SiStripDeDxMipBuilder.cc 
>> Compiling edm plugin /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/CMSSW_9_1_X_2017-03-30-2300/src/CondFormats/PhysicsToolsObjects/test/TFormulaWriter.cc 
/build/cmsbld/jenkins-workarea/workspace/ib-any-integration/CMSSW_9_1_X_2017-03-30-2300/src/CondFormats/PCLConfig/plugins/AlignPCLThresholdsReader.cc:14:63: error: expected template-name before '<' token
   class AlignPCLThresholdsReader : public edm::one::EDAnalyzer<> {
                                                               ^
/build/cmsbld/jenkins-workarea/workspace/ib-any-integration/CMSSW_9_1_X_2017-03-30-2300/src/CondFormats/PCLConfig/plugins/AlignPCLThresholdsReader.cc:14:63: error: expected '{' before '<' token
/build/cmsbld/jenkins-workarea/workspace/ib-any-integration/CMSSW_9_1_X_2017-03-30-2300/src/CondFormats/PCLConfig/plugins/AlignPCLThresholdsReader.cc:14:63: error: expected unqualified-id before '<' token
/build/cmsbld/jenkins-workarea/workspace/ib-any-integration/CMSSW_9_1_X_2017-03-30-2300/src/CondFormats/PCLConfig/plugins/AlignPCLThresholdsReader.cc:31:80: error: invalid use of incomplete type 'class edmtest::AlignPCLThresholdsReader'


The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
8898a82
40dcb28
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-18138/18824/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-18138/18824/git-merge-result

@cmsbuild
Copy link
Contributor

Comparison not run due to Build errors (RelVals and Igprof tests were also skipped)

@ghellwig
Copy link
Author

ok, let's wait until #18137 is merged...

@cmsbuild
Copy link
Contributor

-1

Tested at: 76ed399

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-18138/19351/summary.html

I found follow errors while testing this PR

Failed tests: AddOn

  • AddOn:

I found errors in the following addon tests:

cmsDriver.py RelVal -s L1REPACK:Full2015Data --data --scenario=HeavyIons -n 10 --conditions auto:run2_hlt_HIon --relval 9000,50 --datatier "RAW" --eventcontent RAW --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_2016,Run2_HI --magField 38T_PostLS1 --fileout file:RelVal_Raw_HIon_DATA.root --filein /store/hidata/HIRun2015/HIHardProbes/RAW-RECO/HighPtJet-PromptReco-v1/000/263/689/00000/1802CD9A-DDB8-E511-9CF9-02163E0138CA.root : FAILED - time: date Mon Apr 24 23:33:57 2017-date Mon Apr 24 23:25:00 2017 s - exit: 23552
cmsRun /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_9_1_X_2017-04-24-1100/src/HLTrigger/Configuration/test/OnLine_HLT_HIon.py realData=True globalTag=@ inputFiles=@ : FAILED - time: date Mon Apr 24 23:33:57 2017-date Mon Apr 24 23:25:00 2017 s - exit: 21504
cmsDriver.py RelVal -s HLT:HIon,RAW2DIGI,L1Reco,RECO --data --scenario=HeavyIons -n 10 --conditions auto:run2_data_HIon --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2016,Run2_HI --magField 38T_PostLS1 --processName=HLTRECO --filein file:RelVal_Raw_HIon_DATA.root --fileout file:RelVal_Raw_HIon_DATA_HLT_RECO.root : FAILED - time: date Mon Apr 24 23:33:57 2017-date Mon Apr 24 23:25:00 2017 s - exit: 21504
cmsRun /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_9_1_X_2017-04-24-1100/src/Utilities/ReleaseScripts/scripts/read312RV_cfg.py : FAILED - time: date Mon Apr 24 23:32:40 2017-date Mon Apr 24 23:25:30 2017 s - exit: 23552
cmsDriver.py RelVal -s L1REPACK:Full --data --scenario=pp -n 10 --conditions auto:run2_hlt_GRun --relval 9000,50 --datatier "RAW" --eventcontent RAW --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_2017 --magField 38T_PostLS1 --fileout file:RelVal_Raw_GRun_DATA.root --filein /store/data/Run2016B/JetHT/RAW/v1/000/272/762/00000/C666CDE2-E013-E611-B15A-02163E011DBE.root : FAILED - time: date Mon Apr 24 23:35:00 2017-date Mon Apr 24 23:25:37 2017 s - exit: 23552
cmsRun /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_9_1_X_2017-04-24-1100/src/HLTrigger/Configuration/test/OnLine_HLT_GRun.py realData=True globalTag=@ inputFiles=@ : FAILED - time: date Mon Apr 24 23:35:00 2017-date Mon Apr 24 23:25:37 2017 s - exit: 21504
cmsDriver.py RelVal -s HLT:GRun,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_GRun --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2017 --magField 38T_PostLS1 --processName=HLTRECO --filein file:RelVal_Raw_GRun_DATA.root --fileout file:RelVal_Raw_GRun_DATA_HLT_RECO.root : FAILED - time: date Mon Apr 24 23:35:00 2017-date Mon Apr 24 23:25:37 2017 s - exit: 21504

@cmsbuild
Copy link
Contributor

Comparison job queued.

@cmsbuild
Copy link
Contributor

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-18138/19351/summary.html

Comparison Summary:

  • You potentially added 106 lines to the logs
  • Reco comparison results: 69 differences found in the comparisons
  • DQMHistoTests: Total files compared: 23
  • DQMHistoTests: Total histograms compared: 1780042
  • DQMHistoTests: Total failures: 14603
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 1765266
  • DQMHistoTests: Total skipped: 173
  • DQMHistoTests: Total Missing objects: 0
  • Checked 94 log files, 14 edm output root files, 23 DQM output files

@ghellwig
Copy link
Author

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 25, 2017

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/19364/console Started: 2017/04/25 07:17

@cmsbuild
Copy link
Contributor

-1

Tested at: 76ed399

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-18138/19364/summary.html

I found follow errors while testing this PR

Failed tests: RelVals AddOn

  • RelVals:

When I ran the RelVals I found an error in the following worklfows:
4.53 step2

runTheMatrix-results/4.53_RunPhoton2012B+RunPhoton2012B+HLTD+RECODR1reHLT+HARVESTDR1reHLT/step2_RunPhoton2012B+RunPhoton2012B+HLTD+RECODR1reHLT+HARVESTDR1reHLT.log

  • AddOn:

I found errors in the following addon tests:

cmsDriver.py RelVal -s L1REPACK:Full --data --scenario=pp -n 10 --conditions auto:run2_hlt_PRef --relval 9000,50 --datatier "RAW" --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_2017 --magField 38T_PostLS1 --eventcontent RAW --fileout file:RelVal_Raw_PRef_DATA.root --filein /store/data/Run2016B/JetHT/RAW/v1/000/272/762/00000/C666CDE2-E013-E611-B15A-02163E011DBE.root : FAILED - time: date Tue Apr 25 08:25:44 2017-date Tue Apr 25 08:16:27 2017 s - exit: 23552
cmsRun /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_9_1_X_2017-04-24-1100/src/HLTrigger/Configuration/test/OnLine_HLT_PRef.py realData=True globalTag=@ inputFiles=@ : FAILED - time: date Tue Apr 25 08:25:44 2017-date Tue Apr 25 08:16:27 2017 s - exit: 21504
cmsDriver.py RelVal -s HLT:PRef,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_PRef --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2017 --magField 38T_PostLS1 --processName=HLTRECO --filein file:RelVal_Raw_PRef_DATA.root --fileout file:RelVal_Raw_PRef_DATA_HLT_RECO.root : FAILED - time: date Tue Apr 25 08:25:44 2017-date Tue Apr 25 08:16:27 2017 s - exit: 21504
cmsDriver.py RelVal -s L1REPACK:Full2015Data --data --scenario=HeavyIons -n 10 --conditions auto:run2_hlt_HIon --relval 9000,50 --datatier "RAW" --eventcontent RAW --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_2016,Run2_HI --magField 38T_PostLS1 --fileout file:RelVal_Raw_HIon_DATA.root --filein /store/hidata/HIRun2015/HIHardProbes/RAW-RECO/HighPtJet-PromptReco-v1/000/263/689/00000/1802CD9A-DDB8-E511-9CF9-02163E0138CA.root : FAILED - time: date Tue Apr 25 08:25:29 2017-date Tue Apr 25 08:16:32 2017 s - exit: 23552
cmsRun /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_9_1_X_2017-04-24-1100/src/HLTrigger/Configuration/test/OnLine_HLT_HIon.py realData=True globalTag=@ inputFiles=@ : FAILED - time: date Tue Apr 25 08:25:29 2017-date Tue Apr 25 08:16:32 2017 s - exit: 21504
cmsDriver.py RelVal -s HLT:HIon,RAW2DIGI,L1Reco,RECO --data --scenario=HeavyIons -n 10 --conditions auto:run2_data_HIon --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2016,Run2_HI --magField 38T_PostLS1 --processName=HLTRECO --filein file:RelVal_Raw_HIon_DATA.root --fileout file:RelVal_Raw_HIon_DATA_HLT_RECO.root : FAILED - time: date Tue Apr 25 08:25:29 2017-date Tue Apr 25 08:16:32 2017 s - exit: 21504
cmsDriver.py RelVal -s L1REPACK:Full --data --scenario=pp -n 10 --conditions auto:run2_hlt_PIon --relval 9000,50 --datatier "RAW" --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_2017 --magField 38T_PostLS1 --eventcontent RAW --fileout file:RelVal_Raw_PIon_DATA.root --filein /store/data/Run2016B/JetHT/RAW/v1/000/272/762/00000/C666CDE2-E013-E611-B15A-02163E011DBE.root : FAILED - time: date Tue Apr 25 08:28:17 2017-date Tue Apr 25 08:16:52 2017 s - exit: 23552
cmsRun /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_9_1_X_2017-04-24-1100/src/HLTrigger/Configuration/test/OnLine_HLT_PIon.py realData=True globalTag=@ inputFiles=@ : FAILED - time: date Tue Apr 25 08:28:17 2017-date Tue Apr 25 08:16:52 2017 s - exit: 21504
cmsDriver.py RelVal -s HLT:PIon,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_PIon --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2017 --magField 38T_PostLS1 --processName=HLTRECO --filein file:RelVal_Raw_PIon_DATA.root --fileout file:RelVal_Raw_PIon_DATA_HLT_RECO.root : FAILED - time: date Tue Apr 25 08:28:17 2017-date Tue Apr 25 08:16:52 2017 s - exit: 21504
cmsRun /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_9_1_X_2017-04-24-1100/src/Utilities/ReleaseScripts/scripts/read312RV_cfg.py : FAILED - time: date Tue Apr 25 08:24:30 2017-date Tue Apr 25 08:17:12 2017 s - exit: 23552
cmsDriver.py RelVal -s L1REPACK:GT1 --data --scenario=pp -n 10 --conditions auto:run1_hlt_Fake --relval 9000,50 --datatier "RAW" --eventcontent RAW --customise=HLTrigger/Configuration/CustomConfigs.L1T --fileout file:RelVal_Raw_Fake_DATA.root --filein /store/data/Run2012A/MuEG/RAW/v1/000/191/718/14932935-E289-E111-830C-5404A6388697.root : FAILED - time: date Tue Apr 25 08:25:27 2017-date Tue Apr 25 08:17:17 2017 s - exit: 23552
cmsRun /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_9_1_X_2017-04-24-1100/src/HLTrigger/Configuration/test/OnLine_HLT_Fake.py realData=True globalTag=@ inputFiles=@ : FAILED - time: date Tue Apr 25 08:25:27 2017-date Tue Apr 25 08:17:17 2017 s - exit: 21504
cmsDriver.py RelVal -s HLT:Fake,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run1_data_Fake --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --processName=HLTRECO --filein file:RelVal_Raw_Fake_DATA.root --fileout file:RelVal_Raw_Fake_DATA_HLT_RECO.root : FAILED - time: date Tue Apr 25 08:25:27 2017-date Tue Apr 25 08:17:17 2017 s - exit: 21504
cmsDriver.py RelVal -s L1REPACK:Full --data --scenario=pp -n 10 --conditions auto:run2_hlt_GRun --relval 9000,50 --datatier "RAW" --eventcontent RAW --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_2017 --magField 38T_PostLS1 --fileout file:RelVal_Raw_GRun_DATA.root --filein /store/data/Run2016B/JetHT/RAW/v1/000/272/762/00000/C666CDE2-E013-E611-B15A-02163E011DBE.root : FAILED - time: date Tue Apr 25 08:29:05 2017-date Tue Apr 25 08:17:20 2017 s - exit: 23552
cmsRun /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_9_1_X_2017-04-24-1100/src/HLTrigger/Configuration/test/OnLine_HLT_GRun.py realData=True globalTag=@ inputFiles=@ : FAILED - time: date Tue Apr 25 08:29:05 2017-date Tue Apr 25 08:17:20 2017 s - exit: 21504
cmsDriver.py RelVal -s HLT:GRun,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_GRun --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2017 --magField 38T_PostLS1 --processName=HLTRECO --filein file:RelVal_Raw_GRun_DATA.root --fileout file:RelVal_Raw_GRun_DATA_HLT_RECO.root : FAILED - time: date Tue Apr 25 08:29:05 2017-date Tue Apr 25 08:17:20 2017 s - exit: 21504

@cmsbuild
Copy link
Contributor

Comparison not run due to runTheMatrix errors (RelVals and Igprof tests were also skipped)

@arunhep
Copy link
Contributor

arunhep commented Apr 25, 2017

+1
the tests failed because of xrood/eos errors. but they have passed in earlier runs.

@cmsbuild
Copy link
Contributor

This pull request is fully signed and it will be integrated in one of the next master IBs (but tests are reportedly failing). This pull request requires discussion in the ORP meeting before it's merged. @Muzaffar, @davidlange6, @smuzaffar

@cmsbuild cmsbuild merged commit 76ed399 into cms-sw:master Apr 26, 2017
@ghellwig ghellwig deleted the unify-alignment-producers_91X branch May 2, 2017 15:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants