Deprecating Phase2 Tracker T3 and corresp. scenarios + Adding D18 (= D8 but with the non deprec. Tracker) #18852

boudoul · 2017-05-19T13:30:33Z

Greetings
This PR aims to deprecate all scenarios which contain the tracker T3 (I also removed all corresponding T3 xmls) , keeping of course all the ones with the T5 Tracker .
Basically it implies to remove the following scenario :
D7,D4,D8,D9, D12, D13, D15

and therefore keep :
D10, D11, D14,D16,D17

Note that for any remaining scenario the numbering in runthematrix has been preserved (hopefully, it looks ok in my tests...)

@kpedro88 kindly made me noticed that the timing layer was only in D8 (D8 is removed in this PR ) so I followed his suggestion and I have implemented a new scenario D18 which is equivalent to D8 but with the new tracker so that this software will still be there (but with the right tracker) .

For D18 , I choosed to re-use the same WF numbers as D8 , hope this is fine.

Comments and suggestions about this (pretty huge) cleaning are of course welcome , thanks.

cmsbuild · 2017-05-19T13:30:59Z

A new Pull Request was created by @boudoul (boudoul) for master.

It involves the following packages:

Configuration/Geometry
Configuration/PyReleaseValidation
Configuration/StandardSequences
Geometry/CMSCommonData
Geometry/TrackerCommonData
Geometry/TrackerRecoData
Geometry/TrackerSimData

@civanch, @Dr15Jones, @kkousour, @ianna, @mdhildreth, @fabozzi, @cmsbuild, @franzoni, @kpedro88, @GurpreetSinghChahal, @davidlange6 can you please review it and eventually sign? Thanks.
@ghellwig, @makortel, @felicepantaleo, @GiacomoSguazzoni, @rovere, @VinInn, @Martin-Grunewald, @ebrondol, @ghugo83, @dgulhan, @venturia this is something you requested to watch as well.
@davidlange6 you are the release manager for this.

cms-bot commands are listed here

boudoul · 2017-05-19T13:40:51Z

Adding @delaere and @atricomi

kpedro88 · 2017-05-19T14:03:51Z

Configuration/PyReleaseValidation/python/relval_2023.py

-numWFIB.extend([20434.0,20434.2]) #2023D10
-numWFIB.extend([21234.0,21234.2]) #2023D4
-numWFIB.extend([21234.1]) #2023D4 special tracking workflow
-numWFIB.extend([23234.0]) #2023D8 - already has timing


Need to add D18 workflows here

kpedro88 · 2017-05-19T14:18:37Z

Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py

@@ -45,12 +33,12 @@
 # pre-generation of WF numbers
 numWFStart={
    2017: 10000,
-    2023: 20000,
+    2023: 20400,#GB: since the WF numbers are appended from this and I want to skip this one I increased this number to the new actual one 


I would prefer to keep this as the nominal 20000 and just use the numWFConflict to skip it

kpedro88 · 2017-05-19T14:19:16Z

Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py

 upgradeProperties[2023]['2023D16PU'] = deepcopy(upgradeProperties[2023]['2023D16'])
 upgradeProperties[2023]['2023D16PU']['ScenToRun'] = ['GenSimHLBeamSpotFull','DigiFullPU','RecoFullGlobalPU', 'HARVESTFullGlobalPU']
 upgradeProperties[2023]['2023D17PU'] = deepcopy(upgradeProperties[2023]['2023D17'])
 upgradeProperties[2023]['2023D17PU']['ScenToRun'] = ['GenSimHLBeamSpotFull','DigiFullPU','RecoFullGlobalPU', 'HARVESTFullGlobalPU']
+upgradeProperties[2023]['2023D18PU'] = deepcopy(upgradeProperties[2023]['2023D18'])


you don't need to have D18 twice in this list

kpedro88 · 2017-05-19T14:22:39Z

@boudoul sent a few comments for you

it's fine with me to reuse D8 workflow numbers for D18, we did the same thing during a previous round of deprecation.

boudoul · 2017-05-19T14:28:07Z

thanks for the super prompt feedback, I will adress your comments asap

cmsbuild · 2017-05-19T16:49:57Z

Pull request #18852 was updated. @civanch, @Dr15Jones, @kkousour, @ianna, @mdhildreth, @fabozzi, @cmsbuild, @franzoni, @kpedro88, @GurpreetSinghChahal, @davidlange6 can you please check and sign again.

kpedro88 · 2017-05-19T16:54:27Z

Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py

 for year in upgradeKeys:
-    for i in range(1,len(upgradeKeys[year])):


I think this could be fixed a bit more simply/efficiently:

for i in range(0,len(upgradeKeys[year])): numWFtmp = numWFStart[year] if i==0 else (numWFAll[year][i-1] + numWFSkip)

@boudoul can you implement this change?

hi @kpedro88 , my last commit implemented the changes you proposed

davidlange6 · 2017-05-19T17:06:32Z

a case not protected against here is that of two excluded regions that are neighboring. other than that, indeed, they should be the same

…

On May 19, 2017, at 6:54 PM, Kevin Pedro ***@***.***> wrote: @kpedro88 commented on this pull request. In Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py: > for year in upgradeKeys: - for i in range(1,len(upgradeKeys[year])): I think this could be fixed a bit more simply/efficiently: for i in range(0,len(upgradeKeys[year])): numWFtmp = numWFStart[year] if i==0 else (numWFAll[year][i-1] + numWFSkip) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

kpedro88 · 2017-05-19T18:03:13Z

@davidlange6 the (somewhat fragile) assumption is that people will combine neighboring excluded regions in the numWFConflict list

ianna · 2017-05-19T19:33:29Z

Configuration/Geometry/python/dict2023Geometry.py

    ("O1","T5","C3","M2","F1","I1") : "D16",
    ("O2","T5","C3","M2","F2","I1") : "D17",
+    ("O1","T5","C2","M1","F1","I2") : "D18",


@boudoul - should it be O2 since @kpedro88 mentioned at sim meeting that the detailed cavern poses no problems and will be used for TDR samples?

D18 is just intended to be D8 with T5 (not used for TDR). Other changes/updates can happen at a later time.

cmsbuild · 2017-05-25T16:55:39Z

Pull request #18852 was updated. @civanch, @Dr15Jones, @kkousour, @ianna, @mdhildreth, @fabozzi, @cmsbuild, @franzoni, @kpedro88, @GurpreetSinghChahal, @davidlange6 can you please check and sign again.

cmsbuild · 2017-05-25T16:58:52Z

Pull request #18852 was updated. @civanch, @Dr15Jones, @kkousour, @ianna, @mdhildreth, @fabozzi, @cmsbuild, @franzoni, @kpedro88, @GurpreetSinghChahal, @davidlange6 can you please check and sign again.

kpedro88 · 2017-05-25T17:00:38Z

please test

cmsbuild · 2017-05-25T17:00:57Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/20117/console Started: 2017/05/25 19:02

cmsbuild · 2017-05-25T17:55:23Z

+1
Tested at: d3577e9
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-18852/20117/summary.html

cmsbuild · 2017-05-25T17:55:26Z

Comparison job queued.

cmsbuild · 2017-05-25T21:32:58Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-18852/20117/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 1709 differences found in the comparisons
DQMHistoTests: Total files compared: 21
DQMHistoTests: Total histograms compared: 1616438
DQMHistoTests: Total failures: 42052
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 1574234
DQMHistoTests: Total skipped: 152
DQMHistoTests: Total Missing objects: 0
Checked 82 log files, 14 edm output root files, 21 DQM output files

kpedro88 · 2017-05-25T22:25:13Z

+1

ianna · 2017-05-26T05:52:24Z

+1

smuzaffar · 2017-05-30T07:16:37Z

@ianna , @boudoul , could this PR be the cause of failing unittest in 92X IBs https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_9_2_X_2017-05-29-2300/unitTestLogs/DetectorDescription/RegressionTest

run_DOMCount.py Geometry/CMSCommonData/cmsExtendedGeometry2023D7XML_cfi
Failure run_DOMCount.py Geometry/CMSCommonData/cmsExtendedGeometry2023D7XML_cfi: status 1
status = 256

---> test DetectorDescriptionRegressionTestDOMCount had ERRORS

can you please check and provide a fix?

boudoul · 2017-05-30T08:08:51Z

hi @smuzaffar , thanks for reporting- I just made a PR #19006 which should cure this, I let @ianna comments though . Thanks

slava77 · 2017-05-31T14:51:28Z

I notice that workflow numbers are not in order of D* anymore: D18 comes before D17. Was there a good reason for that?

Also, the flat tracker was never in the short matrix tests. why is it added now?

kpedro88 · 2017-05-31T14:53:09Z

@slava77 they weren't in order before, either. We replaced D8 with D18, as we had done in the past for some other workflows when deprecating T1.

The track trigger worked with T3, but doesn't yet work with T5. In order to make sure that section of the code doesn't break, I decided to add D10 (w/ T4) to the matrix tests.

boudoul · 2017-05-31T14:58:06Z

right, and now that this PR is merged, I will proceed with the track trigger with T5

boudoul added 10 commits May 18, 2017 12:31

removing all WFs with T3

7194e8c

removing scenario, preparing D18

6181ed9

creating dict for D18

077f7c4

output of the geom scripts for D18

9b28eb3

removing XML cfis from deprec scenario

8b458fd

Old tracker go to trash

572909e

numbering for D18 to be the same as D8

3750c05

removing deprec scenario in neutronkeys + IB test

5faed8c

removing my stupid print

697c4a0

fix indent

e8eab7b

cmsbuild added this to the CMSSW_9_2_X milestone May 19, 2017

cmsbuild added comparison-pending geometry-pending operations-pending orp-pending pdmv-pending pending-signatures tests-pending upgrade-pending labels May 19, 2017

kpedro88 reviewed May 19, 2017

View reviewed changes

removeD18 dupl + add D18 in tests + struggled with 20000

f5291a0

kpedro88 reviewed May 19, 2017

View reviewed changes

ianna reviewed May 19, 2017

View reviewed changes

cmsbuild added comparison-pending geometry-pending tests-pending labels May 25, 2017

forgot a comment

d3577e9

cmsbuild added tests-started and removed tests-pending labels May 25, 2017

cmsbuild added tests-approved and removed tests-started labels May 25, 2017

cmsbuild added comparison-available and removed comparison-pending labels May 25, 2017

cmsbuild added upgrade-approved and removed upgrade-pending labels May 25, 2017

cmsbuild added geometry-approved and removed geometry-pending labels May 26, 2017

davidlange6 merged commit 726aad3 into cms-sw:master May 29, 2017

boudoul mentioned this pull request May 30, 2017

Use non-deprecated phase2 scenario in detector description unit tests #19006

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deprecating Phase2 Tracker T3 and corresp. scenarios + Adding D18 (= D8 but with the non deprec. Tracker) #18852

Deprecating Phase2 Tracker T3 and corresp. scenarios + Adding D18 (= D8 but with the non deprec. Tracker) #18852

boudoul commented May 19, 2017

cmsbuild commented May 19, 2017

boudoul commented May 19, 2017

kpedro88 May 19, 2017

kpedro88 May 19, 2017

kpedro88 May 19, 2017

kpedro88 commented May 19, 2017

boudoul commented May 19, 2017

cmsbuild commented May 19, 2017

kpedro88 May 19, 2017

kpedro88 May 23, 2017

boudoul May 25, 2017

davidlange6 commented May 19, 2017 via email

kpedro88 commented May 19, 2017

ianna May 19, 2017

kpedro88 May 19, 2017

cmsbuild commented May 25, 2017

cmsbuild commented May 25, 2017

kpedro88 commented May 25, 2017

cmsbuild commented May 25, 2017 •

edited

cmsbuild commented May 25, 2017

cmsbuild commented May 25, 2017

cmsbuild commented May 25, 2017

kpedro88 commented May 25, 2017

ianna commented May 26, 2017

smuzaffar commented May 30, 2017

boudoul commented May 30, 2017

slava77 commented May 31, 2017

kpedro88 commented May 31, 2017

boudoul commented May 31, 2017

		for year in upgradeKeys:
		for i in range(1,len(upgradeKeys[year])):

Deprecating Phase2 Tracker T3 and corresp. scenarios + Adding D18 (= D8 but with the non deprec. Tracker) #18852

Deprecating Phase2 Tracker T3 and corresp. scenarios + Adding D18 (= D8 but with the non deprec. Tracker) #18852

Conversation

boudoul commented May 19, 2017

cmsbuild commented May 19, 2017

boudoul commented May 19, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kpedro88 commented May 19, 2017

boudoul commented May 19, 2017

cmsbuild commented May 19, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidlange6 commented May 19, 2017 via email

kpedro88 commented May 19, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cmsbuild commented May 25, 2017

cmsbuild commented May 25, 2017

kpedro88 commented May 25, 2017

cmsbuild commented May 25, 2017 • edited

cmsbuild commented May 25, 2017

cmsbuild commented May 25, 2017

cmsbuild commented May 25, 2017

kpedro88 commented May 25, 2017

ianna commented May 26, 2017

smuzaffar commented May 30, 2017

boudoul commented May 30, 2017

slava77 commented May 31, 2017

kpedro88 commented May 31, 2017

boudoul commented May 31, 2017

cmsbuild commented May 25, 2017 •

edited