Bugfix: Make Use of the 2019 Geometry from DB #24824

ianna · 2018-10-08T13:15:24Z

Both 2019 Geometry scenario and GEM reco geometry have been updated in the 2019 GTs in [10.3]Change Gts. Fix geometry in LS2 GTs, and new HI 2018 MC GT #24782
Replaces Bugfix: Configuration for 2019 Geometry #24768

There are issues in the local relval tests, for example:

11624.0 step2:

08-Oct-2018 14:56:53 CEST  Initiating request to open file file:step1.root
08-Oct-2018 14:56:54 CEST  Successfully opened file file:step1.root
Begin processing the 1st record. Run 1, Event 1, LumiSection 1 on stream 0 at 08-Oct-2018 14:57:05.964 CEST
%MSG-e L1T:  L1TRawToDigi:hltGtStage2Digis 08-Oct-2018 14:57:22 CEST  Run: 1 Event: 1
Cannot unpack: no FEDRawDataCollection found
%MSG
----- Begin Fatal Exception 08-Oct-2018 14:57:22 CEST-----------------------
An exception of category 'Out of Range' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'digitisation_step'
   [2] Calling method for module HcalTrigPrimDigiProducer/'simHcalTriggerPrimitiveDigis'
Exception Message:
LUT has 1024 entries for (HcalTrigTower v0: -4,36) but 1057 was requested.
----- End Fatal Exception -------------------------------------------------
Another exception was caught while trying to clean up files after the primary fatal exception.
08-Oct-2018 14:57:22 CEST  Closed file file:step1.root

ianna · 2018-10-08T13:15:43Z

please test

cmsbuild · 2018-10-08T13:15:43Z

The code-checks are being triggered in jenkins.

cmsbuild · 2018-10-08T13:17:36Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-24824/6774

cmsbuild · 2018-10-08T13:17:53Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/30924/console Started: 2018/10/08 15:18

cmsbuild · 2018-10-08T13:17:54Z

A new Pull Request was created by @ianna (Ianna Osborne) for master.

It involves the following packages:

Configuration/Geometry
Configuration/PyReleaseValidation

@cmsbuild, @prebello, @Dr15Jones, @cvuosalo, @civanch, @ianna, @mdhildreth, @pgunnell, @kpedro88, @zhenhu can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @felicepantaleo, @makortel this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

cmsbuild · 2018-10-08T15:53:19Z

-1

Tested at: a7fe6dc

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-24824/30924/summary.html

I found follow errors while testing this PR

Failed tests: RelVals

RelVals:

When I ran the RelVals I found an error in the following worklfows:
11624.0 step2

runTheMatrix-results/11624.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2019_GenSimFull+DigiFull_2019+RecoFull_2019+HARVESTFull_2019+ALCAFull_2019/step2_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2019_GenSimFull+DigiFull_2019+RecoFull_2019+HARVESTFull_2019+ALCAFull_2019.log

cmsbuild · 2018-10-08T15:53:21Z

Comparison not run due to runTheMatrix errors (RelVals and Igprof tests were also skipped)

kpedro88 · 2018-10-08T15:56:12Z

@abdoulline @christopheralanwest regarding the error observed:

LUT has 1024 entries for (HcalTrigTower v0: -4,36) but 1057 was requested.

Can you check the 2019 GT to see if some condition has to be updated for HCAL trigger?

christopheralanwest · 2018-10-08T16:52:27Z

@kpedro88 For concreteness, I will refer to the 103X_postLS2_realistic_v3 GT. Only the electronics map tag is updated from the 2018 tag so all conditions that reference upgraded HB channels need to be updated. Using the GT rather than hardcoded HCAL conditions is not supported at all for postLS2 scenarios.

I think that the error that you quote arises because the HcalParameters tag specified by the GT (HCALParameters_Geometry_91YV3) is a 2018 scenario [1]. This in turn specifies the 2018 trigger mode [2] rather than the the 2019 trigger mode [3]. Therefore LUTs with 1024 rather than 2048 entries are used for HB trigger towers [4].

[1] https://twiki.cern.ch/twiki/bin/view/CMSPublic/SWGuideDataBaseGeometry
[2]

cmssw/CalibCalorimetry/CaloTPG/src/CaloTPGTranscoderULUT.cc

Lines 236 to 242 in 02d4198

    
           case HcalTopologyMode::TriggerMode_2018: 
        
              if (id.ietaAbs() <= theTopology->lastHBRing()) 
        
                 return QIE8_OUTPUT_LUT_SIZE; 
        
              else if (id.ietaAbs() <= theTopology->lastHERing()) 
        
                 return QIE11_OUTPUT_LUT_SIZE; 
        
              else 
        
                 return QIE10_OUTPUT_LUT_SIZE;

[3]

cmssw/CalibCalorimetry/CaloTPG/src/CaloTPGTranscoderULUT.cc

Lines 243 to 247 in 02d4198

    
           case HcalTopologyMode::TriggerMode_2019: 
        
              if (id.ietaAbs() <= theTopology->lastHERing()) 
        
                 return QIE11_OUTPUT_LUT_SIZE; 
        
              else 
        
                 return QIE10_OUTPUT_LUT_SIZE;

[4]

cmssw/CalibCalorimetry/CaloTPG/src/CaloTPGTranscoderULUT.h

Lines 48 to 54 in 02d4198

    
           static const unsigned int REDUCE10BIT = 1024; 
        
           static const unsigned int REDUCE11BIT = 2048; 
        
           // Map different QIE to the right linearization 
        
           static const unsigned int QIE8_OUTPUT_LUT_SIZE = REDUCE10BIT; 
        
           static const unsigned int QIE10_OUTPUT_LUT_SIZE = REDUCE11BIT; 
        
           static const unsigned int QIE11_OUTPUT_LUT_SIZE = REDUCE11BIT;

abdoulline · 2018-10-09T05:50:22Z

@christopheralanwest
Thank you Chris for a prompt look at the issue.
I believe Kevin @kpedro88 knows well that we're still using mostly hardcode conditions for 2019/Run3, as he has largely modified those in the past:
http://cmslxr.fnal.gov/source/CalibCalorimetry/HcalPlugins/python/Hcal_Conditions_forGlobalTag_cff.py

Whatever (except emap) conditions are in GT - they are "place holders".

abdoulline · 2018-10-09T07:08:36Z

Just in case, I've dumped step 2 of the workflow in question 11624.0 as it's now in 10_3_0_pre5 (runs OK), so that Yana (@ianna) could look at the list of xml files (in the bottom of the dump) - if all these are included in the current PR?

/afs/cern.ch/user/a/abdullin/public/xml_Geometry_2019_11624.0/step2_dump.txt

ianna · 2018-10-09T09:42:12Z

@abdoulline - I've uploaded the 2019 HcalParameters tag (HCALParameters_Geometry_103YV5) and created a candidate GT queue: 103X_postLS2_design_Candidate_2018_10_09_09_19_28

Now the 11624.0 fails with a different message:

runTheMatrix.py -l 11624.0 --command "--conditions 103X_postLS2_design_Candidate_2018_10_09_09_19_28"

Begin processing the 1st record. Run 1, Event 1, LumiSection 1 on stream 0 at 09-Oct-2018 11:36:27.059 CEST
%MSG-e L1T:  L1TRawToDigi:hltGtStage2Digis 09-Oct-2018 11:36:27 CEST  Run: 1 Event: 1
Cannot unpack: no FEDRawDataCollection found
%MSG
----- Begin Fatal Exception 09-Oct-2018 11:36:27 CEST-----------------------
An exception of category 'Conditions mismatch' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'digitisation_step'
   [2] Calling method for module MixingModule/'mix'
Exception Message:
Requested conditions of type HcalQIETypes for cell (0x43404001) (HB -16,1,4) got conditions for cell (0x0) 
----- End Fatal Exception -------------------------------------------------

fabiocos · 2018-10-24T07:30:52Z

please test workflow 11624.0

cmsbuild · 2018-10-24T07:31:11Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/31246/console Started: 2018/10/24 09:32

cmsbuild · 2018-10-24T08:56:39Z

+1
Tested at: a7fe6dc
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-24824/31246/summary.html

cmsbuild · 2018-10-24T08:56:43Z

Comparison job queued.

cmsbuild · 2018-10-24T10:43:22Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-24824/31246/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 2469 differences found in the comparisons
DQMHistoTests: Total files compared: 32
DQMHistoTests: Total histograms compared: 2994773
DQMHistoTests: Total failures: 8360
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2986216
DQMHistoTests: Total skipped: 197
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -77.023 KiB( 31 files compared)
DQMHistoSizes: changed ( 11624.0 ): -76.513 KiB MuonGEMRecHitsV/GEMRecHitsTask
DQMHistoSizes: changed ( 11624.0 ): -0.511 KiB MuonGEMHitsV/GEMHitsTask
Checked 134 log files, 14 edm output root files, 32 DQM output files

kpedro88 · 2018-10-24T13:03:16Z

The TrackerHitsV plots are still showing unexpected discrepancies. @perrotta, are you sure these came out okay in your private test?

perrotta · 2018-10-24T13:51:31Z

@kpedro88 : yes, I confirm it.

The same plot as in https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_10_4_X_2018-10-23-2300+24824/29061/11624.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2019_GenSimFull+DigiFull_2019+RecoFull_2019+HARVESTFull_2019+ALCAFull_2019/TrackerHitsV__TrackerHit_TOBHit_Localy_TOB_2.png (jenkins tests, run with 10 events) looks as follows in my 70 events sample:

Just for test, I rerun locally with only 10 events, and I can reproduce exactly what was found by the automated tests:

This looks like being a matter of statistics: not all TOBs (in this example) are hit with only 10, events, while all them are hit at least once in 70 events. In detail:

Empty TOBs (10 events): 1, 2, 4, 6, 12
Empty TOBs (70 events): none

And evidently, the empty ones do not correspond with and without this PR. In the reference the empty TOBs (with 10 events) are: 3, 6, 10

kpedro88 · 2018-10-24T13:52:54Z

Ah, I see: it is just a very specific plot, so statistical fluctuations can leave it empty. Thanks for the followup, I am satisfied.

kpedro88 · 2018-10-24T13:53:01Z

+upgrade

kpedro88 · 2018-10-30T16:48:41Z

@pgunnell, @zhenhu, @prebello please sign

zhenhu · 2018-10-30T17:01:38Z

+1

cmsbuild · 2018-10-30T17:02:39Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

fabiocos · 2018-10-31T09:16:02Z

+1

Make use of the 2019 geometry from DB

a7fe6dc

cmsbuild added this to the CMSSW_10_3_X milestone Oct 8, 2018

cmsbuild added code-checks-pending comparison-pending geometry-pending orp-pending pdmv-pending pending-signatures tests-pending upgrade-pending labels Oct 8, 2018

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 8, 2018

cmsbuild added tests-started and removed tests-pending labels Oct 8, 2018

cmsbuild added comparison-notrun tests-rejected and removed comparison-pending tests-started labels Oct 8, 2018

cmsbuild added tests-pending and removed tests-approved labels Oct 24, 2018

cmsbuild added tests-started and removed tests-pending labels Oct 24, 2018

cmsbuild added tests-approved and removed tests-started labels Oct 24, 2018

cmsbuild added comparison-available and removed comparison-pending labels Oct 24, 2018

cmsbuild added upgrade-approved and removed upgrade-pending labels Oct 24, 2018

cmsbuild added fully-signed pdmv-approved and removed pdmv-pending pending-signatures labels Oct 30, 2018

cmsbuild added orp-approved and removed orp-pending labels Oct 31, 2018

cmsbuild merged commit 51ae380 into cms-sw:master Oct 31, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bugfix: Make Use of the 2019 Geometry from DB #24824

Bugfix: Make Use of the 2019 Geometry from DB #24824

ianna commented Oct 8, 2018

ianna commented Oct 8, 2018

cmsbuild commented Oct 8, 2018

cmsbuild commented Oct 8, 2018

cmsbuild commented Oct 8, 2018 •

edited

cmsbuild commented Oct 8, 2018

cmsbuild commented Oct 8, 2018

cmsbuild commented Oct 8, 2018

kpedro88 commented Oct 8, 2018

christopheralanwest commented Oct 8, 2018

abdoulline commented Oct 9, 2018 •

edited

abdoulline commented Oct 9, 2018 •

edited

ianna commented Oct 9, 2018

fabiocos commented Oct 24, 2018

cmsbuild commented Oct 24, 2018 •

edited

cmsbuild commented Oct 24, 2018

cmsbuild commented Oct 24, 2018

cmsbuild commented Oct 24, 2018

kpedro88 commented Oct 24, 2018

perrotta commented Oct 24, 2018

kpedro88 commented Oct 24, 2018

kpedro88 commented Oct 24, 2018

kpedro88 commented Oct 30, 2018

zhenhu commented Oct 30, 2018

cmsbuild commented Oct 30, 2018

fabiocos commented Oct 31, 2018

Bugfix: Make Use of the 2019 Geometry from DB #24824

Bugfix: Make Use of the 2019 Geometry from DB #24824

Conversation

ianna commented Oct 8, 2018

ianna commented Oct 8, 2018

cmsbuild commented Oct 8, 2018

cmsbuild commented Oct 8, 2018

cmsbuild commented Oct 8, 2018 • edited

cmsbuild commented Oct 8, 2018

cmsbuild commented Oct 8, 2018

cmsbuild commented Oct 8, 2018

kpedro88 commented Oct 8, 2018

christopheralanwest commented Oct 8, 2018

abdoulline commented Oct 9, 2018 • edited

abdoulline commented Oct 9, 2018 • edited

ianna commented Oct 9, 2018

fabiocos commented Oct 24, 2018

cmsbuild commented Oct 24, 2018 • edited

cmsbuild commented Oct 24, 2018

cmsbuild commented Oct 24, 2018

cmsbuild commented Oct 24, 2018

kpedro88 commented Oct 24, 2018

perrotta commented Oct 24, 2018

kpedro88 commented Oct 24, 2018

kpedro88 commented Oct 24, 2018

kpedro88 commented Oct 30, 2018

zhenhu commented Oct 30, 2018

cmsbuild commented Oct 30, 2018

fabiocos commented Oct 31, 2018

cmsbuild commented Oct 8, 2018 •

edited

abdoulline commented Oct 9, 2018 •

edited

abdoulline commented Oct 9, 2018 •

edited

cmsbuild commented Oct 24, 2018 •

edited