EMTF BIG emulator update - attempt #1 #527

abrinke1 · 2017-05-11T10:07:21Z

Full update of EMTF code, including:

Bit-wise emulation of 2016 firmware (CSC-only)
Incorporation of RPCs into emulator track-building, unpacking, and DQM

Crucial missing piece:

I removed the L1Trigger/L1TMuonEndCap/data/ [1] folder from @jiafulow , since I'm pretty sure this should be O2O'd. @kkotov , can you confirm? Can we get this in ASAP?
[1] https://github.com/abrinke1/cmssw/tree/EMTF_April_2017_dev/L1Trigger/L1TMuonEndCap/data

Items to be added:

2017 pT assignment with RPC hits
Additional DQM plots, quality tests

Major directories affected
DataFormats/L1TMuon/

Significant changes to internal EMTF data formats
EventFilter/L1TRawToDigi/
Unpack input link error flags and RPC hits
L1Trigger/L1TMuonEndCap/
Complete replacement of emulator code
L1Trigger/L1TMuon
Significant changes to GeometryTranslator.h, MuonTriggerPrimitive.h, and MuonTriggerPrimitiveFwd.h
Also, moved them out of deprecate/ folder

Unusual or unexpected file changes
DQM/L1TMonitor/src/L1TMP7ZeroSupp.cc
EventFilter/L1TRawToDigi/interface/Block.h

Appear to contain recent commits from @thomreis that got lost in the L1T Offline migration from l1t-integration-CMSSW_9_0_0 to l1t-integration-CMSSW_9_1_0_pre3

Geometry/GEMGeometry/test/ME0GeometryAnalyzer.cc
Geometry/GEMGeometryBuilder/
Geometry/MuonCommonData/
Geometry/MuonNumbering/src/ME0NumberingScheme.cc

Files related to GEM detectors. I think these changes are from @jiafulow , but GEM/Phase II experts will have to take a look to see which version is correct

Lastly, @rekovic , I still can't see our DQM developer Preston Epps (ICleanPools [2]) anywhere - for example, I can't add him to the Reviewers of this PR. Do you know why this is?
[2] https://github.com/ICleanPools

…configForTiltedTracker pr90x Configure TTStubAlgorithm from L1TrackTrigger sequence for TiltedTracker

update 2017 wf in IB: replace TenMuE with ZMM

Cleaning pixel certification to prevent crash at Tier0 [90X]

…CMSSW_9_0_0_pre4 Reject 0 pt Puppi candidates in jet-track association

Convert Validation/RecoTau modules from legacy to global producers

Resolve segfault in MuonGEMHitsHarvestor::dqmEndJob

…for-9_0 Ignoring calibration channels in the dataframes

HBHE M2 = fix Memory Leak (90X)

Various fixes: conddb core, conddb tools, runinfo o2o

thomreis · 2017-05-11T12:13:53Z

@abrinke1 there are still a lot of other commits in the PR.

abrinke1 · 2017-05-11T12:19:31Z

@thomreis I'm aware of this: the changes were initially made on top of the l1t-integration-CMSSW_9_0_0 branch, and I spent the last day rebasing onto l1t-integration-CMSSW_9_1_0_pre3. You can see that there are ~300 files changed, of which > 250 are in the L1Trigger/L1TMuonEndCap/ area. These do not need to be tracked, since it's a total replacement. All the other changed files have been noted explicitly in my comments above.

thomreis · 2017-05-11T12:42:09Z

The DQM/L1TMonitor/src/L1TMP7ZeroSupp.cc and EventFilter/L1TRawToDigi/interface/Block.h src/Block.cc changes seem to have been lost in the transition to the 9_1_0_pre3 integration branch indeed. However, the corresponding PR cms-sw#17956 is now merged in the official CMSSW 91x.

rekovic · 2017-05-16T15:04:00Z

This is a huge number of files!

abrinke1 · 2017-05-16T15:16:41Z

Yep - hence the ALL CAPS :)

thomreis

A few additional comments:

I think the emulator should be in the l1t namespace.
header files have the .hh ending compared to .h for most other packages. Is there a rule for this?
How does the changed unpacker handle data from 2016? Does this work?
I did not go through the new unpacker code in detail since I'm not familiar with the EMTF algorithm.

thomreis · 2017-05-17T12:20:06Z

Configuration/Geometry/python/dict2023Geometry.py

            'Geometry/DTGeometryBuilder/data/dtSpecsFilter.xml',
            'Geometry/CSCGeometryBuilder/data/cscSpecsFilter.xml',
            'Geometry/CSCGeometryBuilder/data/cscSpecs.xml',
            'Geometry/RPCGeometryBuilder/data/PhaseII/RPCSpecs.xml',
-            'Geometry/GEMGeometryBuilder/data/v7/GEMSpecsFilter.xml',
-            'Geometry/GEMGeometryBuilder/data/v7/GEMSpecs.xml',


Decrease in version number?

thomreis · 2017-05-17T12:37:28Z

DataFormats/L1TMuon/interface/EMTFHit.h

+    int   is_CSC      ; //  0 or 1.
+    int   is_RPC      ; //  0 or 1.
+    int   is_GEM      ; //  0 or 1.
+    int   subsystem   ; //  1 or ?.  1 for CSC, 2 for RPC, 3 for GEM


A couple of question marks remain to be resolved.

These comments are for documentation, I need to check them manually to be sure before removing question marks. Prefer to keep them as is for now, as a "to do" reminder.

thomreis · 2017-05-17T12:45:37Z

DataFormats/L1TMuon/interface/EMTFHit.h

+      phi_fp(-99), theta_fp(-99), phzvl(-99), ph_hit(-99), zone_hit(-99), zone_code(-99),
+      fs_segment(-99), fs_zone_code(-99), bt_station(-99), bt_segment(-99),
+      phi_loc(-99), phi_glob(-99), theta(-99), eta(-99),
+      phi_sim(-99), theta_sim(-99), eta_sim(-99),


phi_glob(-99) and phi_sim(-99) are allowed values.

thomreis · 2017-05-17T12:49:30Z

DataFormats/L1TMuon/interface/EMTFTrack.h

+      rank(-99), winner(-99), charge(-99), bx(-99), first_bx(-99), second_bx(-99),
+      pt(-99), pt_XML(-99), zone(-99), ph_num(-99), ph_q(-99),
+      theta_fp(-99), theta(-99), eta(-99), phi_fp(-99), phi_loc(-99), phi_glob(-99),
+      gmt_pt(-99), gmt_phi(-99), gmt_eta(-99), gmt_quality(-99), gmt_charge(-99), gmt_charge_valid(-99),


phi_glob(-99) is an allowed value. Check also gmt_phi(-99) and gmt_eta(-99)

thomreis · 2017-05-17T12:51:34Z

DataFormats/L1TMuon/src/EMTFHit.cc

+
+  CSCDetId EMTFHit::CreateCSCDetId() const {
+    return CSCDetId( (endcap == 1) ? 1 : 2, station,    // For now, leave "layer" unfilled, defaults to 0.
+         (ring == 4) ? 1 : ring, chamber ); // Not sure if this is correct, or what "layer" does. - AWB 27.04.16


Is it correct?

@abrinke1 layer should be 0, e.g. https://github.com/cms-sw/cmssw/blob/master/L1Trigger/CSCTriggerPrimitives/src/CSCTriggerPrimitivesBuilder.cc#L198-L199.

thomreis · 2017-05-17T13:37:54Z

Geometry/CMSCommonData/python/cmsExtendedGeometry2023D10XML_cfi.py

-        'Geometry/GEMGeometryBuilder/data/v7/GEMSpecsFilter.xml',
-        'Geometry/GEMGeometryBuilder/data/v7/GEMSpecs.xml',
+        'Geometry/GEMGeometryBuilder/data/GEMSpecsFilter.xml',
+        'Geometry/GEMGeometryBuilder/data/v5/GEMSpecs.xml',


Decreasing version number?

thomreis · 2017-05-17T13:38:12Z

Geometry/CMSCommonData/python/cmsExtendedGeometry2023D11XML_cfi.py

-        'Geometry/GEMGeometryBuilder/data/v7/GEMSpecsFilter.xml',
-        'Geometry/GEMGeometryBuilder/data/v7/GEMSpecs.xml',
+        'Geometry/GEMGeometryBuilder/data/GEMSpecsFilter.xml',
+        'Geometry/GEMGeometryBuilder/data/v5/GEMSpecs.xml',


Decreasing version number?

thomreis · 2017-05-17T13:38:19Z

Geometry/CMSCommonData/python/cmsExtendedGeometry2023D4XML_cfi.py

-        'Geometry/GEMGeometryBuilder/data/v7/GEMSpecsFilter.xml',
-        'Geometry/GEMGeometryBuilder/data/v7/GEMSpecs.xml',
+        'Geometry/GEMGeometryBuilder/data/GEMSpecsFilter.xml',
+        'Geometry/GEMGeometryBuilder/data/v5/GEMSpecs.xml',


Decreasing version number?

thomreis · 2017-05-17T13:38:26Z

Geometry/CMSCommonData/python/cmsExtendedGeometry2023D7XML_cfi.py

-        'Geometry/GEMGeometryBuilder/data/v7/GEMSpecsFilter.xml',
-        'Geometry/GEMGeometryBuilder/data/v7/GEMSpecs.xml',
+        'Geometry/GEMGeometryBuilder/data/GEMSpecsFilter.xml',
+        'Geometry/GEMGeometryBuilder/data/v5/GEMSpecs.xml',


Decreasing version number?

thomreis · 2017-05-17T13:38:36Z

Geometry/CMSCommonData/python/cmsExtendedGeometry2023D8XML_cfi.py

-        'Geometry/GEMGeometryBuilder/data/v7/GEMSpecsFilter.xml',
-        'Geometry/GEMGeometryBuilder/data/v7/GEMSpecs.xml',
+        'Geometry/GEMGeometryBuilder/data/GEMSpecsFilter.xml',
+        'Geometry/GEMGeometryBuilder/data/v5/GEMSpecs.xml',


Decreasing version number?

fwyzard · 2017-05-18T00:36:23Z

header files have the .hh ending compared to .h for most other packages. Is there a rule for this?

CMSSW coding rules use .h for header files: https://cms-cpt-software.web.cern.ch/cms-cpt-software/General/CodingAndStyleRules.pdf , § 2.1

thomreis · 2017-05-18T07:03:44Z

@fwyzard interesting document. I did not know it existed. It could be advertised more for people making PRs :-)

thomreis · 2017-05-19T13:25:46Z

I get the exception:

----- Begin Fatal Exception 19-May-2017 15:14:46 CEST-----------------------
An exception of category 'FileInPathError' occurred while
   [0] Processing  Event run: 1 lumi: 53153 event: 10630401 stream: 0
   [1] Running path 'L1TReEmulPath'
   [2] Calling method for module L1TMuonEndCapTrackProducer/'simEmtfDigis'
   [3] Using EventSetup component L1TMuonEndCapForestESProducer/'emtfForests' to make data L1TMuonEndCapForest/'' in record L1TMuonEndCapForestRcd
Exception Message:
edm::FileInPath unable to find file L1Trigger/L1TMuonEndCap/data/emtf_luts/v_16_02_21/ModeVariables/trees/3/0.xml anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /afs/cern.ch/work/t/treis/ugmt/tests/new_emtf/CMSSW_9_1_0_pre3/src:/afs/cern.ch/work/t/treis/ugmt/tests/new_emtf/CMSSW_9_1_0_pre3/external/slc6_amd64_gcc530/data:/cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_9_1_0_pre3/src:/cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_9_1_0_pre3/external/slc6_amd64_gcc530/data
Current directory is: /afs/cern.ch/work/t/treis/ugmt/tests/new_emtf/CMSSW_9_1_0_pre3/src
----- End Fatal Exception -------------------------------------------------

@abrinke1 you mentioned that you removed the directory. But there is also a emtf_luts directory in L1Trigger/L1TMuon/data that appears to contain the missing file. How are the two related?

… output

abrinke1 · 2017-05-20T23:50:44Z

Hi all,

Thanks for all the helpful comments. I haven't responded so far because I was working on getting the EMTF pT assignment with RPCs included, since this is what will be running at P5 when the collisions arrive. You can see the preliminary version in my latest commits (with a fair number of debugging output, and some temporary hacks). I'll be working with @kkotov on Monday to get things all ship-shape with O2O and versioning, and I'll try to address the comments above as well.

Best regards,
Andrew

dildick · 2017-05-21T22:52:34Z

@abrinke1 please do not modify the GEM geometries in this pull request. This is taken care of by @bsunanda.

…aning)

abrinke1 · 2017-05-22T17:17:57Z

Hi all,

I've just pushed a working version of the 2017 pT assignment with RPCs, with bit-wise internal debugging on 10k events. The performance looks pretty good. Took me (and Khristian) a bit longer than I expected, so the code cleaning will have to wait for tomorrow.

Best,
Andrew

…-offline/cmssw into abrinke1/EMTF_April_2017_merge

abrinke1 · 2017-05-24T16:16:09Z

THIS PR IS READY TO BE TESTED!

@thomreis is there any chance you can try running again, either on a T&P data sample or some MC, and cross-check that the efficiency is OK?

@rekovic @bortigno @kkotov @jiafulow please take a look

p.s. Just remembered that I didn't address the ".hh" vs. ".h" comment - can deal with this before pulling into CMSSW.

thomreis · 2017-05-26T11:34:23Z

@abrinke1 I will look at #542 instead.

thomreis · 2017-06-16T10:45:44Z

Superseded by #542

mariadalfonso and others added 30 commits March 9, 2017 01:17

use std::unique_ptr

ba6a953

Various fixes: conddb core, conddb tools, runinfo o2o

8562ded

Update HLT STORM validation to Phase1

757f278

Merged refs/pull/17853/head from repository cms-sw

774355b

update Fake menus

6ae229d

Removing pixel cert

495becd

update 2017 wf in IB: replace TenMuE with ZMM

0e0375a

fix HLT jenkins and relval menu

e2305c9

fix era, again

50ffbe3

fix era, again

0faffc9

Merge pull request cms-sw#17824 from rekovic/pr90x-TTrackStubCluster-…

e691242

…configForTiltedTracker pr90x Configure TTStubAlgorithm from L1TrackTrigger sequence for TiltedTracker

Fix for empty histograms

ec53543

Merged PhaseIHLTvalidation from repository cms-tsg-storm

6f15efd

Add Fake2 addOn test

7c0993f

FastSim2 test back to 2016 era

640397f

Assorted fixes and developments

35b1913

Remove ALCAELECTRONOutput endpath

b9c49aa

resynch

d253603

add backwards support for previous calo params version

5f7f240

Make HCAL LutMetadata more realistic

5227f7d

Fix eta range for HCAL LUT metadata

7185707

Merge pull request cms-sw#17858 from fabozzi/from-CMSSW_9_0_X_forIB

bea4be5

update 2017 wf in IB: replace TenMuE with ZMM

Merge pull request cms-sw#17856 from boudoul/pixelcert_90X

b1ba664

Cleaning pixel certification to prevent crash at Tier0 [90X]

Merge pull request cms-sw#17626 from cms-btv-pog/JTAforPuppi-PR_from-…

95d3b85

…CMSSW_9_0_0_pre4 Reject 0 pt Puppi candidates in jet-track association

Merge pull request cms-sw#17656 from knoepfel/convert-RecoTauModules

429c196

Convert Validation/RecoTau modules from legacy to global producers

Merge pull request cms-sw#17759 from davidlt/fix-dqm-segfault-labels

620af41

Resolve segfault in MuonGEMHitsHarvestor::dqmEndJob

Merge pull request cms-sw#17780 from igv4321/hcal-subdet-restriction-…

fbff60f

…for-9_0 Ignoring calibration channels in the dataframes

Merge pull request cms-sw#17822 from mariadalfonso/fixMemLeakM2_90

337d390

HBHE M2 = fix Memory Leak (90X)

Merge pull request cms-sw#17845 from ggovi/condcore-fixes-0

ac72983

Various fixes: conddb core, conddb tools, runinfo o2o

make workflow injection ready for Unified

186d4b2

abrinke1 requested review from jiafulow, kkotov, bortigno and thomreis May 11, 2017 12:04

EMTF corrected RPC phi in emulator, added track mode for CSC and RPC

f24962c

EMTF small fix to emulated RPC hit phi and theta

4bcd5dc

thomreis reviewed May 17, 2017

View reviewed changes

abrinke1 added 2 commits May 21, 2017 01:31

EMTF initial 2017 pT assignment with RPCs, still some hacks and debug…

3089073

… output

EMTF two versions of coordinate LUTs: v1 for 2016, v2 for 2017

fd74ff6

EMTF full debugged 2017 pT assignment with RPCs (code still needs cle…

b502815

…aning)

abrinke1 added 4 commits May 24, 2017 11:58

EMTF fixed bug accessing 2017 pT XMLs

3254f27

Merge branch 'l1t-integration-CMSSW_9_1_0_pre3' of github.com:cms-l1t…

41bef3e

…-offline/cmssw into abrinke1/EMTF_April_2017_merge

EMTF first full, debugged, validated emulation of 2017 pT assignment

f6b1b97

EMTF removed bad printout statement

87f81e9

abrinke1 mentioned this pull request May 26, 2017

EMTF BIG emulator update - attempt #2 #542

Merged

thomreis closed this Jun 16, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EMTF BIG emulator update - attempt #1 #527

EMTF BIG emulator update - attempt #1 #527

abrinke1 commented May 11, 2017 •

edited

thomreis commented May 11, 2017

abrinke1 commented May 11, 2017

thomreis commented May 11, 2017

rekovic commented May 16, 2017

abrinke1 commented May 16, 2017

thomreis left a comment

thomreis May 17, 2017

thomreis May 17, 2017

abrinke1 May 24, 2017

thomreis May 17, 2017

abrinke1 May 24, 2017

thomreis May 17, 2017

abrinke1 May 24, 2017

thomreis May 17, 2017

dildick May 21, 2017

abrinke1 May 24, 2017

thomreis May 17, 2017

thomreis May 17, 2017

thomreis May 17, 2017

thomreis May 17, 2017

thomreis May 17, 2017

fwyzard commented May 18, 2017

thomreis commented May 18, 2017

thomreis commented May 19, 2017

abrinke1 commented May 20, 2017

dildick commented May 21, 2017

abrinke1 commented May 22, 2017

abrinke1 commented May 24, 2017

thomreis commented May 26, 2017

thomreis commented Jun 16, 2017

EMTF BIG emulator update - attempt #1 #527

EMTF BIG emulator update - attempt #1 #527

Conversation

abrinke1 commented May 11, 2017 • edited

thomreis commented May 11, 2017

abrinke1 commented May 11, 2017

thomreis commented May 11, 2017

rekovic commented May 16, 2017

abrinke1 commented May 16, 2017

thomreis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fwyzard commented May 18, 2017

thomreis commented May 18, 2017

thomreis commented May 19, 2017

abrinke1 commented May 20, 2017

dildick commented May 21, 2017

abrinke1 commented May 22, 2017

abrinke1 commented May 24, 2017

thomreis commented May 26, 2017

thomreis commented Jun 16, 2017

abrinke1 commented May 11, 2017 •

edited