EMTF unpacker problems

[thomreis]

After the fix of the EMTF unpacker with cms-sw#22090 there are still mismatches seen in cosmics mode.
The mismatches are mostly in the hwSign and the track address (key8: kTrkNum).

Example from run 309360 (Coll1 is EMTF, Coll2 is uGMT input from EMTF):

13-Feb-2018 11:34:38 CET  Successfully opened file root://eoscms.cern.ch//eos/cms/tier0/store/data/Commissioning2018/Cosmics/RAW/v1/000/309/360/00000/66701400-640D-E811-A07B-02163E01A1C6.root
...
Begin processing the 12th record. Run 309360, Event 21875234, LumiSection 1142 at 13-Feb-2018 11:36:59.511 CET
Track address: BX=0, key=8, coll 1 value=0, coll 2 value=2 MISMATCH
Coll 1 muon: BX=0, hwPt=511, hwEta=-153, hwPhi=60, hwSign=0, hwSignValid=1, hwQual=0, link=70, processor=4, trackFinderType=3
Coll 2 muon: BX=0, hwPt=511, hwEta=-153, hwPhi=60, hwSign=1, hwSignValid=1, hwQual=0, link=70, processor=4, trackFinderType=3
Begin processing the 13th record. Run 309360, Event 21875237, LumiSection 1142 at 13-Feb-2018 11:36:59.601 CET
Begin processing the 14th record. Run 309360, Event 21876234, LumiSection 1142 at 13-Feb-2018 11:36:59.691 CET
Begin processing the 15th record. Run 309360, Event 21881633, LumiSection 1142 at 13-Feb-2018 11:36:59.781 CET
Track address: BX=0, key=8, coll 1 value=0, coll 2 value=2 MISMATCH
Coll 1 muon: BX=0, hwPt=511, hwEta=-135, hwPhi=10, hwSign=0, hwSignValid=1, hwQual=0, link=71, processor=5, trackFinderType=3
Coll 2 muon: BX=0, hwPt=511, hwEta=-135, hwPhi=10, hwSign=1, hwSignValid=1, hwQual=0, link=71, processor=5, trackFinderType=3

@abrinke1 , @jiafulow

[abrinke1]

Hi @thomreis
Do you see any particular pattern with the charge, like uGMT unpacked always 1, or EMTF unpacked always 0, or always opposite, for tracks with hwQual = 0? This bit is just directly unpacked from the EMTF DAQ output, so it basically has to be a firmware error in either the DAQ output or EMTF-to-uGMT transmission.

Also, for kTrkNum, is the uGMT unpacked value always "2" for tracks with hwQual = 0? Or do you see a range of values?

Thanks, Andrew

[thomreis]

Hi @abrinke1
All the mismatching events ar like the ones I have posted. uGMT hwSign is 1 and EMTF hwSign is 0. The track address values are also always 0 and 2 for EMTF and uGMT, respectively.
They are not necessarily correlated, although a track address mismatch is also always present when a hwSign mismatch is there. Not the other way around.

[thomreis]

The hwPt for the hwSign mismatches is either 510 or 511 in all the events I have checked.

[abrinke1]

Hi @thomreis

The track number issue comes from single-LCT tracks, which are used only in cosmics at the moment. I've fixed that here:
abrinke1@516605b

The hwSign issue appears to be connected to the signed/unsigned pT issue causing the hole in cosmics. I take it this only appears in the negative endcap? Alex will be fixing this in the firmware soon, so hopefully it will take care of the hwSign issue as well. Perhaps we can wait and see in MWGR#3 before making another PR just for these 2 lines - and besides, we might find other lower-frequency mismatches that can be easily fixed.

Cheers, Andrew

[thomreis]

Hi @abrinke1

I saw that there was a FW upgrade yesterday. We can wait to see if this fixes the issues. The we can make a PR with the commit you have pointed two.
There are some more mismatches for other variables as well but they are below the 1% level. It is going to be easier to pin them down once the high frequency ones are gone. I think there are at least these BX shift issues still present.

Regards, Thomas

[thomreis]

Hi @abrinke1

It seems the new FW does not resolve the problems.
https://cmsweb.cern.ch/dqm/online/start?runnr=310087;dataset=/Global/Online/ALL;sampletype=online_data;filter=all;referencepos=overlay;referenceshow=customise;referencenorm=True;referenceobj1=refobj;referenceobj2=none;referenceobj3=none;referenceobj4=none;search=;striptype=object;stripruns=;stripaxis=run;stripomit=none;workspace=Everything;size=M;root=L1T/L1TStage2uGMT/EMTFoutput_vs_uGMTinput;focus=L1T/L1TStage2uGMT/EMTFoutput_vs_uGMTinput/mismatchRatio;zoom=yes;

Regards,
Thomas

[abrinke1]

OK, so it's a sign mismatch on ~73% of muons, not just those with pT = 511, right? If that's the case, I can follow up with Alex.

[thomreis]

There seem to be not many hwPt = 511 tracks anymore in the mismatching tracks. This looks unrelated to hwPt.

[thomreis]

The EMTF charge distribution at the uGMT inputs is very different from the positive and the negative detector side.

negative:
https://cmsweb.cern.ch/dqm/online/start?runnr=310087;dataset=/Global/Online/ALL;sampletype=online_data;filter=all;referencepos=overlay;referenceshow=customise;referencenorm=True;referenceobj1=refobj;referenceobj2=none;referenceobj3=none;referenceobj4=none;search=;striptype=object;stripruns=;stripaxis=run;stripomit=none;workspace=Everything;size=M;root=L1T/L1TStage2uGMT/intermediate_muons/EMTF_neg;focus=L1T/L1TStage2uGMT/intermediate_muons/EMTF_neg/ugmtMuonCharge;zoom=yes;

positive:
https://cmsweb.cern.ch/dqm/online/start?runnr=310087;dataset=/Global/Online/ALL;sampletype=online_data;filter=all;referencepos=overlay;referenceshow=customise;referencenorm=True;referenceobj1=refobj;referenceobj2=none;referenceobj3=none;referenceobj4=none;search=;striptype=object;stripruns=;stripaxis=run;stripomit=none;workspace=Everything;size=M;root=L1T/L1TStage2uGMT/intermediate_muons/EMTF_pos;focus=L1T/L1TStage2uGMT/intermediate_muons/EMTF_pos/ugmtMuonCharge;zoom=yes;

[abrinke1]

@thomreis
As of run 310408, new EMTF FW has fixed the charge mismatches. [1] Shall we proceed with the PR of the EMTF unpacker fix for single-LCT "kTrkNum"? [2]

[1] https://cmsweb.cern.ch/dqm/online/start?runnr=0;dataset=/Global/Online/ALL;sampletype=live;filter=all;referencepos=overlay;referenceshow=customise;referencenorm=True;referenceobj1=refobj;referenceobj2=none;referenceobj3=none;referenceobj4=none;search=;striptype=object;stripruns=;stripaxis=run;stripomit=none;workspace=Everything;size=L;root=L1T/L1TStage2uGMT/EMTFoutput_vs_uGMTinput;focus=L1T/L1TStage2uGMT/EMTFoutput_vs_uGMTinput/errorSummaryNum;zoom=yes;

[2] cms-sw#22325

[thomreis]

Hi @abrinke1
with the improvements in cms-sw#22325 the track address mismatches between EMTF and uGMT goes down to below 2%. Listed below are examples for remaining mismatches.

27-Feb-2018 17:48:06 CET  Successfully opened file root://eoscms.cern.ch//eos/cms/tier0/store/data/Commissioning2018/Cosmics/RAW/v1/000/310/449/00000/440C9794-7119-E811-911E-02163E0145C4.root

Begin processing the 244th record. Run 310449, Event 8350764, LumiSection 420 at 27-Feb-2018 17:48:46.211 CET
Track address: BX=0, key=3, coll 1 value=4, coll 2 value=0 MISMATCH
Coll 1 muon: BX=0, hwPt=10, hwEta=130, hwPhi=80, hwSign=0, hwSignValid=1, hwQual=14, link=40, processor=4, trackFinderType=4
Coll 2 muon: BX=0, hwPt=10, hwEta=130, hwPhi=80, hwSign=0, hwSignValid=1, hwQual=14, link=40, processor=4, trackFinderType=4
Track address: BX=0, key=3, coll 1 value=2, coll 2 value=0 MISMATCH
Coll 1 muon: BX=0, hwPt=8, hwEta=130, hwPhi=-16, hwSign=0, hwSignValid=1, hwQual=8, link=41, processor=5, trackFinderType=4
Coll 2 muon: BX=0, hwPt=8, hwEta=130, hwPhi=-16, hwSign=0, hwSignValid=1, hwQual=8, link=41, processor=5, trackFinderType=4

Begin processing the 345th record. Run 310449, Event 8350746, LumiSection 420 at 27-Feb-2018 17:41:51.495 CET
Track address: BX=0, key=3, coll 1 value=4, coll 2 value=0 MISMATCH
Track address: BX=0, key=5, coll 1 value=4, coll 2 value=0 MISMATCH
Coll 1 muon: BX=0, hwPt=15, hwEta=110, hwPhi=82, hwSign=0, hwSignValid=1, hwQual=15, link=40, processor=4, trackFinderType=4
Coll 2 muon: BX=0, hwPt=15, hwEta=110, hwPhi=82, hwSign=0, hwSignValid=1, hwQual=15, link=40, processor=4, trackFinderType=4
Track address: BX=0, key=3, coll 1 value=2, coll 2 value=0 MISMATCH
Track address: BX=0, key=5, coll 1 value=2, coll 2 value=0 MISMATCH
Coll 1 muon: BX=0, hwPt=14, hwEta=110, hwPhi=-14, hwSign=0, hwSignValid=1, hwQual=14, link=41, processor=5, trackFinderType=4
Coll 2 muon: BX=0, hwPt=14, hwEta=110, hwPhi=-14, hwSign=0, hwSignValid=1, hwQual=14, link=41, processor=5, trackFinderType=4

Begin processing the 1126th record. Run 310449, Event 8348998, LumiSection 420 at 27-Feb-2018 17:51:45.449 CET
Track address: BX=0, key=9, coll 1 value=2026, coll 2 value=1494 MISMATCH
Coll 1 muon: BX=0, hwPt=8, hwEta=92, hwPhi=54, hwSign=0, hwSignValid=1, hwQual=0, link=41, processor=5, trackFinderType=4
Coll 2 muon: BX=0, hwPt=8, hwEta=92, hwPhi=54, hwSign=0, hwSignValid=1, hwQual=0, link=41, processor=5, trackFinderType=4

Begin processing the 1814th record. Run 310449, Event 8355146, LumiSection 420 at 27-Feb-2018 17:52:31.849 CET
Track address: BX=0, key=9, coll 1 value=2039, coll 2 value=1507 MISMATCH
Coll 1 muon: BX=0, hwPt=7, hwEta=-117, hwPhi=51, hwSign=1, hwSignValid=1, hwQual=0, link=71, processor=5, trackFinderType=3
Coll 2 muon: BX=0, hwPt=7, hwEta=-117, hwPhi=51, hwSign=1, hwSignValid=1, hwQual=0, link=71, processor=5, trackFinderType=3

As usual coll1 is EMTF and coll2 is uGMT

[thomreis]

From the limited data one could conclude that there are two types of mismatches: One involving two muons and another one involving one muon. The first type could be duplicates.

[thomreis]

From an email by Alex Madorsky:

There are two different track "addresses":

• one is reported in DAQ stream for each track. This includes all track stubs IDs that have been used >in each track
• another is reported in the datastream that comes from EMTF to uGMT. This address includes only >stub IDs needed for cancellation in uGMT

Could the type 1 mismatches come from this? The EMTF unpacker builds a track address with the first scheme and the uGMT with the second?

[thomreis]

If my previous comment is correct then there should be many more of the type 1 mismatches in a pp dataset than in the cosmics dataset used before since there are more duplicate muons.

27-Feb-2018 19:52:49 CET  Successfully opened file root://eoscms.cern.ch//eos/cms/tier0/store/data/Run2017F/SingleMuon/RAW/v1/000/306/459/00000/C6F9F5A2-9DC5-E711-BF94-02163E01A609.root
Begin processing the 1st record. Run 306459, Event 95381599, LumiSection 59 at 27-Feb-2018 19:52:51.285 CET
Begin processing the 2nd record. Run 306459, Event 96619099, LumiSection 59 at 27-Feb-2018 19:52:53.093 CET
Begin processing the 3rd record. Run 306459, Event 96942047, LumiSection 59 at 27-Feb-2018 19:52:53.126 CET
Begin processing the 4th record. Run 306459, Event 96518067, LumiSection 59 at 27-Feb-2018 19:52:53.152 CET
Begin processing the 5th record. Run 306459, Event 96303736, LumiSection 59 at 27-Feb-2018 19:52:53.181 CET
Track address: BX=0, key=3, coll 1 value=3, coll 2 value=0 MISMATCH
Track address: BX=0, key=7, coll 1 value=3, coll 2 value=0 MISMATCH
Coll 1 muon: BX=0, hwPt=111, hwEta=-180, hwPhi=81, hwSign=1, hwSignValid=1, hwQual=13, link=66, processor=0, trackFinderType=3, raw data word 0x0
Coll 2 muon: BX=0, hwPt=111, hwEta=-180, hwPhi=81, hwSign=1, hwSignValid=1, hwQual=13, link=66, processor=0, trackFinderType=3, raw data word 0x3eb0000328a99a6f
Track address: BX=0, key=3, coll 1 value=1, coll 2 value=0 MISMATCH
Track address: BX=0, key=7, coll 1 value=1, coll 2 value=0 MISMATCH
Coll 1 muon: BX=0, hwPt=117, hwEta=-180, hwPhi=-15, hwSign=1, hwSignValid=1, hwQual=13, link=67, processor=1, trackFinderType=3, raw data word 0x0
Coll 2 muon: BX=0, hwPt=117, hwEta=-180, hwPhi=-15, hwSign=1, hwSignValid=1, hwQual=13, link=67, processor=1, trackFinderType=3, raw data word 0x3eb0000b78a99a75
Track address: BX=0, key=4, coll 1 value=1, coll 2 value=0 MISMATCH
Track address: BX=0, key=5, coll 1 value=3, coll 2 value=0 MISMATCH
Coll 1 muon: BX=0, hwPt=7, hwEta=-196, hwPhi=95, hwSign=1, hwSignValid=1, hwQual=10, link=67, processor=1, trackFinderType=3, raw data word 0x0
Coll 2 muon: BX=0, hwPt=7, hwEta=-196, hwPhi=95, hwSign=1, hwSignValid=1, hwQual=10, link=67, processor=1, trackFinderType=3, raw data word 0xbeb40023afa79407
Track address: BX=0, key=4, coll 1 value=1, coll 2 value=0 MISMATCH
Track address: BX=0, key=5, coll 1 value=1, coll 2 value=0 MISMATCH
Coll 1 muon: BX=0, hwPt=7, hwEta=-196, hwPhi=-1, hwSign=1, hwSignValid=1, hwQual=10, link=68, processor=2, trackFinderType=3, raw data word 0x0
Coll 2 muon: BX=0, hwPt=7, hwEta=-196, hwPhi=-1, hwSign=1, hwSignValid=1, hwQual=10, link=68, processor=2, trackFinderType=3, raw data word 0x3eb0000b7fa79407
Begin processing the 6th record. Run 306459, Event 96309575, LumiSection 59 at 27-Feb-2018 19:52:53.212 CET
Track address: BX=0, key=3, coll 1 value=3, coll 2 value=0 MISMATCH
Track address: BX=0, key=5, coll 1 value=3, coll 2 value=0 MISMATCH
Track address: BX=0, key=7, coll 1 value=3, coll 2 value=0 MISMATCH
Coll 1 muon: BX=0, hwPt=92, hwEta=210, hwPhi=80, hwSign=0, hwSignValid=1, hwQual=15, link=40, processor=4, trackFinderType=4, raw data word 0x0
Coll 2 muon: BX=0, hwPt=92, hwEta=210, hwPhi=80, hwSign=0, hwSignValid=1, hwQual=15, link=40, processor=4, trackFinderType=4, raw data word 0x9900022281a5e5c
Track address: BX=0, key=3, coll 1 value=1, coll 2 value=0 MISMATCH
Track address: BX=0, key=5, coll 1 value=1, coll 2 value=0 MISMATCH
Track address: BX=0, key=7, coll 1 value=1, coll 2 value=0 MISMATCH
Coll 1 muon: BX=0, hwPt=92, hwEta=210, hwPhi=-16, hwSign=0, hwSignValid=1, hwQual=15, link=41, processor=5, trackFinderType=4, raw data word 0x0
Coll 2 muon: BX=0, hwPt=92, hwEta=210, hwPhi=-16, hwSign=0, hwSignValid=1, hwQual=15, link=41, processor=5, trackFinderType=4, raw data word 0x990000a781a5e5c

This seems to be the case.