Add Hybrid Format SiStrip Zero Suppression algorithm for 2018 PbPb data-taking

[CesarBernardes]

Introduces Hybrid Format SiStrip Zero Suppresion for HI 2018 data-taking.
The code has the double utility of:

1. Emulates hybrid format: it takes as input VirginRaw data and it coverts to Zero Suppressed 10 bits hybrid format as expected from the tracker FEDs.

2. Performs hybrid data zero suppression: takes Zero Suppressed 10 bits hybrid format and converts it to 8 bits zero suppressed data

@mmusich @echabert @pieterdavid @icali @abaty @mandrenguyen

[cmsbuild]

Comparison job queued.

[pieterdavid]

On Wed, 5 Sep 2018, 23:34 Slava Krutelyov, ***@***.***> wrote: Is this SiStripAPVRestorer::cleaner_LocalMinimumAdder contributing to the digi step

Yes (it is called for APVs where the BaselineFollower finds a nonflat baseline)

If this is addressing the regression in the old/classic ZS (what's called by the siStripZeroSuppression module), then perhaps one really has to run this VR -> ZS repacking step a la HLT for HI. This step is not present in any of the existing relval workflows. This could explain why it escaped our attention until I was remaking this step for this PR tests.

It should indeed be the cause of those differences. The only realistic way to test this code path in the relvals is by running the ZS on VR (or hybrid) HI data (nonflat baselines are negligibly rare in pp), so I agree it's a good idea to add this step (but we should also have checked this specifically while preparing the code changes).

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-24339/30280/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 32
• DQMHistoTests: Total histograms compared: 3143975
• DQMHistoTests: Total failures: 2
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3143776
• DQMHistoTests: Total skipped: 197
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 31 files compared)
• Checked 133 log files, 14 edm output root files, 32 DQM output files

[pieterdavid]

Addressing a point raised by @slava77 in an email thread about the HI relvals: with the latest version of the code the HLT and zerosuppression+repacking can be run in the same step, but with some workarounds for the fact that the 2015 HI files have already been repacked in the HLT before.
This seems to work (but step2 is probably still way too naive an attempt, I suppose RAW2DIGI is not there, so we would also need to insert the full SiStrip unpacker in the repacking sequence):

cmsDriver.py step1 --eventcontent RAW --datatier RAW --conditions 
auto:run2_data --step RAW2DIGI,REPACK:DigiToHybridRawRepack --scenario 
HeavyIons --data --era Run2_HI --customise RecoLocalTracker/
SiStripZeroSuppression/customiseHybrid.addHybridEmulationBeforeRepack --
processName=EMULATEHYBRID --filein /store/hidata/HIRun2015/HITrackerVirginRaw/
RAW/v1/000/263/400/00000/40322926-4AA3-E511-95F7-02163E0146A8.root --fileout 
hybridemulated.root --customise_commands 
'process.RAWoutput.outputCommands.append("drop *_*_*_HLT*")'

cmsDriver.py step2 --eventcontent RAW --datatier RAW --conditions 
auto:run2_data --step RAW2DIGI,L1,HLT,REPACK:DigiToRawRepack --scenario 
HeavyIons --data --era Run2_HI --customise RecoLocalTracker/
SiStripZeroSuppression/customiseHybrid.runOnHybridZS --customise_commands 
'from Configuration.Applications.ConfigBuilder import MassReplaceInputTag; 
MassReplaceInputTag(process, new="hybridRawDataRepacker")' --customise 
RecoLocalTracker/SiStripZeroSuppression/customiseHybrid.repackZSHybrid --
processName=REHLT --filein file:hybridemulated.root --fileout 
hltlikerepacked.root -n -1

cmsDriver.py step3 --eventcontent RECO --datatier RECO --conditions 
auto:run2_data --step RAW2DIGI,L1Reco,RECO --scenario HeavyIons --data --era 
Run2_HI --runUnscheduled --repacked --customise Validation/Performance/
TimeMemoryInfo.py --filein file:hltlikerepacked.root --fileout step3.root -n 
-1

(where dropping the previous HLT products in step1 and --customise_commands 'from Configuration.Applications.ConfigBuilder import MassReplaceInputTag; MassReplaceInputTag(process, new="hybridRawDataRepacker")' and -- processName=REHLT in step2 are only needed because this is emulated hybrid data)

@Martin-Grunewald what would be the proper way to have the zs+repacking done in the HLT? What would be needed is essentially --customise RecoLocalTracker/ SiStripZeroSuppression/customiseHybrid.runOnHybridZS --customise RecoLocalTracker/SiStripZeroSuppression/customiseHybrid.repackZSHybrid (with possibly also the unpacker added, see above).
Should this all go into the HLT menu for 2018 HI data taking, or do we need to introduce a new era (or something else) at this point?

[slava77]

+1

for #24339 bcf84c8

• code changes are in line with the PR description and the follow up review.
  • the default behavior is unchanged for pp/cosmics or even HI scenario (running on already repacked RAW data, or even with the old/classic VR->ZS repacking)
  • the new hybrid ZS can be tested with the cmsDriver as described in e.g. Add Hybrid Format SiStrip Zero Suppression algorithm for 2018 PbPb data-taking #24339 (comment)
• jenkins tests pass and comparisons with the baseline show no differences
• local tests confirm the expected behavior (notably, the last commit fixes the regression observed in the old/classic VR->ZS repacking)

[fabiocos]

+operations

changes to the StandardSequences look coherent with the purpose of the PR

[fabiocos]

@Martin-Grunewald do you still have pending questions/comments that are not addressed? Or can we move forward with this PR? I see a couple of questions of 18 days ago that were not explicitly answered by @CesarBernardes if I am not mistaken

[Martin-Grunewald]

He confirmed that the HLT settings reproduce the old behaviour.

[Martin-Grunewald]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

[fabiocos]

+1