SpySoftware affected by a commit on PoolSource

[Jangbae]

Hi,

We've found that there was a commit [1] on PoolSource which changes SiStripSpyEventMatcher under DQM/SiStripMonitorHardware package to use EmbeddedRootSource. The EmbeddedRootSource seems not working so we'd like to raise an issue. These followings are what we've tested for the spy event matcher.

1. We tested CMSSW_9_2_6 and add package "DQM/SiStripMonitorHardware"

   • Compiled and cmsenv done.

2. Gave main stream data and digitized spy run data as inputs for the spyMatcher config

   • Digitization for spy data was done by running the unpacker and digi-converter before running the spy matcher config

3. Tested the event matcher config with two different SecSources, "PoolSource" and "EmbeddedRootSource"

   • With "EmbeddedRootSource" -> Problem : the digitized spy data is opened and closed infinitely.
   • With "PoolSource" -> Problem : Exception occurs with message "Unable to find plugin 'PoolSource' in category 'CMS EDM Framework VectorInputSource'."

4. Inheritance relation check for PoolSource
   The vectorinputsource class has been removed from CMSSW8XX in the inheritance relation with PoolSource. You can find this from cmsdoxygens for CMSSW_8_0_24[2] and CMSSW_7_3_2[3]

5. In CMSSW 7, the event matcher works with PoolSource.

In summary, the spy event matcher works well in CMSSW 7 but it's not anymore with CMSSW 8 and 9. This is due to the commit on the PoolSource and we'd like to ask some help to make the spy event matcher run again in CMSSW8 and 9.

Regards,
Jangbae

[1] https://github.com/cms-sw/cmssw/commit/b0371c1187c522c1367eedb6779230e74805c535
[2]http://cmsdoxygen.web.cern.ch/cmsdoxygen/CMSSW_8_0_24/doc/html/d9/d5a/classedm_1_1PoolSource.html
[3]http://cmsdoxygen.web.cern.ch/cmsdoxygen/CMSSW_7_3_2/doc/html/d9/d5a/classedm_1_1PoolSource.html

[cmsbuild]

A new Issue was created by @Jangbae .

@davidlange6, @Dr15Jones, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[Dr15Jones]

@dan131riley could you take a look at this?

[dan131riley]

@Jangbae can you give me a recipe for reproducing this?

[Jangbae]

@dan131riley
Sure, Following is the recipe.

cmsrel CMSSW_9_2_6
cmsenv in src
git addpkg-cms DQM/SiStripMonitorHardware
scram b
DQM/SiStripMonitorHardware/test/spyEventMatching_cfg.py
* Line 12 : a main stream data file should be given [1]
* Line 27 : global tag = 92X_upgrade2017_TSG_For83XSamples_V4
* Line 32 : a digitized spy data file should be given[2]
* Line 44 : PrimaryEventRawDataTag = cms.InputTag('rawDataCollector’)
* Line 48 : RawSpyDataTag = cms.InputTag('rawDataCollector')
cmsRun spyEventMatching_cfg.py in DQM/SiStripMonitorHardware/test
DQM/SiStripMonitorHardware/python/SiStripSpyEventMatcher_cfi.py
* Line 16 : Can be “PoolSource"

[1] and [2] are placed at /afs/cern.ch/work/j/jblee/public/forDan

[dan131riley]

It looks like this was broken in the 74x timeframe, before PoolSource was refactored. In 73x, RootInputFileSequence::readOneSequential() returns false when it runs off the end of the file list:

    if(!found) {
      ++fileIter_;
      if(fileIter_ == fileIterEnd_) {
        return false;
      }

In 74x, that changed to

    if(!found) {
      ++fileIter_;
      if(fileIter_ == fileIterEnd_) {
        fileIter_ = fileIterBegin_;
      }

i.e., when it reaches the end of the list it loops back to the first file. This makes sense for the mixing module, but not for the SpyEventMatcher, it does

    source_->loopOverEvents(*eventPrincipal_,fileNameHash,std::numeric_limits<size_t>::max(),boost::bind(&SpyEventMatcher::addNextEventToMap,this,_1));

which is gonna loop a lot of times to read std::numeric_limits<size_t>::max() events.

Least disruptive option is probably adding a pset option to EmbeddedRootSource to turn off looping back to the beginning of the list (for sequential access only).

[dan131riley]

PR #20403 adds an option to restore the old behavior where EmbeddedRootSource stops at the end of the last file. With this change, your test case runs to completion, producing a SpyMatchedEvents.root with 1 run, 49 lumis and 0 events. This looks to be correct, as SpyRawToDigis.root contains run 298270 lumi 1, while the raw data file only has data from run 201679, so there shouldn't be any match.

[Jangbae]

Thank you very much!