Addition of IOExerciser, a simple module for emulating various IO patterns #320

bbockelm · 2013-08-14T14:43:21Z

This is a simple OutputModule (must be an output module in order to access the product information) which reads event data in a configurable pattern. The data read is determined by desired IO pattern, not file contents; the actual object contents are ignored.

This is meant to test out new IO code and to watch for regressions between releases.

patterns.

ktf · 2013-08-14T14:46:35Z

Utilities/IOExerciser/python/__init__.py

@@ -0,0 +1,3 @@
+#Automatically created by SCRAM


Please do not include file which are created by scram.

This file should be removed

This file must be removed. Please do not sign or merge until it is removed.

ktf · 2013-08-14T14:47:30Z

Do you really want to have a separate package? Why not just an additional test?

@Dr15Jones

Dr15Jones · 2013-08-14T14:48:00Z

Utilities/IOExerciser/src/IOExerciser.cc

+//
+// Original Author:  Brian Bockelman
+//         Created:  Mon Jun  4 17:35:30 CDT 2012
+// $Id$


Please remove this line

bbockelm · 2013-08-14T14:54:42Z

Additionally - Chris reminded me, the reason I needed this to be an OutputModule is because I need access to the TTree itself. I use this to determine the size of branches in the input (as this module wants to read N% of the input data by volume, not N% of the branches).

- Move the plugin to IOPool/Input test directory. - Remove CVS-specific $Id$.

ktf · 2013-08-16T08:44:43Z

@nclopezo can you test?

ktf · 2013-08-16T08:46:13Z

@nclopezo for the record, this needs to be tested standalone. have a look at the readme. @bbockelm any chance we can get it to run inside scram b runtests?

nclopezo · 2013-08-16T12:14:28Z

Hi,

I created a test file ( test.py) with the example configuration shown in the readme, like this.

import FWCore.ParameterSet.Config as cms
process = cms.Process("Demo")
process.load("FWCore.MessageService.MessageLogger_cfi")
process.maxEvents = cms.untracked.PSet( input = cms.untracked.int32(-1) )
process.source = cms.Source("PoolSource",
    # replace 'myfile.root' with the source file you want to use
    fileNames = cms.untracked.vstring(
        '/store/mc/HC/GenericTTbar/GEN-SIM-RECO/CMSSW_5_3_1_START53_V5-v1/0010/A2B8D47D-BCAD-E111-B16A-BCAEC53296FB.root'
    )
)
process.demo = cms.EDAnalyzer('IOExerciser',
    percentBranches = cms.untracked.uint32(30),
    triggerFactor = cms.untracked.uint32(20),
)
process.p = cms.Path(process.demo)

I did cmsRun test.py,

But I got the following output:

16-Aug-2013 14:11:23 CEST  Initiating request to open file root://eoscms//eos/cms/store/mc/HC/GenericTTbar/GEN-SIM-RECO/CMSSW_5_3_1_START53_V5-v1/0010/A2B8D47D-BCAD-E111-B16A-BCAEC53296FB.root?svcClass=default
16-Aug-2013 14:11:27 CEST  Successfully opened file root://eoscms//eos/cms/store/mc/HC/GenericTTbar/GEN-SIM-RECO/CMSSW_5_3_1_START53_V5-v1/0010/A2B8D47D-BCAD-E111-B16A-BCAEC53296FB.root?svcClass=default
16-Aug-2013 14:11:29 CEST  Closed file root://eoscms//eos/cms/store/mc/HC/GenericTTbar/GEN-SIM-RECO/CMSSW_5_3_1_START53_V5-v1/0010/A2B8D47D-BCAD-E111-B16A-BCAEC53296FB.root?svcClass=default
----- Begin Fatal Exception 16-Aug-2013 14:11:29 CEST-----------------------
An exception of category 'PluginNotFound' occurred while
   [0] Constructing the EventProcessor
Exception Message:
Unable to find plugin 'IOExerciser' in category 'CMS EDM Framework Module'. Please check spelling of name.
----- End Fatal Exception -------------------------------------------------

Am I doing something wrong?

nclopezo · 2013-08-16T12:17:53Z

Sorry, the actual output that I get is the following, I copy pasted wrong in the previous message.

16-Aug-2013 14:16:31 CEST  Initiating request to open file root://eoscms//eos/cms/store/mc/HC/GenericTTbar/GEN-SIM-RECO/CMSSW_5_3_1_START53_V5-v1/0010/A2B8D47D-BCAD-E111-B16A-BCAEC53296FB.root?svcClass=default
16-Aug-2013 14:16:35 CEST  Successfully opened file root://eoscms//eos/cms/store/mc/HC/GenericTTbar/GEN-SIM-RECO/CMSSW_5_3_1_START53_V5-v1/0010/A2B8D47D-BCAD-E111-B16A-BCAEC53296FB.root?svcClass=default
16-Aug-2013 14:16:37 CEST  Closed file root://eoscms//eos/cms/store/mc/HC/GenericTTbar/GEN-SIM-RECO/CMSSW_5_3_1_START53_V5-v1/0010/A2B8D47D-BCAD-E111-B16A-BCAEC53296FB.root?svcClass=default
----- Begin Fatal Exception 16-Aug-2013 14:16:37 CEST-----------------------
An exception of category 'Configuration' occurred while
   [0] Constructing the EventProcessor
   [1] Validating configuration of module: class=IOExerciser label='demo'
Exception Message:
The base type in the python configuration is EDAnalyzer, but the base type
for the module's C++ class is OutputModule. Please fix the configuration.
It must use the same base type as the C++ class.
----- End Fatal Exception -------------------------------------------------

ktf · 2013-08-16T12:20:16Z

You did run scram b, right?

nclopezo · 2013-08-16T12:21:59Z

Yes, I did.

Dr15Jones · 2013-08-16T13:02:27Z

The error message has the answer, you declared IOExerciser a 'EDAnalyzer' in python but in C++ it is an 'OutputModule'. The python and C++ types must match. The fix is

process.demo = cms.OutputModule('IOExerciser',
    percentBranches = cms.untracked.uint32(30),
    triggerFactor = cms.untracked.uint32(20),
)

process.p = cms.EndPath(process.demo)

This is an OutputModule, not an EDAnalyzer.

bbockelm · 2013-08-16T13:40:07Z

Example in documentation is now fixed with the correct C++ type name.

bbockelm · 2013-08-16T13:42:19Z

@ktf - I'm not sure why one would want to run this inside runtests. It's meant to do IO benchmarks for large files.

ktf · 2013-08-16T15:23:41Z

Ok.

bbockelm · 2013-08-16T17:34:32Z

@nclopezo - were you able to successfully test with the fix to the .py file?

ktf · 2013-08-19T07:49:37Z

@nclopezo ping

nclopezo · 2013-08-19T08:23:24Z

Hi,

I ran it again, it seems to work. The following is the output that I got:

19-Aug-2013 10:20:07 CEST  Initiating request to open file root://eoscms//eos/cms/store/mc/HC/GenericTTbar/GEN-SIM-RECO/CMSSW_5_3_1_START53_V5-v1/0010/A2B8D47D-BCAD-E111-B16A-BCAEC53296FB.root?svcClass=default
19-Aug-2013 10:20:13 CEST  Successfully opened file root://eoscms//eos/cms/store/mc/HC/GenericTTbar/GEN-SIM-RECO/CMSSW_5_3_1_START53_V5-v1/0010/A2B8D47D-BCAD-E111-B16A-BCAEC53296FB.root?svcClass=default
Begin processing the 1st record. Run 1, Event 13201, LumiSection 666930 at 19-Aug-2013 10:20:15.663 CEST
Begin processing the 101st record. Run 1, Event 14101, LumiSection 666948 at 19-Aug-2013 10:20:20.923 CEST
Begin processing the 201st record. Run 1, Event 14451, LumiSection 666955 at 19-Aug-2013 10:20:24.707 CEST
Begin processing the 301st record. Run 1, Event 15951, LumiSection 666985 at 19-Aug-2013 10:20:28.477 CEST
Begin processing the 401st record. Run 1, Event 16051, LumiSection 666987 at 19-Aug-2013 10:20:32.342 CEST
Begin processing the 501st record. Run 1, Event 16551, LumiSection 666997 at 19-Aug-2013 10:20:36.597 CEST
Begin processing the 601st record. Run 1, Event 17001, LumiSection 667006 at 19-Aug-2013 10:20:40.371 CEST
Begin processing the 701st record. Run 1, Event 18301, LumiSection 667032 at 19-Aug-2013 10:20:44.313 CEST
Begin processing the 801st record. Run 1, Event 18851, LumiSection 667043 at 19-Aug-2013 10:20:48.211 CEST
Begin processing the 901st record. Run 1, Event 19001, LumiSection 667046 at 19-Aug-2013 10:20:52.160 CEST
Begin processing the 1001st record. Run 1, Event 19501, LumiSection 667056 at 19-Aug-2013 10:20:55.938 CEST
Begin processing the 1101st record. Run 1, Event 20401, LumiSection 667074 at 19-Aug-2013 10:20:59.729 CEST
Begin processing the 1201st record. Run 1, Event 21501, LumiSection 667096 at 19-Aug-2013 10:21:03.541 CEST
Begin processing the 1301st record. Run 1, Event 21951, LumiSection 667105 at 19-Aug-2013 10:21:07.343 CEST
Begin processing the 1401st record. Run 1, Event 23201, LumiSection 667130 at 19-Aug-2013 10:21:11.287 CEST
Begin processing the 1501st record. Run 1, Event 24101, LumiSection 667148 at 19-Aug-2013 10:21:15.060 CEST
Begin processing the 1601st record. Run 1, Event 24451, LumiSection 667155 at 19-Aug-2013 10:21:18.855 CEST
19-Aug-2013 10:21:20 CEST  Closed file root://eoscms//eos/cms/store/mc/HC/GenericTTbar/GEN-SIM-RECO/CMSSW_5_3_1_START53_V5-v1/0010/A2B8D47D-BCAD-E111-B16A-BCAEC53296FB.root?svcClass=default
=============================================
MessageLogger Summary
 type     category        sev    module        subroutine        count    total
 ---- -------------------- -- ---------------- ----------------  -----    -----
    1 fileAction           -s file_close                             1        1
    2 fileAction           -s file_open                              2        2
 type    category    Examples: run/evt        run/evt          run/evt
 ---- -------------------- ---------------- ---------------- ----------------
    1 fileAction           PostEndRun                        
    2 fileAction           pre-events       pre-events       
Severity    # Occurrences   Total Occurrences
--------    -------------   -----------------
System                  3                   3

ktf · 2013-08-19T10:05:52Z

Looks reasonable. Merging this.

Addition of IOExerciser, a simple module for emulating various IO patterns

re-enable mva

…cms-sw#320)

* add HiGenAnalyzer This analyzer is copied from its AOD counterpart https://github.com/CmsHI/cmssw/blob/2c806f88506f7ef732b725142ae85750a31dc646/HeavyIonsAnalysis/EventAnalysis/src/HiEvtAnalyzer.cc and adapted for gen info in miniAOD Notes : GenHIEvent is currently missing from miniAOD * fix sube collisionId_ is not kept in pat::PackedGenParticle, use "packedGenParticlesSignal" (added by cms-sw#32668) to tag particles from signal process

pixelPairStep addition

Addition of IOExerciser, a simple module for emulating various IO

a92dd88

patterns.

ktf reviewed Aug 14, 2013
View reviewed changes

Dr15Jones reviewed Aug 14, 2013
View reviewed changes

ghost assigned bbockelm Aug 14, 2013

Fixes based on github review from Giulio and Chris.

ff34090

- Move the plugin to IOPool/Input test directory. - Remove CVS-specific $Id$.

ghost assigned nclopezo Aug 16, 2013

ghost assigned bbockelm Aug 16, 2013

Correct documentation example.

c008e0b

This is an OutputModule, not an EDAnalyzer.

ghost assigned nclopezo Aug 19, 2013

ktf added a commit that referenced this pull request Aug 19, 2013

Merge pull request #320 from bbockelm/ioexerciser

4479f9d

Addition of IOExerciser, a simple module for emulating various IO patterns

ktf merged commit 4479f9d into cms-sw:CMSSW_7_0_X Aug 19, 2013

silviodonato pushed a commit to silviodonato/cmssw that referenced this pull request Feb 1, 2016

Merge pull request cms-sw#320 from arizzi/minorfix3

8080070

re-enable mva

fwyzard pushed a commit to fwyzard/cmssw that referenced this pull request Apr 16, 2019

Fix the CUDA product availability logic in SiPixelRecHitHeterogeneous (…

505dee5

…cms-sw#320)

slava77 pushed a commit to slava77/cmssw that referenced this pull request Oct 9, 2021

Merge pull request cms-sw#320 from trackreco/pixelPairStepAddition

22f28c5

pixelPairStep addition

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Addition of IOExerciser, a simple module for emulating various IO patterns #320

Addition of IOExerciser, a simple module for emulating various IO patterns #320

bbockelm commented Aug 14, 2013

ktf Aug 14, 2013

Dr15Jones Aug 14, 2013

wmtan Aug 14, 2013

ktf commented Aug 14, 2013

Dr15Jones Aug 14, 2013

bbockelm commented Aug 14, 2013

ktf commented Aug 16, 2013

ktf commented Aug 16, 2013

nclopezo commented Aug 16, 2013

nclopezo commented Aug 16, 2013

ktf commented Aug 16, 2013

nclopezo commented Aug 16, 2013

Dr15Jones commented Aug 16, 2013

bbockelm commented Aug 16, 2013

bbockelm commented Aug 16, 2013

ktf commented Aug 16, 2013

bbockelm commented Aug 16, 2013

ktf commented Aug 19, 2013

nclopezo commented Aug 19, 2013

ktf commented Aug 19, 2013

Addition of IOExerciser, a simple module for emulating various IO patterns #320

Addition of IOExerciser, a simple module for emulating various IO patterns #320

Conversation

bbockelm commented Aug 14, 2013

ktf Aug 14, 2013

Choose a reason for hiding this comment

Dr15Jones Aug 14, 2013

Choose a reason for hiding this comment

wmtan Aug 14, 2013

Choose a reason for hiding this comment

ktf commented Aug 14, 2013

Dr15Jones Aug 14, 2013

Choose a reason for hiding this comment

bbockelm commented Aug 14, 2013

ktf commented Aug 16, 2013

ktf commented Aug 16, 2013

nclopezo commented Aug 16, 2013

nclopezo commented Aug 16, 2013

ktf commented Aug 16, 2013

nclopezo commented Aug 16, 2013

Dr15Jones commented Aug 16, 2013

bbockelm commented Aug 16, 2013

bbockelm commented Aug 16, 2013

ktf commented Aug 16, 2013

bbockelm commented Aug 16, 2013

ktf commented Aug 19, 2013

nclopezo commented Aug 19, 2013

ktf commented Aug 19, 2013