EmDQM HI workflow fix #1846

thomreis · 2013-12-17T08:40:29Z

Added new mode without automatic configuration to EmDQM module to fix broken HI workflow.

…paths but from a python config file

…standalone

cmsbuild · 2013-12-17T09:06:38Z

A new Pull Request was created by @thomreis (Thomas Reis) for CMSSW_7_0_X.

EmDQM HI workflow fix

It involves the following packages:

HLTriggerOffline/Egamma

@nclopezo, @danduggan, @rovere, @cmsbuild, @deguio, @eliasron can you please review it and eventually sign? Thanks.
You can sign-off by replying to this message having '+1' in the first line of your reply.
You can reject by replying to this message having '-1' in the first line of your reply.
@ktf you are the release manager for this.

cmsbuild · 2013-12-17T16:15:24Z

-1
When I ran the RelVals I found an error in the following worklfows:
4.53 step2

runTheMatrix-results/4.53_RunPhoton2012B+RunPhoton2012B+HLTD+RECODreHLT+HARVESTDreHLT/step2_RunPhoton2012B+RunPhoton2012B+HLTD+RECODreHLT+HARVESTDreHLT.log
----- Begin Fatal Exception 17-Dec-2013 16:58:51 CET-----------------------
An exception of category 'FallbackFileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initFile()
   [3] Calling StorageFactory::open()
   [4] Calling XrdFile::open()
Exception Message:
Failed to open the file 'root://xrootd.ba.infn.it//store/data/Run2012B/SinglePhoton/RAW/v1/000/194/533/1084D9DA-AEA2-E111-B3AB-001D09F24DA8.root'
   Additional Info:
      [a] Input file root://eoscms//eos/cms/store/data/Run2012B/SinglePhoton/RAW/v1/000/194/533/1084D9DA-AEA2-E111-B3AB-001D09F24DA8.root?svcClass=default could not be opened.
Fallback Input file root://xrootd.ba.infn.it//store/data/Run2012B/SinglePhoton/RAW/v1/000/194/533/1084D9DA-AEA2-E111-B3AB-001D09F24DA8.root also could not be opened.
      [b] XrdClient::Open(name='root://xrootd.ba.infn.it//store/data/Run2012B/SinglePhoton/RAW/v1/000/194/533/1084D9DA-AEA2-E111-B3AB-001D09F24DA8.root', flags=0x10, permissions=0666) => error 'cannot obtain credentials for protocol: Secgsi: ErrParseBuffer: unknown CA: cannot verify server certificate: kXGS_init: unable to get protocol object.' (errno=3010)
      [c] Current server connection: root://xrootd.ba.infn.it:1094//store/data/Run2012B/SinglePhoton/RAW/v1/000/194/533/1084D9DA-AEA2-E111-B3AB-001D09F24DA8.root
----- End Fatal Exception -------------------------------------------------

4.22 step2

runTheMatrix-results/4.22_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC/step2_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC.log
----- Begin Fatal Exception 17-Dec-2013 16:58:59 CET-----------------------
An exception of category 'FallbackFileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initFile()
   [3] Calling StorageFactory::open()
   [4] Calling XrdFile::open()
Exception Message:
Failed to open the file 'root://xrootd.ba.infn.it//store/data/Run2011A/Cosmics/RAW/v1/000/160/960/049F6443-8E53-E011-A943-003048F117EA.root'
   Additional Info:
      [a] Input file root://eoscms//eos/cms/store/data/Run2011A/Cosmics/RAW/v1/000/160/960/049F6443-8E53-E011-A943-003048F117EA.root?svcClass=default could not be opened.
Fallback Input file root://xrootd.ba.infn.it//store/data/Run2011A/Cosmics/RAW/v1/000/160/960/049F6443-8E53-E011-A943-003048F117EA.root also could not be opened.
      [b] XrdClient::Open(name='root://xrootd.ba.infn.it//store/data/Run2011A/Cosmics/RAW/v1/000/160/960/049F6443-8E53-E011-A943-003048F117EA.root', flags=0x10, permissions=0666) => error 'cannot obtain credentials for protocol: Secgsi: ErrParseBuffer: unknown CA: cannot verify server certificate: kXGS_init: unable to get protocol object.' (errno=3010)
      [c] Current server connection: root://xrootd.ba.infn.it:1094//store/data/Run2011A/Cosmics/RAW/v1/000/160/960/049F6443-8E53-E011-A943-003048F117EA.root
----- End Fatal Exception -------------------------------------------------

401.0 step1

runTheMatrix-results/401.0_TTbarNewMix+TTbarFSPU2+HARVESTFS/step1_TTbarNewMix+TTbarFSPU2+HARVESTFS.log
----- Begin Fatal Exception 17-Dec-2013 16:59:16 CET-----------------------
An exception of category 'FallbackFileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=MixingModule label='mixGenPU'
   [2] Calling RootInputFileSequence::initFile()
   [3] Calling StorageFactory::open()
   [4] Calling XrdFile::open()
Exception Message:
Failed to open the file 'root://xrootd.ba.infn.it//store/relval/CMSSW_5_3_6-START53_V14/RelValProdMinBias/GEN-SIM-RAW/v2/00000/4677049F-042A-E211-8525-0026189438E8.root'
   Additional Info:
      [a] Input file root://eoscms//eos/cms/store/relval/CMSSW_5_3_6-START53_V14/RelValProdMinBias/GEN-SIM-RAW/v2/00000/4677049F-042A-E211-8525-0026189438E8.root?svcClass=default could not be opened.
Fallback Input file root://xrootd.ba.infn.it//store/relval/CMSSW_5_3_6-START53_V14/RelValProdMinBias/GEN-SIM-RAW/v2/00000/4677049F-042A-E211-8525-0026189438E8.root also could not be opened.
      [b] XrdClient::Open(name='root://xrootd.ba.infn.it//store/relval/CMSSW_5_3_6-START53_V14/RelValProdMinBias/GEN-SIM-RAW/v2/00000/4677049F-042A-E211-8525-0026189438E8.root', flags=0x10, permissions=0666) => error 'cannot obtain credentials for protocol: Secgsi: ErrParseBuffer: unknown CA: cannot verify server certificate: kXGS_init: unable to get protocol object.' (errno=3010)
      [c] Current server connection: root://xrootd.ba.infn.it:1094//store/relval/CMSSW_5_3_6-START53_V14/RelValProdMinBias/GEN-SIM-RAW/v2/00000/4677049F-042A-E211-8525-0026189438E8.root
----- End Fatal Exception -------------------------------------------------

1000.0 step2

runTheMatrix-results/1000.0_RunMinBias2011A+RunMinBias2011A+TIER0+SKIMD+HARVESTDfst2+ALCASPLIT/step2_RunMinBias2011A+RunMinBias2011A+TIER0+SKIMD+HARVESTDfst2+ALCASPLIT.log
----- Begin Fatal Exception 17-Dec-2013 16:59:21 CET-----------------------
An exception of category 'FallbackFileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initFile()
   [3] Calling StorageFactory::open()
   [4] Calling XrdFile::open()
Exception Message:
Failed to open the file 'root://xrootd.ba.infn.it//store/data/Run2011A/MinimumBias/RAW/v1/000/165/121/0699429A-B37F-E011-A57A-0019B9F72D71.root'
   Additional Info:
      [a] Input file root://eoscms//eos/cms/store/data/Run2011A/MinimumBias/RAW/v1/000/165/121/0699429A-B37F-E011-A57A-0019B9F72D71.root?svcClass=default could not be opened.
Fallback Input file root://xrootd.ba.infn.it//store/data/Run2011A/MinimumBias/RAW/v1/000/165/121/0699429A-B37F-E011-A57A-0019B9F72D71.root also could not be opened.
      [b] XrdClient::Open(name='root://xrootd.ba.infn.it//store/data/Run2011A/MinimumBias/RAW/v1/000/165/121/0699429A-B37F-E011-A57A-0019B9F72D71.root', flags=0x10, permissions=0666) => error 'cannot obtain credentials for protocol: Secgsi: ErrParseBuffer: unknown CA: cannot verify server certificate: kXGS_init: unable to get protocol object.' (errno=3010)
      [c] Current server connection: root://xrootd.ba.infn.it:1094//store/data/Run2011A/MinimumBias/RAW/v1/000/165/121/0699429A-B37F-E011-A57A-0019B9F72D71.root
----- End Fatal Exception -------------------------------------------------

1001.0 step2

runTheMatrix-results/1001.0_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVD/step2_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVD.log
----- Begin Fatal Exception 17-Dec-2013 16:59:22 CET-----------------------
An exception of category 'FallbackFileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initFile()
   [3] Calling StorageFactory::open()
   [4] Calling XrdFile::open()
Exception Message:
Failed to open the file 'root://xrootd.ba.infn.it//store/data/Run2011A/MinimumBias/RAW/v1/000/165/121/0699429A-B37F-E011-A57A-0019B9F72D71.root'
   Additional Info:
      [a] Input file root://eoscms//eos/cms/store/data/Run2011A/MinimumBias/RAW/v1/000/165/121/0699429A-B37F-E011-A57A-0019B9F72D71.root?svcClass=default could not be opened.
Fallback Input file root://xrootd.ba.infn.it//store/data/Run2011A/MinimumBias/RAW/v1/000/165/121/0699429A-B37F-E011-A57A-0019B9F72D71.root also could not be opened.
      [b] XrdClient::Open(name='root://xrootd.ba.infn.it//store/data/Run2011A/MinimumBias/RAW/v1/000/165/121/0699429A-B37F-E011-A57A-0019B9F72D71.root', flags=0x10, permissions=0666) => error 'cannot obtain credentials for protocol: Secgsi: ErrParseBuffer: unknown CA: cannot verify server certificate: kXGS_init: unable to get protocol object.' (errno=3010)
      [c] Current server connection: root://xrootd.ba.infn.it:1094//store/data/Run2011A/MinimumBias/RAW/v1/000/165/121/0699429A-B37F-E011-A57A-0019B9F72D71.root
----- End Fatal Exception -------------------------------------------------

1003.0 step2

runTheMatrix-results/1003.0_RunMinBias2012A+RunMinBias2012A+RECODDQM+HARVESTDDQM/step2_RunMinBias2012A+RunMinBias2012A+RECODDQM+HARVESTDDQM.log
----- Begin Fatal Exception 17-Dec-2013 16:59:25 CET-----------------------
An exception of category 'FallbackFileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initFile()
   [3] Calling StorageFactory::open()
   [4] Calling XrdFile::open()
Exception Message:
Failed to open the file 'root://xrootd.ba.infn.it//store/data/Run2012A/MinimumBias/RAW/v1/000/191/226/02F7EB61-0986-E111-A66F-5404A63886B4.root'
   Additional Info:
      [a] Input file root://eoscms//eos/cms/store/data/Run2012A/MinimumBias/RAW/v1/000/191/226/02F7EB61-0986-E111-A66F-5404A63886B4.root?svcClass=default could not be opened.
Fallback Input file root://xrootd.ba.infn.it//store/data/Run2012A/MinimumBias/RAW/v1/000/191/226/02F7EB61-0986-E111-A66F-5404A63886B4.root also could not be opened.
      [b] XrdClient::Open(name='root://xrootd.ba.infn.it//store/data/Run2012A/MinimumBias/RAW/v1/000/191/226/02F7EB61-0986-E111-A66F-5404A63886B4.root', flags=0x10, permissions=0666) => error 'cannot obtain credentials for protocol: Secgsi: ErrParseBuffer: unknown CA: cannot verify server certificate: kXGS_init: unable to get protocol object.' (errno=3010)
      [c] Current server connection: root://xrootd.ba.infn.it:1094//store/data/Run2012A/MinimumBias/RAW/v1/000/191/226/02F7EB61-0986-E111-A66F-5404A63886B4.root
----- End Fatal Exception -------------------------------------------------

you can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1846/1819/summary.html

deguio · 2013-12-17T16:15:36Z

ciao @thomreis
could you please give me more details?
which is the broken sequence you are talking about?
has the non-automatic mode to be run by hand?

thanks

thomreis · 2013-12-17T16:45:20Z

Hi @deguio ,
see this thread here for more details on the problem: https://hypernews.cern.ch/HyperNews/CMS/get/swDevelopment/2910/2/1/2/1/1.html
Essentially, Martin noticed that a HI validation workflow was not working anymore due to the changes introduced previously to make the EmDQM threadsave. An additional parameter 'isData' was added that did not exist in the HI config file and an exception was thrown.
Due to the fact that the internals of the EmDQM were changed a lot and would not have supported the HI workflow anymore since the the EmDQM now searched automatically for Egamma paths to validate, I introduced the non-automatic mode which bypasses the automatic configuration generation and takes the configuration from a python config file as the HI workflow uses.
If necessary the EmDQM can now also be run "by hand" with a python config file such as the one in python/HLT_Ele8_CaloIdT_TrkIdVL_DQM_cfi.py which I created for testing the new mode.

Cheers,
Thomas

thomreis · 2013-12-17T16:50:02Z

I do not think a FallbackFileOpenError is caused by the changed module. Can someone check this and run the tests again?

deguio · 2013-12-17T17:10:06Z

+1
understood and tested. :)
thanks @thomreis

cmsbuild · 2013-12-17T17:13:02Z

This pull request is fully signed and it will be integrated in one of the next IBs unless changes (but tests are reportedly failing). @ktf can you please take care of it?

ktf · 2013-12-17T18:06:54Z

Trusting @deguio here. Approving.

DQM fixes -- EmDQM HI workflow fix

thomreis added 2 commits December 16, 2013 18:37

introduce a mode to run EmDQM without automatic configuration of the …

d30e58e

…paths but from a python config file

Merge remote-tracking branch 'official-cmssw/CMSSW_7_0_X' into emdqm_…

c71938c

…standalone

ktf added a commit that referenced this pull request Dec 17, 2013

Merge pull request #1846 from thomreis/emdqm_standalone

1fca221

DQM fixes -- EmDQM HI workflow fix

ktf merged commit 1fca221 into cms-sw:CMSSW_7_0_X Dec 17, 2013

thomreis deleted the emdqm_standalone branch December 17, 2013 22:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EmDQM HI workflow fix #1846

EmDQM HI workflow fix #1846

thomreis commented Dec 17, 2013

cmsbuild commented Dec 17, 2013

cmsbuild commented Dec 17, 2013

deguio commented Dec 17, 2013

thomreis commented Dec 17, 2013

thomreis commented Dec 17, 2013

deguio commented Dec 17, 2013

cmsbuild commented Dec 17, 2013

ktf commented Dec 17, 2013

EmDQM HI workflow fix #1846

EmDQM HI workflow fix #1846

Conversation

thomreis commented Dec 17, 2013

cmsbuild commented Dec 17, 2013

cmsbuild commented Dec 17, 2013

deguio commented Dec 17, 2013

thomreis commented Dec 17, 2013

thomreis commented Dec 17, 2013

deguio commented Dec 17, 2013

cmsbuild commented Dec 17, 2013

ktf commented Dec 17, 2013