warn when no data is available for CalibTracker/SiStripChannelGain unit test #33065

mmusich · 2021-03-04T10:04:31Z

PR description:

As pointed out in #33055 (comment) when the first cmsRun instance of testSSTGainPCL_fromRECO

cmssw/CalibTracker/SiStripChannelGain/test/testSSTGain_PCL_FromRECO.sh

    
           (cmsRun ${LOCAL_TEST_DIR}/testSSTGain_PCL_FromRECO_cfg.py era=A) || die 'Failure running cmsRun testSSTGain_PCL_FromRECO_cfg.py era=A' $?

(silently) failed because of lack of suitable input data coming from the dedicate DAS query, the subsequent harvesting step cmsRun

cmssw/CalibTracker/SiStripChannelGain/test/testSSTGain_PCL_FromRECO.sh

Line 6 in 837e8e8

    
           (cmsRun ${LOCAL_TEST_DIR}/testSSTGain_HARVEST_FromRECO.py 0) || die 'Failure running cmsRun testSSTGain_HARVEST_FromRECO.py 0' $?

might fail with somewhat confusing errors about a missing (local) file, see #33055 (comment).
By using a python warning it should be now clear from the logs that the initial DAS query failed.

PR validation:

Run the unit tests in a local check-out area, by using scramv1 b runtests - without having set valid GRID credentials (to force the test to fail) and got back:

cmsRun:26: UserWarning: DAS query status: error
cmsRun:93: UserWarning: ** WARNING: ** According to a DAS query no suitable data for test is available. Skipping test
Failure running cmsRun testSSTGain_PCL_FromRECO_cfg.py era=A: status 1
status = 256

---> test testSSTGainPCL_fromRECO had ERRORS

if this PR is a backport please specify the original PR and why you need to backport that PR:

Not a backport, no backport needed.

…it test

cmsbuild · 2021-03-04T10:10:01Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-33065/21375

This PR adds an extra 12KB to repository

cmsbuild · 2021-03-04T10:10:25Z

A new Pull Request was created by @mmusich (Marco Musich) for master.

It involves the following packages:

CalibTracker/SiStripChannelGain

@malbouis, @yuanchao, @christopheralanwest, @cmsbuild, @tlampen, @pohsun, @francescobrivio can you please review it and eventually sign? Thanks.
@echabert, @pieterdavid, @robervalwalsh, @gbenelli, @tocheng, @alesaggio, @mmusich this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

mmusich · 2021-03-04T10:14:13Z

@cmsbuild please test

makortel · 2021-03-04T13:31:06Z

Thanks!

cmsbuild · 2021-03-04T13:52:33Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-71ae45/13264/summary.html
COMMIT: 0bb1f8d
CMSSW: CMSSW_11_3_X_2021-03-03-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/33065/13264/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test testSSTGainPCL_fromRECO had ERRORS
---> test checkMultiRunHarvestingOutput had ERRORS

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 2869 differences found in the comparisons
DQMHistoTests: Total files compared: 37
DQMHistoTests: Total histograms compared: 2749431
DQMHistoTests: Total failures: 64469
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2684940
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 36 files compared)
Checked 156 log files, 37 edm output root files, 37 DQM output files

mmusich · 2021-03-04T19:00:35Z

The error reported is:

Using Dataset: /ZeroBias/Run2018A-SiStripCalMinBias-PromptReco-v3/ALCARECO
04-Mar-2021 11:29:17 CET  Initiating request to open file root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/data/Run2018A/ZeroBias/ALCARECO/SiStripCalMinBias-PromptReco-v3/000/316/569/00000/102770E5-A164-E811-B4A0-FA163EF55202.root
04-Mar-2021 11:29:18 CET  Initiating request to open file root://cms-xrd-global.cern.ch//store/data/Run2018A/ZeroBias/ALCARECO/SiStripCalMinBias-PromptReco-v3/000/316/569/00000/102770E5-A164-E811-B4A0-FA163EF55202.root
----- Begin Fatal Exception 04-Mar-2021 11:29:49 CET-----------------------
An exception of category 'FileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   [3] Calling StorageFactory::open()
   [4] Calling XrdFile::open()
Exception Message:
Failed to open the file 'root://cms-xrd-global.cern.ch//store/data/Run2018A/ZeroBias/ALCARECO/SiStripCalMinBias-PromptReco-v3/000/316/569/00000/102770E5-A164-E811-B4A0-FA163EF55202.root'
   Additional Info:
      [a] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/data/Run2018A/ZeroBias/ALCARECO/SiStripCalMinBias-PromptReco-v3/000/316/569/00000/102770E5-A164-E811-B4A0-FA163EF55202.root
      [b] Input file root://cms-xrd-global.cern.ch//store/data/Run2018A/ZeroBias/ALCARECO/SiStripCalMinBias-PromptReco-v3/000/316/569/00000/102770E5-A164-E811-B4A0-FA163EF55202.root could not be opened.
      [c] XrdCl::File::Open(name='root://cms-xrd-global.cern.ch//store/data/Run2018A/ZeroBias/ALCARECO/SiStripCalMinBias-PromptReco-v3/000/316/569/00000/102770E5-A164-E811-B4A0-FA163EF55202.root', flags=0x10, permissions=0660) => error '[ERROR] Server responded with an error: [3011] No servers are available to read the file.�
' (errno=3011, code=400). No additional data servers were found.
      [d] Last URL tried: root://cms-xrd-global.cern.ch:1094//store/data/Run2018A/ZeroBias/ALCARECO/SiStripCalMinBias-PromptReco-v3/000/316/569/00000/102770E5-A164-E811-B4A0-FA163EF55202.root?tried=&xrdcl.requuid=e0ac1395-a073-4578-8d9c-c4112e4a2885
      [e] Problematic data server: cms-xrd-global.cern.ch:1094
      [f] Disabled source: cms-xrd-global.cern.ch:1094
----- End Fatal Exception -------------------------------------------------
Failure running cmsRun testSSTGain_PCL_FromRECO_cfg.py era=A: status 84

@smuzaffar I don't quite understand how this LFN ends up being used in the test.
The DAS query is here:

cmssw/CalibTracker/SiStripChannelGain/test/testSSTGain_PCL_FromRECO_cfg.py

Line 21 in a43633b

    
           query = "dataset dataset=/ZeroBias/Run2*"+era_name+"*SiStripCalMinBias-*/ALCARECO site=T2_CH_CERN"

and the only dataset fulfilling that query that I see at CERN at the moment is:

$ dasgoclient -query='dataset dataset=/ZeroBias/Run2*A*SiStripCalMinBias-*/ALCARECO site=T2_CH_CERN'
/ZeroBias/Run2018A-SiStripCalMinBias-12Nov2019_UL2018-v1/ALCARECO

while

$ dasgoclient -query='site dataset=/ZeroBias/Run2018A-SiStripCalMinBias-PromptReco-v3/ALCARECO'
T1_US_FNAL_Tape

is the DAS version used by the bot different than the one available on lxplus?

smuzaffar · 2021-03-05T09:14:06Z

please test
@mmusich, bot was using das cached results ( cms-sw/cms-sw.github.io@f6c249f#diff-1505e761d2329169b78d9f6520a65425fea469f6873e7b8df9bf45069b247c6a ) which had 12 matching datasets for /ZeroBias/Run2*A*SiStripCalMinBias-*/ALCARECO . The cache is updated now and now there is only matching results /ZeroBias/Run2018A-SiStripCalMinBias-12Nov2019_UL2018-v1/ALCARECO . I have re-started the test so it should use the correct dataset now

cmsbuild · 2021-03-05T14:24:05Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-71ae45/13297/summary.html
COMMIT: 0bb1f8d
CMSSW: CMSSW_11_3_X_2021-03-04-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/33065/13297/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-71ae45/34634.0_TTbar_14TeV+2026D76+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-71ae45/34834.999_TTbar_14TeV+2026D76PU_PMXS1S2PR+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+PREMIX_PremixHLBeamSpot14PU+DigiTriggerPU+RecoGlobalPU+HARVESTGlobalPU

Summary:

No significant changes to the logs found
Reco comparison results: 3001 differences found in the comparisons
DQMHistoTests: Total files compared: 38
DQMHistoTests: Total histograms compared: 2849195
DQMHistoTests: Total failures: 64883
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2784290
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 37 files compared)
Checked 160 log files, 37 edm output root files, 38 DQM output files

yuanchao · 2021-03-08T09:07:50Z

+1

only python script modified to have warning instead of simple print() message
python build ok.

cmsbuild · 2021-03-08T09:08:14Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

silviodonato · 2021-03-08T09:19:08Z

+1
the differences are in the DD4HEP workflows, we see something similar in many other PR tests

warn when no data is avaialble for CalibTracker/SiStripChannelGain un…

0bb1f8d

…it test

cmsbuild added this to the CMSSW_11_3_X milestone Mar 4, 2021

cmsbuild added alca-pending code-checks-pending orp-pending pending-signatures tests-pending labels Mar 4, 2021

mmusich mentioned this pull request Mar 4, 2021

Unit test failure in CalibTracker/SiStripChannelGain #33055

Closed

mmusich changed the title ~~warn when no data is avaialble for CalibTracker/SiStripChannelGain unit test~~ warn when no data is available for CalibTracker/SiStripChannelGain unit test Mar 4, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Mar 4, 2021

cmsbuild added tests-started and removed tests-pending labels Mar 4, 2021

cmsbuild added tests-rejected and removed tests-started labels Mar 4, 2021

cmsbuild added tests-started and removed tests-rejected labels Mar 5, 2021

cmsbuild added tests-approved and removed tests-started labels Mar 5, 2021

cmsbuild removed alca-pending pending-signatures labels Mar 8, 2021

cmsbuild added alca-approved fully-signed labels Mar 8, 2021

cmsbuild added orp-approved and removed orp-pending labels Mar 8, 2021

cmsbuild merged commit 7b3b44a into cms-sw:master Mar 8, 2021

mmusich deleted the warn_when_no_data_in_CalibTracker_SiStripChannelGain_test branch March 8, 2021 09:34

mmusich restored the warn_when_no_data_in_CalibTracker_SiStripChannelGain_test branch March 8, 2021 09:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

warn when no data is available for CalibTracker/SiStripChannelGain unit test #33065

warn when no data is available for CalibTracker/SiStripChannelGain unit test #33065

mmusich commented Mar 4, 2021

cmsbuild commented Mar 4, 2021

cmsbuild commented Mar 4, 2021

mmusich commented Mar 4, 2021

makortel commented Mar 4, 2021

cmsbuild commented Mar 4, 2021

mmusich commented Mar 4, 2021

smuzaffar commented Mar 5, 2021 •

edited

cmsbuild commented Mar 5, 2021

yuanchao commented Mar 8, 2021

cmsbuild commented Mar 8, 2021

silviodonato commented Mar 8, 2021

warn when no data is available for CalibTracker/SiStripChannelGain unit test #33065

warn when no data is available for CalibTracker/SiStripChannelGain unit test #33065

Conversation

mmusich commented Mar 4, 2021

PR description:

PR validation:

if this PR is a backport please specify the original PR and why you need to backport that PR:

cmsbuild commented Mar 4, 2021

cmsbuild commented Mar 4, 2021

mmusich commented Mar 4, 2021

makortel commented Mar 4, 2021

cmsbuild commented Mar 4, 2021

Unit Tests

Comparison Summary

mmusich commented Mar 4, 2021

smuzaffar commented Mar 5, 2021 • edited

cmsbuild commented Mar 5, 2021

Comparison Summary

yuanchao commented Mar 8, 2021

cmsbuild commented Mar 8, 2021

silviodonato commented Mar 8, 2021

smuzaffar commented Mar 5, 2021 •

edited