Fix in popcon workflow: handling of empty files #30090

ggovi · 2020-06-03T10:32:00Z

PR description:

This fix follows the recent changes in DbOutputService, that have removed the implicit schema creation. Since no db is created when no IOV insertion is requested, the PopCon workflow requires now an explicit call. In any case, non-emtpy files with 0 IOVs are still ignored.

PR validation:

O2O unit/integration tests

cmsbuild · 2020-06-03T10:32:29Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-06-03T10:37:38Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30090/15829

This PR adds an extra 16KB to repository

cmsbuild · 2020-06-03T10:38:04Z

A new Pull Request was created by @ggovi for master.

It involves the following packages:

CondCore/PopCon
CondCore/Utilities

@christopheralanwest, @tocheng, @cmsbuild, @tlampen, @ggovi, @pohsun can you please review it and eventually sign? Thanks.
@mmusich, @tocheng this is something you requested to watch as well.
@silviodonato, @dpiparo you are the release manager for this.

cms-bot commands are listed here

ggovi · 2020-06-03T10:48:36Z

please test

cmsbuild · 2020-06-03T10:48:59Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/6764/console Started: 2020/06/03 12:53
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/6809/console Started: 2020/06/04 15:39

cmsbuild · 2020-06-03T12:07:38Z

+1
Tested at: 98ead29
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-700a80/6764/summary.html
CMSSW: CMSSW_11_2_X_2020-06-02-2300
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-06-03T12:07:41Z

Comparison job queued.

cmsbuild · 2020-06-03T14:21:38Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-700a80/6764/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 36
DQMHistoTests: Total histograms compared: 2783666
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2783615
DQMHistoTests: Total skipped: 50
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 35 files compared)
Checked 152 log files, 16 edm output root files, 36 DQM output files

ggovi · 2020-06-03T20:10:22Z

+1

christopheralanwest · 2020-06-03T20:50:32Z

@ggovi How are you testing this? I executed the following:

cmsrel CMSSW_11_2_X_2020-06-03-1100
cd CMSSW_11_2_X_2020-06-03-1100/src
cmsenv
git cms-merge-topic -u 30090
git cms-addpkg CondTools/Ecal
scram b -j 16
scram b runtests

which fails in the following way:

Package CondTools/Ecal: Running test EcalDCS_O2O_test
 
===== Test "EcalDCS_O2O_test" ====
[2020-06-03 22:40:35,147] INFO: Connecting to pro [frontier://PromptProd/CMS_CONDITIONS]
[2020-06-03 22:40:36,598] INFO: Connecting to EcalDCSTowerStatus_online_O2OTEST.db [sqlite:///EcalDCSTowerStatus_online_O2OTEST.db]
[2020-06-03 22:40:36,635] INFO: Initializing database...
[2020-06-03 22:40:37,132] INFO: Copying tag pro::EcalDCSTowerStatus_online to EcalDCSTowerStatus_online_O2OTEST.db::EcalDCSTowerStatus_online from since=1
[2020-06-03 22:40:37,201] INFO: Found 1 iovs and 1 referenced payloads to copy.
[2020-06-03 22:40:37,207] INFO: Copying payload pro::06e5459f27375dd0ac9880be508b87f671fe0a0c to EcalDCSTowerStatus_online_O2OTEST.db::06e5459f27375dd0ac9880be508b87f671fe0a0c ...
[2020-06-03 22:40:37,218] INFO: 1 payload(s) copied.
[2020-06-03 22:40:37,220] INFO: 1 iov(s) copied.
[2020-06-03 22:40:37,231] INFO: Changes committed.
[2020-06-03 22:40:37,433] INFO: Executing command: cmsRun ./src/CondTools/Ecal/python/EcalDCS_popcon.py  targetFile=2020-06-03_20-40-37-433697.db destinationDatabase=sqlite_file:EcalDCSTowerStatus_online_O2OTEST.db destinationTag=EcalDCSTowerStatus_online 2>&1
----- Begin Fatal Exception 03-Jun-2020 22:40:37 CEST-----------------------
An exception of category 'ConfigFileReadError' occurred while
   [0] Processing the python configuration file named ./src/CondTools/Ecal/python/EcalDCS_popcon.py
Exception Message:
 unknown python problem occurred.
Exception: Entry for service cms_omds_adg/CMS_ECAL_R not found in .netrc

At:
  /tmp/cawest/CMSSW_11_2_X_2020-06-03-1100/python/CondTools/Ecal/db_credentials.py(16): get_db_credentials
  ./src/CondTools/Ecal/python/EcalDCS_popcon.py(24): <module>

----- End Fatal Exception -------------------------------------------------

[2020-06-03 22:40:37,894] INFO: PopCon Analyzer return code is: 90
[2020-06-03 22:40:37,894] ERROR: O2O job failed. Skipping upload.
[2020-06-03 22:40:38,311] INFO: Connecting to EcalDCSTowerStatus_online_O2OTEST.db [sqlite:///EcalDCSTowerStatus_online_O2OTEST.db]
[2020-06-03 22:40:38,351] INFO: Listing with a limit of 500 IOVs, starting from the highest since. If you need to see more, please increase the limit of returned results.
Since: Run   Insertion Time              Payload                                   Object Type                                           
-----------  --------------------------  ----------------------------------------  ---------------------------------------------------   
215906       2020-06-03 20:40:37.101268  06e5459f27375dd0ac9880be508b87f671fe0a0c  EcalCondTowerObjectContainer<EcalChannelStatusCode>   
 
return code is 90

---> test EcalDCS_O2O_test had ERRORS
 
^^^^ End Test EcalDCS_O2O_test ^^^^

If I try removing my .netrc file (which is more likely to correspond to the setup used in the IB tests), I get a different error:

Package CondTools/Ecal: Running test EcalDCS_O2O_test
 
===== Test "EcalDCS_O2O_test" ====
[2020-06-03 22:43:49,079] INFO: Connecting to pro [frontier://PromptProd/CMS_CONDITIONS]
[2020-06-03 22:43:50,515] INFO: Connecting to EcalDCSTowerStatus_online_O2OTEST.db [sqlite:///EcalDCSTowerStatus_online_O2OTEST.db]
[2020-06-03 22:43:50,606] INFO: Copying tag pro::EcalDCSTowerStatus_online to EcalDCSTowerStatus_online_O2OTEST.db::EcalDCSTowerStatus_online from since=1
[2020-06-03 22:43:50,609] WARNING: Destination tag "EcalDCSTowerStatus_online" already exists.
[2020-06-03 22:43:50,671] INFO: Found 1 iovs and 1 referenced payloads to copy.
[2020-06-03 22:43:50,674] INFO: 1 iov(s) copied.
[2020-06-03 22:43:50,686] INFO: Changes committed.
[2020-06-03 22:43:50,869] INFO: Executing command: cmsRun ./src/CondTools/Ecal/python/EcalDCS_popcon.py  targetFile=2020-06-03_20-43-50-869866.db destinationDatabase=sqlite_file:EcalDCSTowerStatus_online_O2OTEST.db destinationTag=EcalDCSTowerStatus_online 2>&1
----- Begin Fatal Exception 03-Jun-2020 22:43:51 CEST-----------------------
An exception of category 'ConfigFileReadError' occurred while
   [0] Processing the python configuration file named ./src/CondTools/Ecal/python/EcalDCS_popcon.py
Exception Message:
 unknown python problem occurred.
IOError: (2, 'No such file or directory', '/afs/cern.ch/user/c/cawest/.netrc')

At:
  /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc820/external/python/2.7.15-bcolbf2/lib/python2.7/netrc.py(34): __init__
  /tmp/cawest/CMSSW_11_2_X_2020-06-03-1100/python/CondCore/Utilities/credentials.py(13): get_credentials_from_file
  /tmp/cawest/CMSSW_11_2_X_2020-06-03-1100/python/CondCore/Utilities/credentials.py(25): get_credentials
  /tmp/cawest/CMSSW_11_2_X_2020-06-03-1100/python/CondTools/Ecal/db_credentials.py(11): get_db_credentials
  ./src/CondTools/Ecal/python/EcalDCS_popcon.py(24): <module>

----- End Fatal Exception -------------------------------------------------

[2020-06-03 22:43:51,298] INFO: PopCon Analyzer return code is: 90
[2020-06-03 22:43:51,299] ERROR: O2O job failed. Skipping upload.
[2020-06-03 22:43:51,711] INFO: Connecting to EcalDCSTowerStatus_online_O2OTEST.db [sqlite:///EcalDCSTowerStatus_online_O2OTEST.db]
[2020-06-03 22:43:51,749] INFO: Listing with a limit of 500 IOVs, starting from the highest since. If you need to see more, please increase the limit of returned results.
Since: Run   Insertion Time              Payload                                   Object Type                                           
-----------  --------------------------  ----------------------------------------  ---------------------------------------------------   
215906       2020-06-03 20:40:37.101268  06e5459f27375dd0ac9880be508b87f671fe0a0c  EcalCondTowerObjectContainer<EcalChannelStatusCode>   
215906       2020-06-03 20:43:50.577108  06e5459f27375dd0ac9880be508b87f671fe0a0c  EcalCondTowerObjectContainer<EcalChannelStatusCode>   
 
return code is 90

---> test EcalDCS_O2O_test had ERRORS

ggovi · 2020-06-04T13:31:19Z

@christopheralanwest
to run this test you need to have the ECal DB read-only credentials, which I have. Also the IB infrastructure is equipped with them. Looking at the above test summary, it looks like only the unit tests for the CondCore/Utilities package have been executed?
@silviodonato is this expected?

smuzaffar · 2020-06-04T13:36:58Z

@ggovi , yes this is expected. For PR tests we only run unit tests for only checkout packages. In this case only CondCore/Utilities might be the only package which was checked out.

smuzaffar · 2020-06-04T13:39:14Z

by the way, I have restarted the PR tests to explicitly checkout CondTools/Ecal too so that it can run the extra unit test

ggovi · 2020-06-04T13:48:32Z

@smuzaffar thanks, you are right, the changes only involve the CondCore/Utilities package

cmsbuild · 2020-06-04T14:56:25Z

+1
Tested at: 98ead29
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-700a80/6809/summary.html
CMSSW: CMSSW_11_2_X_2020-06-04-1100
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-06-04T14:56:28Z

Comparison job queued.

christopheralanwest · 2020-06-04T14:59:58Z

+1

@ggovi Thanks for the explanation.

cmsbuild · 2020-06-04T15:00:18Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo (and backports should be raised in the release meeting by the corresponding L2)

cmsbuild · 2020-06-04T17:30:25Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-700a80/6809/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 36
DQMHistoTests: Total histograms compared: 2783960
DQMHistoTests: Total failures: 48
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2783862
DQMHistoTests: Total skipped: 50
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 35 files compared)
Checked 152 log files, 16 edm output root files, 36 DQM output files

silviodonato · 2020-06-04T19:19:17Z

+1

Fix for handling of empty popcon files

98ead29

cmsbuild added this to the CMSSW_11_2_X milestone Jun 3, 2020

cmsbuild added alca-pending code-checks-pending comparison-pending db-pending orp-pending pending-signatures tests-pending labels Jun 3, 2020

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 3, 2020

cmsbuild added tests-started and removed tests-pending labels Jun 3, 2020

cmsbuild added tests-approved and removed tests-started labels Jun 3, 2020

cmsbuild added comparison-available and removed comparison-pending labels Jun 3, 2020

cmsbuild added db-approved and removed db-pending labels Jun 3, 2020

ggovi mentioned this pull request Jun 3, 2020

EcalDCS O2O test failing in 11.2.X #30049

Closed

cmsbuild added comparison-pending and removed comparison-available labels Jun 4, 2020

cmsbuild removed the alca-pending label Jun 4, 2020

cmsbuild added alca-approved fully-signed and removed pending-signatures labels Jun 4, 2020

cmsbuild added comparison-available and removed comparison-pending labels Jun 4, 2020

cmsbuild added orp-approved and removed orp-pending labels Jun 4, 2020

cmsbuild merged commit 3f433eb into cms-sw:master Jun 4, 2020

ggovi mentioned this pull request Jun 15, 2020

Fix in PopCon worklow for empy files + DBOuputService clean up: back-port #30238

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix in popcon workflow: handling of empty files #30090

Fix in popcon workflow: handling of empty files #30090

ggovi commented Jun 3, 2020

cmsbuild commented Jun 3, 2020

cmsbuild commented Jun 3, 2020

cmsbuild commented Jun 3, 2020

ggovi commented Jun 3, 2020

cmsbuild commented Jun 3, 2020 •

edited

cmsbuild commented Jun 3, 2020

cmsbuild commented Jun 3, 2020

cmsbuild commented Jun 3, 2020

ggovi commented Jun 3, 2020

christopheralanwest commented Jun 3, 2020

ggovi commented Jun 4, 2020

smuzaffar commented Jun 4, 2020

smuzaffar commented Jun 4, 2020

ggovi commented Jun 4, 2020

cmsbuild commented Jun 4, 2020

cmsbuild commented Jun 4, 2020

christopheralanwest commented Jun 4, 2020

cmsbuild commented Jun 4, 2020

cmsbuild commented Jun 4, 2020

silviodonato commented Jun 4, 2020

Fix in popcon workflow: handling of empty files #30090

Fix in popcon workflow: handling of empty files #30090

Conversation

ggovi commented Jun 3, 2020

PR description:

PR validation:

cmsbuild commented Jun 3, 2020

cmsbuild commented Jun 3, 2020

cmsbuild commented Jun 3, 2020

ggovi commented Jun 3, 2020

cmsbuild commented Jun 3, 2020 • edited

cmsbuild commented Jun 3, 2020

cmsbuild commented Jun 3, 2020

cmsbuild commented Jun 3, 2020

ggovi commented Jun 3, 2020

christopheralanwest commented Jun 3, 2020

ggovi commented Jun 4, 2020

smuzaffar commented Jun 4, 2020

smuzaffar commented Jun 4, 2020

ggovi commented Jun 4, 2020

cmsbuild commented Jun 4, 2020

cmsbuild commented Jun 4, 2020

christopheralanwest commented Jun 4, 2020

cmsbuild commented Jun 4, 2020

cmsbuild commented Jun 4, 2020

silviodonato commented Jun 4, 2020

cmsbuild commented Jun 3, 2020 •

edited