Fixes for RunInfo O2O and cleanup of PopCon interface (80X) #14421

diguida · 2016-05-09T16:51:24Z

The RunInfo O2O has been modified:

Make the tag population in synch with the CondDBv2 tag policy. In https://github.com/diguida/cmssw/blob/fix_RunInfo_and_PopCon/CondCore/CondDB/src/IOVEditor.cc#L189-L201, as specified in the comment too, if the tag has synchronisation ANY or VALIDATION, IOV insertion in the past or overriding are allowed, while they are forbidden (an exception is thrown) for "production" tags. The RunInfo code is now protected so that it first tests the input run number against the last IOV of the target tag: if the new run number is smaller than or equal to the latest IOV in the target tag, the job exits, and no new IOVs are added irrespectively of the synchronisation;
Use CondCore API as defined in ConnectionPool in order to authenticate Oracle access via roles, instead of the CORAL API in RelationalAccess;
Improve queries by performing joins on PK-FK (query time improved by factor 2);
Throw exceptions in case CORAL fails to execute a query;
Improve logging messages;
Improve code style;
Review the configuration file to be fully complaint with the O2O online framework.

Removed a dead data member in PopCon.

Added a comment in CondCore/CondDB/src/IOVEditor.cc clarifying the tag update policy w.r.t. tag synchronisation.

Back port of #14420 with one difference: use boost::shared_ptr instead of std::shared_ptr in CMSSW_8_0_X...diguida:fix_RunInfo_and_PopCon_80X#diff-d0d2d7d67286c9941ff28996f956d33cR71 due to smart pointer boost -> std migration in CMSSW_8_1_X.

… is smaller than or equal to the latest IOV, exit.

… for production and offline for testing via VarParsing support.

…ect CORAL RelationalAccess API; use joins in queries; throw in case a query goes wrong; improve code style.

cmsbuild · 2016-05-09T16:58:10Z

A new Pull Request was created by @diguida (Salvatore Di Guida) for CMSSW_8_0_X.

It involves the following packages:

CondCore/CondDB
CondCore/PopCon
CondTools/RunInfo

@ghellwig, @cerminar, @cmsbuild, @franzoni, @ggovi, @mmusich, @davidlange6 can you please review it and eventually sign? Thanks.
@ghellwig, @apfeiffer1, @tocheng this is something you requested to watch as well.
@slava77, @Degano, @smuzaffar you are the release manager for this.

cms-bot commands are list here #13028

mmusich · 2016-05-09T17:51:33Z

please test

cmsbuild · 2016-05-09T17:52:06Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/12879/console

cmsbuild · 2016-05-09T18:13:35Z

Comparison not run due to Build errors (RelVals and Igprof tests were also skipped)

cmsbuild · 2016-05-09T18:13:57Z

-1

Tested at: a36856e

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-14421/12879/summary.html

I found follow errors while testing this PR

Failed tests: Build

Build:

I found an error when building:

>> Compiling edm plugin /build/cmsbuild/jenkins-workarea/workspace/ib-any-integration/CMSSW_8_0_X_2016-05-08-0000/src/CondTools/SiPixel/test/SiPixelFakeCPEGenericErrorParmSourceReader.cc 
>> Compiling edm plugin /build/cmsbuild/jenkins-workarea/workspace/ib-any-integration/CMSSW_8_0_X_2016-05-08-0000/src/CalibMuon/CSCCalibration/plugins/CSCPedestalsDBConditions.cc 
>> Compiling edm plugin /build/cmsbuild/jenkins-workarea/workspace/ib-any-integration/CMSSW_8_0_X_2016-05-08-0000/src/CalibMuon/CSCCalibration/plugins/CSCChipSpeedCorrectionDBConditions.cc 
>> Compiling edm plugin /build/cmsbuild/jenkins-workarea/workspace/ib-any-integration/CMSSW_8_0_X_2016-05-08-0000/src/CalibMuon/CSCCalibration/plugins/CSCNoiseMatrixDBConditions.cc 
/build/cmsbuild/jenkins-workarea/workspace/ib-any-integration/CMSSW_8_0_X_2016-05-08-0000/src/CondTools/SiPixel/test/SiPixelBadModuleReader.cc: In member function 'virtual void SiPixelBadModuleReader::analyze(const edm::Event&, const edm::EventSetup&)':
/build/cmsbuild/jenkins-workarea/workspace/ib-any-integration/CMSSW_8_0_X_2016-05-08-0000/src/CondTools/SiPixel/test/SiPixelBadModuleReader.cc:43:17: error: 'SiPixelQuality' was not declared in this scope
   edm::ESHandle SiPixelBadModule_;
                 ^
/build/cmsbuild/jenkins-workarea/workspace/ib-any-integration/CMSSW_8_0_X_2016-05-08-0000/src/CondTools/SiPixel/test/SiPixelBadModuleReader.cc:43:31: error: template argument 1 is invalid
   edm::ESHandle SiPixelBadModule_;
                               ^

diguida · 2016-05-09T19:53:46Z

The failures are unrelated to this PR, and the tests done on top of CMSSW_8_0_7 with git cms-checkdeps -a were ok:

$ git cms-checkdeps -a
>> Checking CondCore/CondDB CMSSW_8_0_7
   x CondCore/CondDB/src/IOVEditor.cc
>> Checking CondCore/PopCon CMSSW_8_0_7
   x CondCore/PopCon/interface/PopCon.h
   x CondCore/PopCon/src/PopCon.cc
>> Checking CondTools/RunInfo CMSSW_8_0_7
   x CondTools/RunInfo/interface/RunInfoHandler.h
   x CondTools/RunInfo/interface/RunInfoRead.h
   x CondTools/RunInfo/src/RunInfoHandler.cc
   x CondTools/RunInfo/src/RunInfoRead.cc
   x CondTools/RunInfo/test/RunInfoPopConAnalyzer.py
Checking out these packages: 12
CalibMuon/CSCCalibration (header)
CondTools/DQM (header)
CondTools/DT (header)
CondTools/Ecal (header)
CondTools/Hcal (header)
CondTools/RPC (header)
CondTools/SiPixel (header)
DQM/SiStripHistoricInfoClient (header)
DQMOffline/CalibTracker (header)
DQMServices/Diagnostic (header)
OnlineDB/CSCCondDB (header)
OnlineDB/SiStripO2O (header)
$ scram b -r -j 32

Let's have a look to the warnings:

In file included from /build/cmsbuild/jenkins-workarea/workspace/ib-any-integration/CMSSW_8_0_X_2016-05-08-0000/src/CondTools/SiPixel/test/SiPixelBadModuleReader.cc:1:0:
/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_X_2016-05-08-0000/src/CondFormats/SiPixelObjects/interface/SiPixelQuality.h:1:1: warning: null character(s) ignored

 ^
...
In file included from /build/cmsbuild/jenkins-workarea/workspace/ib-any-integration/CMSSW_8_0_X_2016-05-08-0000/src/CondTools/SiPixel/test/SealModules.cc:14:0:
/build/cmsbuild/jenkins-workarea/workspace/ib-any-integration/CMSSW_8_0_X_2016-05-08-0000/src/CondTools/SiPixel/test/SiPixelBadModuleByHandBuilder.h:10:65: warning: /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_X_2016-05-08-0000/src/CondFormats/SiPixelObjects/interface/SiPixelQuality.h is shorter than expected
...
In file included from /build/cmsbuild/jenkins-workarea/workspace/ib-any-integration/CMSSW_8_0_X_2016-05-08-0000/src/CondTools/SiPixel/test/SiPixelBadModuleByHandBuilder.cc:1:0:
/build/cmsbuild/jenkins-workarea/workspace/ib-any-integration/CMSSW_8_0_X_2016-05-08-0000/src/CondTools/SiPixel/test/SiPixelBadModuleByHandBuilder.h:10:65: warning: /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_X_2016-05-08-0000/src/CondFormats/SiPixelObjects/interface/SiPixelQuality.h is shorter than expected

these ones are not visible in the base build

/build/cmsbuild/jenkins-workarea/workspace/ib-any-integration/CMSSW_8_0_X_2016-05-08-0000/src/CondTools/SiPixel/test/SiPixelGainCalibrationRejectNoisyAndDead.cc: In member function 'void SiPixelGainCalibrationRejectNoisyAndDead::fillDatabase(const edm::EventSetup&)':
/build/cmsbuild/jenkins-workarea/workspace/ib-any-integration/CMSSW_8_0_X_2016-05-08-0000/src/CondTools/SiPixel/test/SiPixelGainCalibrationRejectNoisyAndDead.cc:236:32: warning: comparison of constant '2' with boolean expression is always false [-Wbool-compare]
  else if(insertnoisypixelsindb_==2){
                                ^

This one is visible in the base build

The errors are:

/build/cmsbuild/jenkins-workarea/workspace/ib-any-integration/CMSSW_8_0_X_2016-05-08-0000/src/CondTools/SiPixel/test/SiPixelBadModuleReader.cc: In member function 'virtual void SiPixelBadModuleReader::analyze(const edm::Event&, const edm::EventSetup&)':
/build/cmsbuild/jenkins-workarea/workspace/ib-any-integration/CMSSW_8_0_X_2016-05-08-0000/src/CondTools/SiPixel/test/SiPixelBadModuleReader.cc:43:17: error: 'SiPixelQuality' was not declared in this scope
   edm::ESHandle<SiPixelQuality> SiPixelBadModule_;
                 ^
[plus all the chain]
...
In file included from /build/cmsbuild/jenkins-workarea/workspace/ib-any-integration/CMSSW_8_0_X_2016-05-08-0000/src/CondTools/SiPixel/test/SiPixelBadModuleByHandBuilder.cc:1:0:
/build/cmsbuild/jenkins-workarea/workspace/ib-any-integration/CMSSW_8_0_X_2016-05-08-0000/src/CondTools/SiPixel/test/SiPixelBadModuleByHandBuilder.h:16:64: error: 'SiPixelQuality' was not declared in this scope
 class SiPixelBadModuleByHandBuilder : public ConditionDBWriter<SiPixelQuality> {
                                                                ^
[plus all the chain]

Running now on top of CMSSW_8_0_X_2016-05-08-0000 with git cms-merge-topic to try to reproduce the issue.

smuzaffar · 2016-05-10T05:58:33Z

please test
looks like it was CVMFS issue on vocms0302

cmsbuild · 2016-05-10T05:59:53Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/12886/console

cmsbuild · 2016-05-10T08:28:47Z

+1
Tested at: a36856e
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-14421/12886/summary.html

cmsbuild · 2016-05-10T10:21:13Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-14421/12886/summary.html

ggovi · 2016-05-12T09:11:35Z

+1

mmusich · 2016-05-24T08:24:48Z

+1

cmsbuild · 2016-05-24T08:25:10Z

This pull request is fully signed and it will be integrated in one of the next CMSSW_8_0_X IBs (tests are also fine). This pull request requires discussion in the ORP meeting before it's merged. @slava77, @davidlange6, @Degano, @smuzaffar

davidlange6 · 2016-05-25T15:41:24Z

+1

Salvatore Di Guida added 9 commits May 6, 2016 14:43

Fix RunInfo O2O: check if a transfer is needed. If the new run number…

3d77e35

… is smaller than or equal to the latest IOV, exit.

Fix RunInfo O2O: Improve logging messages.

8d86279

Fix RunInfo O2O: Improve configuration file. It can be used in online…

615b8aa

… for production and offline for testing via VarParsing support.

Fix RunInfo O2O: use authentication credentials via roles.

da9d084

Improve RunInfo PopCon application: use ConnectionPool instead of dir…

cd88b3b

…ect CORAL RelationalAccess API; use joins in queries; throw in case a query goes wrong; improve code style.

Fix RunInfo O2O: avoid an un-needed copy.

e3b7118

Style changes for RunInfo PopCon application.

e1cb1a2

Fix for RunInfo O2O: use MessageLogger API, and remove cout calls.

97f178d

Cleanup and style fixes in PopCon.

3b0355e

cmsbuild added this to the Next CMSSW_8_0_X milestone May 9, 2016

cmsbuild added alca-pending db-pending pending-signatures tests-pending orp-pending comparison-pending labels May 9, 2016

Added comment in IOVEditor::flush concerning the update policy.

a36856e

diguida force-pushed the fix_RunInfo_and_PopCon_80X branch from 38ff12d to a36856e Compare May 9, 2016 17:18

cmsbuild added tests-started and removed tests-pending labels May 9, 2016

cmsbuild added comparison-notrun and removed comparison-pending labels May 9, 2016

cmsbuild added tests-rejected and removed comparison-notrun tests-started labels May 9, 2016

cmsbuild added the comparison-pending label May 9, 2016

cmsbuild added tests-pending and removed tests-rejected labels May 10, 2016

cmsbuild added tests-started and removed tests-pending labels May 10, 2016

cmsbuild added tests-approved and removed tests-started labels May 10, 2016

cmsbuild added comparison-available and removed comparison-pending labels May 10, 2016

cmsbuild added db-approved and removed db-pending labels May 12, 2016

cmsbuild added alca-approved fully-signed and removed alca-pending pending-signatures labels May 24, 2016

cmsbuild added orp-approved and removed orp-pending labels May 25, 2016

cmsbuild merged commit c5a8dc8 into cms-sw:CMSSW_8_0_X May 25, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes for RunInfo O2O and cleanup of PopCon interface (80X) #14421

Fixes for RunInfo O2O and cleanup of PopCon interface (80X) #14421

diguida commented May 9, 2016 •

edited

cmsbuild commented May 9, 2016

mmusich commented May 9, 2016

cmsbuild commented May 9, 2016 •

edited

cmsbuild commented May 9, 2016

cmsbuild commented May 9, 2016

diguida commented May 9, 2016

smuzaffar commented May 10, 2016

cmsbuild commented May 10, 2016 •

edited

cmsbuild commented May 10, 2016

cmsbuild commented May 10, 2016

ggovi commented May 12, 2016

mmusich commented May 24, 2016

cmsbuild commented May 24, 2016

davidlange6 commented May 25, 2016

Fixes for RunInfo O2O and cleanup of PopCon interface (80X) #14421

Fixes for RunInfo O2O and cleanup of PopCon interface (80X) #14421

Conversation

diguida commented May 9, 2016 • edited

cmsbuild commented May 9, 2016

mmusich commented May 9, 2016

cmsbuild commented May 9, 2016 • edited

cmsbuild commented May 9, 2016

cmsbuild commented May 9, 2016

diguida commented May 9, 2016

smuzaffar commented May 10, 2016

cmsbuild commented May 10, 2016 • edited

cmsbuild commented May 10, 2016

cmsbuild commented May 10, 2016

ggovi commented May 12, 2016

mmusich commented May 24, 2016

cmsbuild commented May 24, 2016

davidlange6 commented May 25, 2016

diguida commented May 9, 2016 •

edited

cmsbuild commented May 9, 2016 •

edited

cmsbuild commented May 10, 2016 •

edited