[91x] Ecal pedestals PCL integration

[cerminar]

Complete the integration of the new EcalPedestal workflow in the PCL framework. This allows to run the configuration from cmsDriver and from ConfigDP for Tier0 tests.

I put here for reference the various commands that it is now possible to execute:

Express-like processing

cmsDriver.py step1  --conditions auto:run2_data \
-s ALCAPRODUCER:PromptCalibProdEcalPedestals \
--process ALCARECO \
--data \
--eventcontent ALCARECO \
--scenario pp \
--datatier ALCARECO \
-n 100 \
--filein file:/data/cerminar/data/store_data_Run2016D_TestEnablesEcalHcal_RAW_v2_000_276_315_00000_92AB6184-FD41-E611-A550-02163E01464C.root \
--fileout file:step1.root \
--no_exec

or

python Configuration/DataProcessing/test/RunExpressProcessing.py \
 --scenario AlCaTestEnable \
 --global-tag 90X_dataRun2_v6 \
 --lfn file:/data/cerminar/data/store_data_Run2016D_TestEnablesEcalHcal_RAW_v2_000_276_315_00000_92AB6184-FD41-E611-A550-02163E01464C.root \
 --alcareco PromptCalibProdEcalPedestals

AlCaSkimming processing

cmsDriver.py step2 \
--datatier ALCARECO \
--conditions auto:run2_data \
-s ALCAOUTPUT:PromptCalibProdEcalPedestals \
--eventcontent ALCARECO \
-n 100 \
--filein file:step1.root \
--triggerResultsProcess ALCARECO \
--no_exec

or

python Configuration/DataProcessing/test/RunAlcaSkimming.py \
--scenario AlCaTestEnable \
--lfn=file:output.root \
--global-tag 90X_dataRun2_v6 \
--skims PromptCalibProdEcalPedestals

AlCaHarvesting processing

cmsDriver.py step3 \
--data \
--conditions auto:run2_data \
--scenario pp \
-s ALCAHARVEST:EcalPedestals \
--filein file:PromptCalibProdEcalPedestals.root \
-n -1 \
--no_exec

or

python Configuration/DataProcessing/test/RunAlcaHarvesting.py \
--scenario AlCaTestEnable \
--lfn file:PromptCalibProdEcalPedestals.root \
--dataset /A/B/C \
--global-tag 90X_dataRun2_v6 
--workflows=EcalPedestals

NOTE: I could also add a dedicated relval for this workflow but I am not sure where would this be exercised...maybe by the bot? @davidlange6 advice is needed on this topic.

[cmsbuild]

A new Pull Request was created by @cerminar for master.

It involves the following packages:

Calibration/EcalCalibAlgos
Configuration/AlCa
Configuration/DataProcessing
Configuration/EventContent
Configuration/StandardSequences

@ghellwig, @arunhep, @cerminar, @cmsbuild, @franzoni, @mmusich, @davidlange6 can you please review it and eventually sign? Thanks.
@ghellwig, @makortel, @felicepantaleo, @GiacomoSguazzoni, @tocheng, @argiro, @Martin-Grunewald, @rovere, @ebrondol, @VinInn, @dgulhan this is something you requested to watch as well.
@Muzaffar, @davidlange6, @smuzaffar you are the release manager for this.

cms-bot commands are listed here #13028

[cerminar]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/19144/console Started: 2017/04/12 17:14

[cmsbuild]

+1
Tested at: 2d619f5
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-18332/19144/summary.html

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-18332/19144/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 1630 differences found in the comparisons
• DQMHistoTests: Total files compared: 23
• DQMHistoTests: Total histograms compared: 1921525
• DQMHistoTests: Total failures: 9592
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 1911760
• DQMHistoTests: Total skipped: 173
• DQMHistoTests: Total Missing objects: 0
• Checked 94 log files, 14 edm output root files, 23 DQM output files

[franzoni]

+1

[argiro]

there were two bugs of mine: https://github.com/argiro/cmssw/blob/ecalpedeestalsPCL_9_1_X/Calibration/EcalCalibAlgos/src/ECALpedestalPCLworker.cc#L30 did not use the parameter from conf file https://github.com/argiro/cmssw/blob/ecalpedeestalsPCL_9_1_X/Calibration/EcalCalibAlgos/src/ECALpedestalPCLHarvester.cc#L34 the stringstream is defined here and inside the loop (forgot to take out the latter) S.

…

On 19 Apr 2017, at 17:30, David Lange ***@***.***> wrote: Hi @cerminar - i would not have characterized the string stream changes as a "fix". What was the problem you found with the existing implementation? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[cerminar]

Hello @davidlange6
the previous implementation is buggy:

std::stringstream hname;
hname.str("EcalCalibration/EcalPedestalPCL/");
hname<<"eb_"<<1;
std::cout << hname.str() << std::endl;

results in:
eb_1Calibration/EcalPedestalPCL/

I tried quickly to fight setting the open mode to ios_base::ate with no success.
Anyhow this code is run once per run so I traded code efficiency for developer time. If you have suggestions on how to fix it please let me know.

Cheers,
G

[davidlange6]

indeed, it seems my understanding of string stream is not correct. fortunately, there is a better way

std::string myStr= "EcalCalibration/EcalPedestalPCL/eb_" + std::to_string(1)

[cmsbuild]

Pull request #18332 was updated. @ghellwig, @arunhep, @cerminar, @cmsbuild, @franzoni, @mmusich, @davidlange6 can you please check and sign again.

[cerminar]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/19339/console Started: 2017/04/24 11:50

[cerminar]

Hello @davidlange6
the new commit should take care of the offending stringstream

[cmsbuild]

+1
Tested at: 823e2ca
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-18332/19339/summary.html

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-18332/19339/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 1645 differences found in the comparisons
• DQMHistoTests: Total files compared: 23
• DQMHistoTests: Total histograms compared: 1780178
• DQMHistoTests: Total failures: 6170
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 1773835
• DQMHistoTests: Total skipped: 173
• DQMHistoTests: Total Missing objects: 0
• Checked 94 log files, 14 edm output root files, 23 DQM output files

[franzoni]

transferring a small block to the T2_CH_CERN relval
for relval to be developed
https://cmsweb.cern.ch/phedex/prod/Request::View?request=1003291