Run2-alca82 Change output format for MinBias AlCaReco jobs #19524

bsunanda · 2017-07-03T14:14:26Z

Remove the Digi collection and add Raw and Trigger information

cmsbuild · 2017-07-03T14:14:51Z

A new Pull Request was created by @bsunanda for master.

It involves the following packages:

Calibration/HcalAlCaRecoProducers

@ghellwig, @arunhep, @cerminar, @cmsbuild, @franzoni, @davidlange6, @lpernie can you please review it and eventually sign? Thanks.
@ghellwig, @mmusich, @tocheng this is something you requested to watch as well.
@davidlange6 you are the release manager for this.

cms-bot commands are listed here

bsunanda · 2017-07-03T14:19:54Z

@cmsbuild Please test

cmsbuild · 2017-07-03T14:20:08Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/21095/console Started: 2017/07/03 16:21

cmsbuild · 2017-07-03T14:45:15Z

Pull request #19524 was updated. @ghellwig, @arunhep, @cerminar, @cmsbuild, @franzoni, @davidlange6, @lpernie can you please check and sign again.

bsunanda · 2017-07-03T15:06:27Z

@cmsbuild Please test

cmsbuild · 2017-07-03T15:06:49Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/21099/console Started: 2017/07/03 17:07

cmsbuild · 2017-07-03T16:38:45Z

+1
Tested at: 94600ca
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19524/21099/summary.html

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
b1580cd
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19524/21099/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19524/21099/git-merge-result

cmsbuild · 2017-07-03T16:38:48Z

Comparison job queued.

cmsbuild · 2017-07-03T19:12:45Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19524/21099/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 22
DQMHistoTests: Total histograms compared: 1756063
DQMHistoTests: Total failures: 15042
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 1740855
DQMHistoTests: Total skipped: 166
DQMHistoTests: Total Missing objects: 0
Checked 90 log files, 14 edm output root files, 22 DQM output files

arunhep · 2017-07-04T12:38:53Z

Calibration/HcalAlCaRecoProducers/python/ALCARECOHcalCalMinBias_Output_cff.py

+        'keep *_hltTriggerSummaryAOD_*_*',
+        'keep *_TriggerResults_*_*',
+        'keep FEDRawDataCollection_rawDataCollector_*_*',
+        'keep FEDRawDataCollection_source_*_*',


@bsunanda
how much increase in size of event content and eventually dataset we expect by keeping all this information?

franzoni · 2017-07-24T16:15:14Z

We have a back up solution to bring back the data volume at the same level as previous time

Can you elaborate what such "solution" ?
The size of the relevant 2x ALCARECO for 2017B is (see below) is 1.2 TB, hence not negligible. If
we need a factor x2 we need to be convinced it's essential...

/HcalNZS/Run2017B-HcalCalMinBias-PromptReco-v1/ALCARECO

/HcalNZS/Run2017B-HcalCalMinBias-PromptReco-v2/ALCARECO

cmsbuild · 2017-07-24T16:15:32Z

Pull request #19524 was updated. @ghellwig, @arunhep, @cerminar, @cmsbuild, @franzoni, @davidlange6, @lpernie can you please check and sign again.

bsunanda · 2017-07-24T16:16:04Z

@cmsbuild Please test

cmsbuild · 2017-07-24T16:16:20Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/21715/console Started: 2017/07/24 18:16

bsunanda · 2017-07-24T16:20:15Z

The old Data Format used to require 0.21 MB per event and the currently proposed one also needs similar amount (a bit less but not significantly lower). So the overall space requirement will be similar.

cmsbuild · 2017-07-24T17:29:38Z

+1
Tested at: f0f8e05
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19524/21715/summary.html

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
a1f0a3b
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19524/21715/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19524/21715/git-merge-result

cmsbuild · 2017-07-24T17:29:41Z

Comparison job queued.

cmsbuild · 2017-07-24T19:48:55Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19524/21715/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 23
DQMHistoTests: Total histograms compared: 2355459
DQMHistoTests: Total failures: 29950
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2325343
DQMHistoTests: Total skipped: 166
DQMHistoTests: Total Missing objects: 0
Checked 93 log files, 14 edm output root files, 23 DQM output files

bsunanda · 2017-07-25T14:37:49Z

@franzoni Do you have further doubts for this PR?

franzoni · 2017-07-25T14:46:30Z

+1

cmsbuild · 2017-07-25T14:46:51Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request requires discussion in the ORP meeting before it's merged. @davidlange6, @smuzaffar

davidlange6 · 2017-07-26T05:56:06Z

merge

Run2-alca86 Backport corrections to Minimum bias and pedestal o/p commands (#19524)

cmsbuild added this to the CMSSW_9_3_X milestone Jul 3, 2017

cmsbuild added alca-pending comparison-pending orp-pending pending-signatures tests-pending labels Jul 3, 2017

cmsbuild added tests-started tests-pending and removed tests-pending tests-started labels Jul 3, 2017

cmsbuild added tests-started and removed tests-pending labels Jul 3, 2017

cmsbuild added tests-approved and removed tests-started labels Jul 3, 2017

cmsbuild added comparison-available and removed comparison-pending labels Jul 3, 2017

arunhep reviewed Jul 4, 2017

View reviewed changes

Sunanda added 3 commits July 17, 2017 17:21

Change output format for MinBias AlCaReco jobs

13c19dc

Change output format for Pedestal AlCaReco jobs

ea4a780

Update Pedestal

bf08810

bsunanda force-pushed the Run2-alca82 branch from 94600ca to bf08810 Compare July 17, 2017 15:23

Update the output commands to avoid storage of Raw data

f0f8e05

cmsbuild added comparison-pending tests-pending and removed comparison-available tests-approved labels Jul 24, 2017

cmsbuild added tests-started and removed tests-pending labels Jul 24, 2017

cmsbuild added tests-approved and removed tests-started labels Jul 24, 2017

cmsbuild added comparison-available and removed comparison-pending labels Jul 24, 2017

cmsbuild added alca-approved fully-signed and removed alca-pending pending-signatures labels Jul 25, 2017

cmsbuild merged commit 74e2674 into cms-sw:master Jul 26, 2017

cmsbuild added a commit that referenced this pull request Aug 1, 2017

Merge pull request #19908 from bsunanda/Run2-alca86

8d23312

Run2-alca86 Backport corrections to Minimum bias and pedestal o/p commands (#19524)

bsunanda deleted the Run2-alca82 branch August 8, 2017 10:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run2-alca82 Change output format for MinBias AlCaReco jobs #19524

Run2-alca82 Change output format for MinBias AlCaReco jobs #19524

bsunanda commented Jul 3, 2017

cmsbuild commented Jul 3, 2017

bsunanda commented Jul 3, 2017

cmsbuild commented Jul 3, 2017 •

edited

Loading

cmsbuild commented Jul 3, 2017

bsunanda commented Jul 3, 2017

cmsbuild commented Jul 3, 2017 •

edited

Loading

cmsbuild commented Jul 3, 2017

cmsbuild commented Jul 3, 2017

cmsbuild commented Jul 3, 2017

arunhep Jul 4, 2017

franzoni commented Jul 24, 2017

cmsbuild commented Jul 24, 2017

bsunanda commented Jul 24, 2017

cmsbuild commented Jul 24, 2017 •

edited

Loading

bsunanda commented Jul 24, 2017

cmsbuild commented Jul 24, 2017

cmsbuild commented Jul 24, 2017

cmsbuild commented Jul 24, 2017

bsunanda commented Jul 25, 2017

franzoni commented Jul 25, 2017

cmsbuild commented Jul 25, 2017

davidlange6 commented Jul 26, 2017

Run2-alca82 Change output format for MinBias AlCaReco jobs #19524

Run2-alca82 Change output format for MinBias AlCaReco jobs #19524

Conversation

bsunanda commented Jul 3, 2017

cmsbuild commented Jul 3, 2017

bsunanda commented Jul 3, 2017

cmsbuild commented Jul 3, 2017 • edited Loading

cmsbuild commented Jul 3, 2017

bsunanda commented Jul 3, 2017

cmsbuild commented Jul 3, 2017 • edited Loading

cmsbuild commented Jul 3, 2017

cmsbuild commented Jul 3, 2017

cmsbuild commented Jul 3, 2017

arunhep Jul 4, 2017

Choose a reason for hiding this comment

franzoni commented Jul 24, 2017

cmsbuild commented Jul 24, 2017

bsunanda commented Jul 24, 2017

cmsbuild commented Jul 24, 2017 • edited Loading

bsunanda commented Jul 24, 2017

cmsbuild commented Jul 24, 2017

cmsbuild commented Jul 24, 2017

cmsbuild commented Jul 24, 2017

bsunanda commented Jul 25, 2017

franzoni commented Jul 25, 2017

cmsbuild commented Jul 25, 2017

davidlange6 commented Jul 26, 2017

cmsbuild commented Jul 3, 2017 •

edited

Loading

cmsbuild commented Jul 3, 2017 •

edited

Loading

cmsbuild commented Jul 24, 2017 •

edited

Loading