Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix memory blowup in ALCAHARVEST 80X (backport of #14721) #14722

Merged
merged 8 commits into from Jun 7, 2016
Merged

Fix memory blowup in ALCAHARVEST 80X (backport of #14721) #14722

merged 8 commits into from Jun 7, 2016

Conversation

ghellwig
Copy link

@ghellwig ghellwig commented Jun 1, 2016

Fixes Memory blowup reported in Reconstruction Development HN.

  • the blowup was caused by the merging of the FileBlobCollections at the end of the run
  • making this collection a lumi product instead of a run product keeps the memory consumption stable without changing the alignment result (see plots below for the first 600 files of run 273730)

Memory consumption of SiPixelAli workflow:
pede_memory_usage_log

Geometry comparison:
global_tracker_2_final

Backport of #14721

@gregor-mittag
Added 'Mille::resetOutputFile()'.
4b8d2bb 
 - first step towards binary FileBlob as lumi product
 - general clean-up of the 'Mille' class
@gregor-mittag
Implement luminosity block begin and end methods for MillePede.
df29429
@gregor-mittag
Make the producer and the analyzer of the binary FileBlobCollection l…
18f53e6 
…umi-based.

 - updated 'MillePedeFileExtractor' to edm::one::Analyzer
 - general clean-up of the 'MillePedeFileConverter' and 'MillePedeFileExtractor'
@gregor-mittag
Add a switch to select between offline use and PCL.
ae6be4f 
 - at PCL the binaries are reset at lumi-section boundaries
   -> this is not wanted for the offline use where one binary is
      produced per mille job
      -> default value of 'runAtPCL' is False
@gregor-mittag
Fix binary numbering for lumi-based MillePedeFileExtractor.
777955c
@gregor-mittag
Fix wrong warning for missing binary in PCL pede mode.
5a84e1a
@gregor-mittag
Put FileBlobCollection into Lumi even if it is empty.
e170549 
 - fixes a bug in case of merging products from files
   which (partly) contain no collection
@gregor-mittag
Introduce configurable limit on extracted number of binaries.
f31bc46
@cmsbuild cmsbuild added this to the Next CMSSW_8_0_X milestone Jun 1, 2016
@ghellwig
Copy link
Author

ghellwig commented Jun 1, 2016

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 1, 2016

A new Pull Request was created by @ghellwig (Gregor Mittag) for CMSSW_8_0_X.

It involves the following packages:

Alignment/CommonAlignmentProducer
Alignment/MillePedeAlignmentAlgorithm

@ghellwig, @cerminar, @cmsbuild, @franzoni, @mmusich, @davidlange6 can you please review it and eventually sign? Thanks.
@ghellwig, @pakhotin, @tocheng, @tlampen, @mschrode, @mmusich this is something you requested to watch as well.
@slava77, @Degano, @smuzaffar you are the release manager for this.

cms-bot commands are list here #13028

@ghellwig
Copy link
Author

ghellwig commented Jun 1, 2016

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 1, 2016 •
edited

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/13299/console

@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 1, 2016

+1
Tested at: f31bc46
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-14722/13299/summary.html

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
a6764dd
c48d2d0
a14cc8b
96d34fa
442a8d5
ad3570c
227fbd7
056d3ff
2448eb8
0b6220f
4b81a73
88d1819
14dc695
0f04d39
5370cda
f9dacee
d74708f
5ce8684
fbb1567
b1eae2a
82f9fdd
bc82047
0bf418a
d4a4144
ba46a23
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-14722/13299/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-14722/13299/git-merge-result

@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 1, 2016

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-14722/13299/summary.html

The workflows 1001.0 have different files in step1_dasquery.log than the ones found in the baseline. You may want to check and retrigger the tests if necessary. You can check it in the "files" directory in the results of the comparisons

@mmusich
Copy link
Contributor

mmusich commented Jun 1, 2016

+1

@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 1, 2016

This pull request is fully signed and it will be integrated in one of the next CMSSW_8_0_X IBs (tests are also fine). This pull request requires discussion in the ORP meeting before it's merged. @slava77, @davidlange6, @Degano, @smuzaffar

@davidlange6
Copy link
Contributor

+1

@cmsbuild cmsbuild merged commit b33df21 into cms-sw:CMSSW_8_0_X Jun 7, 2016
@ghellwig ghellwig deleted the fix-memory-blowup-ALCAHARVEST_80X branch June 14, 2016 08:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
alca-approved comparison-available fully-signed orp-approved tests-approved
Projects
None yet
Milestone
CMSSW_8_0_X
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@ghellwig @cmsbuild @mmusich @davidlange6 @gregor-mittag