Add cmsDriver option --maxmem_profile to prefix LD_PRELOAD="libPerfToolsAllocMonitorPreload.so libPerfToolsMaxMemoryPreload.so" to cmsRun #44001

gartung · 2024-02-16T19:55:25Z

Add cmsDriver option --maxmem_profile to prefix LD_PRELOAD="libPerfToolsAllocMonitorPreload.so libPerfToolsMaxMemoryPreload.so" to cmsRun

…olsAllocMonitorPreload.so libPerfToolsMaxMemoryPreload.so" to cmsRun

cmsbuild · 2024-02-16T19:55:54Z

cms-bot internal usage

cmsbuild · 2024-02-16T20:00:55Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44001/38921

This PR adds an extra 20KB to repository

cmsbuild · 2024-02-16T20:01:13Z

A new Pull Request was created by @gartung for master.

It involves the following packages:

Configuration/Applications (operations)

@fabiocos, @rappoccio, @cmsbuild, @davidlange6, @antoniovilela can you please review it and eventually sign? Thanks.
@missirol, @fabiocos, @makortel, @Martin-Grunewald this is something you requested to watch as well.
@antoniovilela, @sextonkennedy, @rappoccio you are the release manager for this.

cms-bot commands are listed here

Configuration/Applications/python/Options.py

Configuration/Applications/python/cmsDriverOptions.py

makortel · 2024-02-16T20:31:15Z

Are there enough hooks in runTheMatrix that this new cmsDriver option can be easily passed for all steps of all workflows in PR tests?

gartung · 2024-02-16T20:49:29Z

I don't know. But first I have to determine what causes this error

cmsDriver.py: error: argument --maxmem_profile: expected one argument

cmsbuild · 2024-02-16T20:52:02Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44001/38923

This PR adds an extra 20KB to repository

cmsbuild · 2024-02-16T20:52:23Z

Pull request #44001 was updated. @antoniovilela, @fabiocos, @davidlange6, @rappoccio, @cmsbuild can you please check and sign again.

Configuration/Applications/python/Options.py

makortel · 2024-02-16T20:56:16Z

How/where are the other profiling options passed in the profiling IB/PR jobs?

gartung · 2024-02-16T21:00:41Z

runTheMatrix.py $WF --command " -n $EVENTS --profile $prof --customise Validation/Performance/IgProfInfo.customise --no_exec"
The --profile $prof is the argument that gets converted to a cmsDriver.py argumnent,.

gartung · 2024-02-28T22:54:38Z

please test with cms-sw/cms-bot#2185

cmsbuild · 2024-02-29T06:01:35Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-46f60f/37801/summary.html
COMMIT: 8a8e495
CMSSW: CMSSW_14_1_X_2024-02-28-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/44001/37801/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 7206 lines to the logs
Reco comparison results: 57 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3011576
DQMHistoTests: Total failures: 9
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3011545
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 200 log files, 161 edm output root files, 48 DQM output files
TriggerResults: no differences found

gartung · 2024-02-29T14:06:35Z

please test with cms-sw/cms-bot#2185

cmsbuild · 2024-02-29T17:01:16Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-46f60f/37813/summary.html
COMMIT: 8a8e495
CMSSW: CMSSW_14_1_X_2024-02-29-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/44001/37813/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 7353 lines to the logs
Reco comparison results: 42 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3338784
DQMHistoTests: Total failures: 3
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3338759
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 205 log files, 166 edm output root files, 49 DQM output files
TriggerResults: no differences found

gartung · 2024-03-01T17:14:17Z

please test with cms-sw/cms-bot#2185

gartung · 2024-03-01T17:19:35Z

Example of the variance from the last five completed runTheMatrix test workflow 25034.999

[gartung@cmsdev25 PR-46f60f]$ tail -n6 */runTheMatrix-results/25034.999*/step3*.log
==> 37781/runTheMatrix-results/25034.999_TTbar_14TeV+2026D98PU_PMXS1S2PR/step3_TTbar_14TeV+2026D98PU_PMXS1S2PR.log <==
Memory Report
  total memory requested: 178127362204
  max memory used: 3888251584
  presently used: 659294632
  # allocations calls:   1649760412
  # deallocations calls: 1648912864

==> 37793/runTheMatrix-results/25034.999_TTbar_14TeV+2026D98PU_PMXS1S2PR/step3_TTbar_14TeV+2026D98PU_PMXS1S2PR.log <==
Memory Report
  total memory requested: 178125038615
  max memory used: 3888251360
  presently used: 659294616
  # allocations calls:   1649758031
  # deallocations calls: 1648910484

==> 37798/runTheMatrix-results/25034.999_TTbar_14TeV+2026D98PU_PMXS1S2PR/step3_TTbar_14TeV+2026D98PU_PMXS1S2PR.log <==
Memory Report
  total memory requested: 178128497624
  max memory used: 3888251456
  presently used: 659294616
  # allocations calls:   1649762203
  # deallocations calls: 1648914656

==> 37801/runTheMatrix-results/25034.999_TTbar_14TeV+2026D98PU_PMXS1S2PR/step3_TTbar_14TeV+2026D98PU_PMXS1S2PR.log <==
Memory Report
  total memory requested: 178126101922
  max memory used: 3888250096
  presently used: 659293400
  # allocations calls:   1649761266
  # deallocations calls: 1648913721

==> 37813/runTheMatrix-results/25034.999_TTbar_14TeV+2026D98PU_PMXS1S2PR/step3_TTbar_14TeV+2026D98PU_PMXS1S2PR.log <==
Memory Report
  total memory requested: 178175229722
  max memory used: 3903582880
  presently used: 674592408
  # allocations calls:   1649958856
  # deallocations calls: 1649044997

cmsbuild · 2024-03-01T20:18:09Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-46f60f/37857/summary.html
COMMIT: 8a8e495
CMSSW: CMSSW_14_1_X_2024-03-01-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/44001/37857/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 7354 lines to the logs
Reco comparison results: 68 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3338784
DQMHistoTests: Total failures: 1361
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3337401
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 205 log files, 166 edm output root files, 49 DQM output files
TriggerResults: no differences found

antoniovilela · 2024-03-04T19:10:04Z

+1

cmsbuild · 2024-03-04T19:10:28Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will be automatically merged.
Notice This PR was tested with additional Pull Request(s), please also merge them if necessary: cms-sw/cms-bot#2185

Add cmsDriver option --maxmem_profile to prefix LD_PRELOAD="libPerfTo…

6205926

…olsAllocMonitorPreload.so libPerfToolsMaxMemoryPreload.so" to cmsRun

cmsbuild added this to the CMSSW_14_1_X milestone Feb 16, 2024

cmsbuild added operations-pending pending-signatures tests-pending orp-pending code-checks-pending labels Feb 16, 2024

gartung marked this pull request as ready for review February 16, 2024 19:57

cmsbuild added code-checks-approved and removed code-checks-pending labels Feb 16, 2024

makortel mentioned this pull request Feb 16, 2024

Add option to do the proper LD_PRELOAD to cmsDriver cms-sw/framework-team#671

Closed

makortel reviewed Feb 16, 2024

View reviewed changes

Configuration/Applications/python/Options.py Outdated Show resolved Hide resolved

makortel reviewed Feb 16, 2024

View reviewed changes

Configuration/Applications/python/cmsDriverOptions.py Outdated Show resolved Hide resolved

makortel reviewed Feb 16, 2024

View reviewed changes

Configuration/Applications/python/cmsDriverOptions.py Outdated Show resolved Hide resolved

Remove redundant checkout on options.maxmem_profile

f7b13e8

cmsbuild added code-checks-pending and removed code-checks-approved labels Feb 16, 2024

Change Service to monitor

a6f32ea

cmsbuild added code-checks-approved and removed code-checks-pending labels Feb 16, 2024

makortel reviewed Feb 16, 2024

View reviewed changes

Configuration/Applications/python/Options.py Outdated Show resolved Hide resolved

cmsbuild added tests-started and removed tests-approved labels Feb 28, 2024

cmsbuild added tests-approved and removed tests-started labels Feb 29, 2024

cmsbuild added tests-started and removed tests-approved labels Feb 29, 2024

cmsbuild added tests-approved and removed tests-started labels Feb 29, 2024

cmsbuild added tests-started and removed tests-approved labels Mar 1, 2024

cmsbuild added tests-approved and removed tests-started labels Mar 1, 2024

cmsbuild added operations-approved fully-signed orp-approved and removed operations-pending pending-signatures orp-pending labels Mar 4, 2024

cmsbuild merged commit 1f01765 into cms-sw:master Mar 4, 2024
12 checks passed

gartung deleted the gartung-maxmempreload-cmsdriveropion branch March 4, 2024 19:12

cmsbuild mentioned this pull request Mar 9, 2024

Update to CUDA 12.4.0 cms-sw/cmsdist#9046

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add cmsDriver option --maxmem_profile to prefix LD_PRELOAD="libPerfToolsAllocMonitorPreload.so libPerfToolsMaxMemoryPreload.so" to cmsRun #44001

Add cmsDriver option --maxmem_profile to prefix LD_PRELOAD="libPerfToolsAllocMonitorPreload.so libPerfToolsMaxMemoryPreload.so" to cmsRun #44001

gartung commented Feb 16, 2024 •

edited

Loading

cmsbuild commented Feb 16, 2024 •

edited

Loading

cmsbuild commented Feb 16, 2024

cmsbuild commented Feb 16, 2024

makortel commented Feb 16, 2024

gartung commented Feb 16, 2024

cmsbuild commented Feb 16, 2024

cmsbuild commented Feb 16, 2024

makortel commented Feb 16, 2024

gartung commented Feb 16, 2024

gartung commented Feb 28, 2024

cmsbuild commented Feb 29, 2024

gartung commented Feb 29, 2024

cmsbuild commented Feb 29, 2024

gartung commented Mar 1, 2024

gartung commented Mar 1, 2024

cmsbuild commented Mar 1, 2024

antoniovilela commented Mar 4, 2024

cmsbuild commented Mar 4, 2024

Add cmsDriver option --maxmem_profile to prefix LD_PRELOAD="libPerfToolsAllocMonitorPreload.so libPerfToolsMaxMemoryPreload.so" to cmsRun #44001

Add cmsDriver option --maxmem_profile to prefix LD_PRELOAD="libPerfToolsAllocMonitorPreload.so libPerfToolsMaxMemoryPreload.so" to cmsRun #44001

Conversation

gartung commented Feb 16, 2024 • edited Loading

cmsbuild commented Feb 16, 2024 • edited Loading

cmsbuild commented Feb 16, 2024

cmsbuild commented Feb 16, 2024

makortel commented Feb 16, 2024

gartung commented Feb 16, 2024

cmsbuild commented Feb 16, 2024

cmsbuild commented Feb 16, 2024

makortel commented Feb 16, 2024

gartung commented Feb 16, 2024

gartung commented Feb 28, 2024

cmsbuild commented Feb 29, 2024

Comparison Summary

gartung commented Feb 29, 2024

cmsbuild commented Feb 29, 2024

Comparison Summary

gartung commented Mar 1, 2024

gartung commented Mar 1, 2024

cmsbuild commented Mar 1, 2024

Comparison Summary

antoniovilela commented Mar 4, 2024

cmsbuild commented Mar 4, 2024

gartung commented Feb 16, 2024 •

edited

Loading

cmsbuild commented Feb 16, 2024 •

edited

Loading