Report CPU info to HTCondor monitoring #15767

bbockelm · 2016-09-07T17:36:12Z

This PR fixes #15763 and #15764 by adding CPU information to the HTCondor monitoring. In particular, it adds:

CPU time usage.
CPU model information.
CPU time usage from child processes.

This adds CPU usage information to the HTCondor service reporting; it reports the entire CPU (user + system) usage for this process. As the "job CPU" information reported by the framework excludes the framework startup costs, the "total CPU" is slightly more than what the Timing service prints on stdout.

This commit has the framework forward the CPU model information to the HTCondor monitoring.

With this commit, getCPU() additionally returns all usage information from child processes.

cmsbuild · 2016-09-07T17:38:22Z

A new Pull Request was created by @bbockelm (Brian Bockelman) for CMSSW_8_1_X.

It involves the following packages:

FWCore/Services
FWCore/Utilities

@cmsbuild, @smuzaffar, @Dr15Jones, @davidlange6 can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @makortel, @wddgit, @wmtan this is something you requested to watch as well.
@slava77, @smuzaffar you are the release manager for this.

cms-bot commands are list here #13028

davidlange6 · 2016-09-14T18:31:02Z

please test

cmsbuild · 2016-09-14T18:31:20Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/15144/console

Dr15Jones · 2016-09-14T19:20:30Z

FWCore/Services/plugins/CPU.cc

    private:
      int totalNumberCPUs_;
      double averageCoreSpeed_;
      bool reportCPUProperties_;
-


Updating the member data in the cpuInfo call is not thread safe. As far as I can tell, there is no need for totalNumberCPUs_ or averageCoreSpeed_ to be member data. It also looks like reportCPUProperties_ could be const.

Dr15Jones · 2016-09-14T19:22:06Z

FWCore/Services/plugins/CPU.cc

+        return cpuInfoImpl(models, avgSpeed, nullptr);
+    }
+
+    bool CPU::cpuInfoImpl(std::string &result_models, double &result_avg_speed, Service<JobReport>* reportSvc)


This seems to do a lot more than just the work need to return result_models and result_avg_speed. I think the function should be better factored so that it doesn't know about the JobReport.

cmsbuild · 2016-09-14T19:30:28Z

+1
Tested at: f3c0605
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-15767/15144/summary.html

cmsbuild · 2016-09-14T20:25:41Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-15767/15144/summary.html

Now, one function will parse the cpuinfo file - each different statistic we want will utilize this information in a separate method.

cmsbuild · 2016-09-29T01:42:00Z

Pull request #15767 was updated. @cmsbuild, @smuzaffar, @Dr15Jones, @davidlange6 can you please check and sign again.

…_cpu_info Conflicts: FWCore/Services/plugins/CPU.cc

cmsbuild · 2016-09-29T01:49:55Z

Pull request #15767 was updated. @cmsbuild, @smuzaffar, @Dr15Jones, @davidlange6 can you please check and sign again.

Dr15Jones · 2016-10-06T18:48:22Z

please test

cmsbuild · 2016-10-06T18:48:36Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/15579/console

cmsbuild · 2016-10-06T21:05:15Z

+1
Tested at: e8d9da8
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-15767/15579/summary.html

cmsbuild · 2016-10-06T22:06:01Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-15767/15579/summary.html

Dr15Jones · 2016-10-06T22:16:49Z

@bbockelm how do we know if this is working?

bbockelm · 2016-10-09T17:32:46Z

Hi Chris,

Add the following two lines to your pset:

process.CPU = cms.Service("CPU", reportCPUProperties=cms.untracked.bool(True))
process.CondorStatusService = cms.Service("CondorStatusService", tag=cms.untracked.string("DIGI"), debug=cms.untracked.bool(True))

You should see the condor invocations echoed to stdout and the CPU information in the FJR.

Brian

Dr15Jones · 2016-10-09T22:21:49Z

+1

cmsbuild · 2016-10-09T22:22:04Z

This pull request is fully signed and it will be integrated in one of the next CMSSW_8_1_X IBs (tests are also fine). This pull request requires discussion in the ORP meeting before it's merged. @slava77, @davidlange6, @smuzaffar

davidlange6 · 2016-10-09T22:29:41Z

+1

dan131riley · 2016-10-24T17:12:59Z

Hi @bbockelm

If I add

process.Timing = cms.Service("Timing")
process.CPU = cms.Service("CPU", reportCPUProperties=cms.untracked.bool(True))
process.CondorStatusService = cms.Service("CondorStatusService", tag=cms.untracked.string("DIGI"), debug=cms.untracked.bool(True))

to a job in CMSSW_8_1_0_pre13 (that has this PR) I get a failure

----- Begin Fatal Exception 24-Oct-2016 12:55:33 EDT-----------------------
An exception of category 'Configuration' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing service of type CondorStatusService
Exception Message:
MissingParameter: The required parameter 'summaryOnly' was not specified.
----- End Fatal Exception -------------------------------------------------

which looks like the TimingService is getting constructed before fillDescriptions has been called...this does not happen if the TimingService isn't included in the config. Found by a CRAB user, see this HN and the upstream thread:

https://hypernews.cern.ch/HyperNews/CMS/get/computing-tools/2205/2/1.html

Since this happens while constructing CondorStatusService, I strongly suspect this PR is responsible.

dan131riley · 2016-10-24T17:39:34Z

Never mind, I had missed that this was fixed by #16311

bbockelm added 3 commits September 7, 2016 11:18

Report CPU model to HTCondor.

2f743c1

This commit has the framework forward the CPU model information to the HTCondor monitoring.

Include child processes in CPU timing.

f3c0605

With this commit, getCPU() additionally returns all usage information from child processes.

cmsbuild added this to the Next CMSSW_8_1_X milestone Sep 7, 2016

cmsbuild added comparison-pending core-pending orp-pending pending-signatures tests-pending labels Sep 7, 2016

cmsbuild added tests-started and removed tests-pending labels Sep 14, 2016

Dr15Jones suggested changes Sep 14, 2016

View reviewed changes

cmsbuild added tests-approved and removed tests-started labels Sep 14, 2016

cmsbuild added comparison-available and removed comparison-pending labels Sep 14, 2016

Cleanly separate cpuinfo parsing from statistics collection.

f452d44

Now, one function will parse the cpuinfo file - each different statistic we want will utilize this information in a separate method.

cmsbuild added comparison-pending tests-pending and removed comparison-available tests-approved labels Sep 29, 2016

Merge remote-tracking branch 'official-cmssw/CMSSW_8_1_X' into report…

e8d9da8

…_cpu_info Conflicts: FWCore/Services/plugins/CPU.cc

cmsbuild added tests-started and removed tests-pending labels Oct 6, 2016

cmsbuild added tests-approved and removed tests-started labels Oct 6, 2016

cmsbuild added comparison-available and removed comparison-pending labels Oct 6, 2016

cmsbuild added core-approved fully-signed and removed core-pending pending-signatures labels Oct 9, 2016

cmsbuild added orp-approved and removed orp-pending labels Oct 9, 2016

cmsbuild merged commit 5e56097 into cms-sw:CMSSW_8_1_X Oct 9, 2016

smuzaffar mentioned this pull request May 28, 2020

Automatically account for CPU usage of child processes #15764

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Report CPU info to HTCondor monitoring #15767

Report CPU info to HTCondor monitoring #15767

bbockelm commented Sep 7, 2016

cmsbuild commented Sep 7, 2016

davidlange6 commented Sep 14, 2016

cmsbuild commented Sep 14, 2016 •

edited

Dr15Jones Sep 14, 2016

bbockelm Sep 29, 2016

Dr15Jones Sep 14, 2016

bbockelm Sep 29, 2016

cmsbuild commented Sep 14, 2016

cmsbuild commented Sep 14, 2016

cmsbuild commented Sep 29, 2016

cmsbuild commented Sep 29, 2016

Dr15Jones commented Oct 6, 2016

cmsbuild commented Oct 6, 2016 •

edited

cmsbuild commented Oct 6, 2016

cmsbuild commented Oct 6, 2016

Dr15Jones commented Oct 6, 2016

bbockelm commented Oct 9, 2016

Dr15Jones commented Oct 9, 2016

cmsbuild commented Oct 9, 2016

davidlange6 commented Oct 9, 2016

dan131riley commented Oct 24, 2016

dan131riley commented Oct 24, 2016

Report CPU info to HTCondor monitoring #15767

Report CPU info to HTCondor monitoring #15767

Conversation

bbockelm commented Sep 7, 2016

cmsbuild commented Sep 7, 2016

davidlange6 commented Sep 14, 2016

cmsbuild commented Sep 14, 2016 • edited

Dr15Jones Sep 14, 2016

Choose a reason for hiding this comment

bbockelm Sep 29, 2016

Choose a reason for hiding this comment

Dr15Jones Sep 14, 2016

Choose a reason for hiding this comment

bbockelm Sep 29, 2016

Choose a reason for hiding this comment

cmsbuild commented Sep 14, 2016

cmsbuild commented Sep 14, 2016

cmsbuild commented Sep 29, 2016

cmsbuild commented Sep 29, 2016

Dr15Jones commented Oct 6, 2016

cmsbuild commented Oct 6, 2016 • edited

cmsbuild commented Oct 6, 2016

cmsbuild commented Oct 6, 2016

Dr15Jones commented Oct 6, 2016

bbockelm commented Oct 9, 2016

Dr15Jones commented Oct 9, 2016

cmsbuild commented Oct 9, 2016

davidlange6 commented Oct 9, 2016

dan131riley commented Oct 24, 2016

dan131riley commented Oct 24, 2016

cmsbuild commented Sep 14, 2016 •

edited

cmsbuild commented Oct 6, 2016 •

edited