Integrate CMSSW with HTCondor's update service. #10056

bbockelm · 2015-07-06T20:05:44Z

This commit provides a new default service, CondorStatusService,
which automatically reports basic progress statistics (# events,

The service will automatically detect if it's running as part of
a HTCondor job and probes to see if HTCondor has user-level updates
enabled (and is a sufficient version). If either check fails, then
the service does not register any callbacks with the framework and
effectively becomes a no-op.

To see if it is within a HTCondor job, the service looks for the
_CONDOR_CHIRP_CONFIG environment variable - a very cheap check.
To see if HTCondor supports this feature, it spawns a new process
and looks at the exit code (more expensive).

This uses the 'set_job_attr_delayed' mechanism of HTCondor,
which causes these updates to 'tag along' with the existing updates
for memory / disk / CPU usage. Hence, the extra cost is the few
extra bytes in the update packet that goes out once every 5 minutes.
The update only goes as far as the local daemon; condor_chirp does
not wait for it to propagate to the remote host. Hence, it exits
rapidly.

CMSSW will only update once every updateIntervalSeconds (defaults
to 15 minutes). The HTCondor worker node and central components
all have additional rate limiting mechanisms to prevent overload.

This commit provides a new default service, CondorStatusService, which automatically reports basic progress statistics (# events, The service will automatically detect if it's running as part of a HTCondor job and probes to see if HTCondor has user-level updates enabled (and is a sufficient version). If either check fails, then the service does not register any callbacks with the framework and effectively becomes a no-op. To see if it is within a HTCondor job, the service looks for the _CONDOR_CHIRP_CONFIG environment variable - a very cheap check. To see if HTCondor supports this feature, it spawns a new process and looks at the exit code (more expensive). This uses the 'set_job_attr_delayed' mechanism of HTCondor, which causes these updates to 'tag along' with the existing updates for memory / disk / CPU usage. Hence, the extra cost is the few extra bytes in the update packet that goes out once every 5 minutes. The update only goes as far as the local daemon; condor_chirp does not wait for it to propagate to the remote host. Hence, it exits rapidly. CMSSW will only update once every updateIntervalSeconds (defaults to 15 minutes). The HTCondor worker node and central components all have additional rate limiting mechanisms to prevent overload.

cmsbuild · 2015-07-06T20:11:26Z

A new Pull Request was created by @bbockelm (Brian Bockelman) for CMSSW_7_6_X.

Integrate CMSSW with HTCondor's update service.

It involves the following packages:

FWCore/Framework
FWCore/Services

@cmsbuild, @smuzaffar, @Dr15Jones can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @wddgit, @wmtan this is something you requested to watch as well.
You can sign-off by replying to this message having '+1' in the first line of your reply.
You can reject by replying to this message having '-1' in the first line of your reply.
If you are a L2 or a release manager you can ask for tests by saying 'please test' in the first line of a comment.
@Degano you are the release manager for this.
You can merge this pull request by typing 'merge' in the first line of your comment.

This commit adds a few more reported attributes: - ChirpCMSSWFinished: Unix timestamp of when the job has finished. - ChirpCMSSWLastUpdate: Unix timestamp of when the last update occurred. - ChirpCMSSWMaxEvents: Maximum number of input events CMSSW is configured to process. From the process's maxEvents pset. - ChirpCMSSWMaxLumis: Maximum number of lumis CMSSW is configured to process. From process's maxLuminosityBlocks pset and some simple processing of the source. If no max is configured, the attribute is not reported. The motivation behind these attributes is they: - Simplify deadlock detection. Using the defaults, the LastUpdate attribute should never be more than 30 minutes if Finished isn't set. - Provide simple estimates of percent completion. When we can determine the number of events or lumis to process (something we can for the majority of the grid use cases), we'll be able to determine the number of events/lumis left to process and an aggregate event/lumi processing rate.

bbockelm · 2015-07-07T03:46:11Z

I added a few new attributes:

ChirpCMSSWFinished = 1436240094
ChirpCMSSWLastUpdate = 1436240094
ChirpCMSSWMaxEvents = 5
ChirpCMSSWMaxLumis = 5

These allow us to:

Better detect deadlocks. LastUpdate and Finished will allow us to detect whether CMSSW is running; if so, how long it has been since an event was completed.
Provide rough estimates of percent completion. The logic to detect MaxEvents / MaxLumis from the pset is far from perfect, but ought to work for 99% of the grid jobs out there (the core use case).

cmsbuild · 2015-07-07T03:51:22Z

Pull request #10056 was updated. @cmsbuild, @smuzaffar, @Dr15Jones can you please check and sign again.

Dr15Jones · 2015-07-07T04:41:40Z

please test

cmsbuild · 2015-07-07T04:51:21Z

The tests are being triggered in jenkins.

Dr15Jones · 2015-07-07T05:17:48Z

FWCore/Framework/bin/cmsRun.cpp

@@ -332,6 +332,7 @@ int main(int argc, char* argv[]) {
      defaultServices.push_back("AdaptorConfig");
      defaultServices.push_back("SiteLocalConfigService");
      defaultServices.push_back("StatisticsSenderService");
+      defaultServices.push_back("CondorStatusService");


Why make this a default service? Why not require WMAgent to add it to the configurations since this is only needed on grid jobs?

Dr15Jones · 2015-07-07T05:24:34Z

Why not have CondorStatusUpdater.cc also hold the content of CondorStatusUpdater.h and move it to plugins and then register the service directly in CondorStatusUpdater.cc? As far as I can tell there is no need for anyone to attempt to talk directly to CondorStatusUpdater.

cmsbuild · 2015-07-07T06:05:51Z

+1
Tested at: ddf7626
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-10056/5957/summary.html

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
b8c2bd2
16bc21a
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-10056/5957/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-10056/5957/git-merge-result

cmsbuild · 2015-07-21T14:22:05Z

This pull request is fully signed and it will be integrated in one of the next CMSSW_7_6_X IBs or unless it breaks tests. This pull request requires discussion in the ORP meeting before it's merged. @davidlange6, @Degano, @smuzaffar

cmsbuild · 2015-07-21T15:16:01Z

-1
Tested at: acd0eea
When I ran the RelVals I found an error in the following worklfows:
50202.0 step2

runTheMatrix-results/50202.0_TTbar_13+TTbar_13+DIGIUP15_PU50+RECOUP15_PU50+HARVESTUP15_PU50+MINIAODMCUP1550/step2_TTbar_13+TTbar_13+DIGIUP15_PU50+RECOUP15_PU50+HARVESTUP15_PU50+MINIAODMCUP1550.log
----- Begin Fatal Exception 21-Jul-2015 16:57:31 CEST-----------------------
An exception of category 'Configuration' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=MixingModule label='mix'
Exception Message:
RootEmbeddedFileSequence no input files specified for secondary input source.
----- End Fatal Exception -------------------------------------------------

25202.0 step3

runTheMatrix-results/25202.0_TTbar_13+TTbar_13+DIGIUP15_PU25+RECOUP15_PU25+HARVESTUP15_PU25+MINIAODMCUP15/step3_TTbar_13+TTbar_13+DIGIUP15_PU25+RECOUP15_PU25+HARVESTUP15_PU25+MINIAODMCUP15.log
----- Begin Fatal Exception 21-Jul-2015 17:15:11 CEST-----------------------
An exception of category 'Configuration' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=MixingModule label='mix'
Exception Message:
RootEmbeddedFileSequence no input files specified for secondary input source.
----- End Fatal Exception -------------------------------------------------

you can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-10056/6402/summary.html

cmsbuild · 2015-07-21T15:21:59Z

This pull request is fully signed and it will be integrated in one of the next CMSSW_7_6_X IBs (but tests are reportedly failing). This pull request requires discussion in the ORP meeting before it's merged. @davidlange6, @Degano, @smuzaffar

Integrate CMSSW with HTCondor's update service.

bbockelm · 2015-07-21T18:31:00Z

@Dr15Jones - how do you suggest I do the backport without squashing? Does CMS have a how-to page for that (I always squash...)

Dr15Jones · 2015-07-21T18:34:17Z

I've always just used git cherry-pick. I think the new version of git even allows you to specify multiple commits in one command.

bbockelm · 2015-07-21T18:35:04Z

Ah, ok. I thought there was a deeper secret than that.

cmsbuild added this to the Next CMSSW_7_6_X milestone Jul 6, 2015

cmsbuild added comparison-pending core-pending orp-pending pending-signatures tests-pending labels Jul 6, 2015

cmsbuild added comparison-pending core-pending orp-pending pending-signatures tests-started and removed comparison-pending core-pending orp-pending pending-signatures tests-pending labels Jul 7, 2015

Dr15Jones reviewed Jul 7, 2015
View reviewed changes

cmsbuild removed comparison-pending core-pending orp-pending labels Jul 7, 2015

cmsbuild removed comparison-pending core-approved fully-signed orp-pending tests-pending labels Jul 21, 2015

cmsbuild added comparison-pending core-approved fully-signed orp-pending tests-started labels Jul 21, 2015

cmsbuild added comparison-pending core-approved fully-signed orp-pending tests-rejected and removed comparison-pending core-approved fully-signed orp-pending tests-started labels Jul 21, 2015

davidlange6 added a commit that referenced this pull request Jul 21, 2015

Merge pull request #10056 from bbockelm/chirp_cmssw_updates

4e98e7c

Integrate CMSSW with HTCondor's update service.

davidlange6 merged commit 4e98e7c into cms-sw:CMSSW_7_6_X Jul 21, 2015

This was referenced Jul 28, 2015

Chirp-based HTCondor updates for 7_4_X #10419

Merged

Chirp-based HTCondor updates for 7_5_X #10420

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integrate CMSSW with HTCondor's update service. #10056

Integrate CMSSW with HTCondor's update service. #10056

bbockelm commented Jul 6, 2015

cmsbuild commented Jul 6, 2015

bbockelm commented Jul 7, 2015

cmsbuild commented Jul 7, 2015

Dr15Jones commented Jul 7, 2015

cmsbuild commented Jul 7, 2015

Dr15Jones Jul 7, 2015

Dr15Jones commented Jul 7, 2015

cmsbuild commented Jul 7, 2015

cmsbuild commented Jul 21, 2015

cmsbuild commented Jul 21, 2015

cmsbuild commented Jul 21, 2015

bbockelm commented Jul 21, 2015

Dr15Jones commented Jul 21, 2015

bbockelm commented Jul 21, 2015

Integrate CMSSW with HTCondor's update service. #10056

Integrate CMSSW with HTCondor's update service. #10056

Conversation

bbockelm commented Jul 6, 2015

cmsbuild commented Jul 6, 2015

bbockelm commented Jul 7, 2015

cmsbuild commented Jul 7, 2015

Dr15Jones commented Jul 7, 2015

cmsbuild commented Jul 7, 2015

Dr15Jones Jul 7, 2015

Choose a reason for hiding this comment

Dr15Jones commented Jul 7, 2015

cmsbuild commented Jul 7, 2015

cmsbuild commented Jul 21, 2015

cmsbuild commented Jul 21, 2015

cmsbuild commented Jul 21, 2015

bbockelm commented Jul 21, 2015

Dr15Jones commented Jul 21, 2015

bbockelm commented Jul 21, 2015