Improve resource requirements for utilitarian jobs

[amaltaro]

For cleanup, logcollect and merge jobs.
By default, they use 1 core, request 1GB of RAM and have a MaxRSS watchdog set to ~2.3GB.

We should check ES data and maybe lower these requirements for a better usage of the resources.

[amaltaro]

Might not be a very good idea... I've just found a merge job for the TaskChain_Relval_Multicore template that had a performance failure:

    PerformanceError
        PerformanceKill (Exit Code: 50660)

            Error in CMSSW step cmsRun1
            Number of Cores: None
            Job has exceeded maxRSS: 2355.2
            Job has RSS: 2425

[hufnagel]

Weird. Merge jobs should all use fast-copy of baskets which is fast and should use little memory. Might be worthwhile to get a log of that job and figure out what went wrong...

[amaltaro]

Are you volunteering yourself to look at it? :)

[vlimant]

#8451 maybe a duplicate

[amaltaro]

You mean the other way around :)

[vlimant]

from #8451 make sure you update also what goes in htcondor when you rework this

[thongonary]

So... how straightforward it is to increase the threshold to some higher value, say, 4GB?

[hufnagel]

You don't want to do this for all such jobs. Requesting 4GB for standard merge, cleanup, logcollect etc jobs means you have less resources that can run them (you wait longer to run them and can run less of them) and you leave less resources available for other jobs.

If special types of utility jobs (i.e. NANOAOD merges that aren't really standard merges) need more memory, we should request more memory just for these special types of jobs.

Cleanup and LogCollect could probably be reduced though.

[thongonary]

If special types of utility jobs (i.e. NANOAOD merges that aren't really standard merges) need more memory, we should request more memory just for these special types of jobs.

Thanks! That's what we want.

[amaltaro]

For the record, several of the Task getters/setters methods don't touch "utilitarian" jobs. Right now we cannot change resource requirements for such jobs and if we want to support updates to those tasks too, that's going to be tricky and likely ugly for the assigner/unified side (the only way I see memory updates working without causing issues in other tasks would be specificifying every single tasks and its Memory requirement).

[bbockelm]

Hi,

Note that the NanoAOD merge issues are really a ROOT bug -- and affect how well these files can be effectively read by users. See:

cms-sw/cmssw#22445

For the other merge jobs - are we really seeing memory limits, or are we simply snapshotting cmsRun when it forks? The watchdog should be using PSS, not RSS, in the end.

Brian

[amaltaro]

I suggest we first update the watchdog to PSS instead of RSS. Then we collect data for a couple of months and set those utilitarian jobs with a reasonable resources requirement in order to minimize the resource wastage.