Add env variable to limit number of threads used by ROOT

[EdwardMoyse]

Hi,

ATLAS would like to request the addition of a mechanism to control the number of threads used by ROOT with implicit multi-threading enabled, ideally via an environment variable (e.g. ROOT_MAX_THREADS). The primary motivation for this is to give GRID sites an easier way to limit the number of threads to match that of a queue.

Thanks!

Ed

[Axel-Naumann]

Removed "experiment" label to cross-check that Ed can add it himself :-)

[Axel-Naumann]

Should this not be some generic grid variable that we honor? Does that exist, or if not, should it be defined? What does the GDB say - could you talk to them?

[elmsheus]

This idea is to have an optional ROOT variable we can set in the PanDA pilot or the PanDA worker node wrapper environment that allows to limit the number of threads spawned on the worker node by ROOT. We did this in the past e.g. with ProofLite.Workers to prevent prooflite processes to grab all available CPU cores on a worker node instead of the actual usual 8 cores on a WLCG multi-core slot.

We steer AthenaMP and MT jobs via the environment variables ATHENA_PROC_NUMBER and ATHENA_CORE_NUMBER and set them to the PanDA queue core count.

Asking the GDB might be a bit too high level committee, but if needed this could be brought up in some WLCG operations meetings if others would be interested to steer ROOT in a similar way.

[lukasheinrich]

Hi, hard coding anything related to Athena into ROOT seems weird to me. Maybe the wrapper should take some generic value and then set application specific variables like ROOT_MAX_THREADS?

[EdwardMoyse]

Yes, that’s what we’re proposing.

[lukasheinrich]

Ok - maybe I misunderstood @Axel-Naumann's "generic grid variable that we honor" .. sorry!

[Axel-Naumann]

Asking the GDB might be a bit too high level committee, but if needed this could be brought up in some WLCG operations meetings if others would be interested to steer ROOT in a similar way.

What's wrong with grid sites agreeing on an env var (whatever the mechanism) that sets the max threads to be started by any process for any grid job? Why would we want to only steer ROOT, only through PanDA - what happens if there's an analysis job that an ATLAS physicist starts outside ATHENA, outside PanDA? What if it's a CMS physicist? How isn't this a generic issue to be addressed, outside ATLAS or ROOT?

Maybe such an env var already exists - I sort of remember that the grid sites agreed to define one, actually... Do you want to drive this (my preference!) or do you prefer that I talk to WLCG folks?

[EdwardMoyse]

Hi @Axel-Naumann - we discussed this again internally.

From our side, we're happy with whatever suits you best (it's not really our place to decide what the variable is called or how this is implemented).

However it is at least conceivable that we might want to limit ROOT to less threads than provided by the queue, in which scenario we would need ROOT_MAX_THREADS (or whatever) anyway.

[zlmarshall]

Just a reminder of this ticket. It'd be nice to give users a way to handle this with an environment variable. We can (and often do) set this for virtually all other programs we run on the grid individually.

[jblomer]

Just for the record: ROOT currently already respects the cgroups #CPU limits (see ROOT::Internal::LogicalCPUBandwithControl()).

[Axel-Naumann]

We completely agree and the above PR implements what you propose. Apologies for the long wait for such a simple thing!

[github-actions]

Hi @Axel-Naumann,

It appears this issue is closed, but wasn't yet added to a project. Please add upcoming versions that will include the fix, or 'not applicable' otherwise.

Sincerely,
🤖