Core count

How the Pilot Handles Core Counts

The pilot deals with three distinct "core count" values that are easy to conflate but serve different purposes:

1. queuedata.corecount — the core count for the PanDA queue, sourced from CRIC.
2. job.corecount — the core count for the individual job, sourced from the job definition sent by the PanDA server.
3. job.actualcorecount — the pilot's own runtime measurement of how many cores the payload actually used.

This document explains where each value comes from, how they interact, and where they end up being used.

1. queuedata.corecount (from CRIC)

QueueData (pilot/info/queuedata.py) models the settings CRIC publishes for a PanDA queue. The field is declared as:

corecount = 1  #

and validated by:

def clean__corecount(self, raw, value):
    return value if value else 1

So if CRIC does not publish a value (or publishes 0/empty), the pilot defaults the queue core count to 1. This value describes the queue's nominal core count — e.g. an MCORE queue might declare corecount=8, meaning jobs on that queue are expected to use 8 cores each.

queuedata.corecount is not used to launch the payload. Its role is as a reference/scaling value, used in a few places:

• Memory scaling for UCORE queues (pilot/user/atlas/memory.py, get_ucore_scale_factor): UCORE queues can run jobs with different core counts (SCORE, 4CORE, MCORE) side by side. The pilot computes a scale factor as job.corecount / queuedata.corecount and uses it to adjust memory limits proportionally to how many cores the individual job actually requested relative to the queue's nominal value.
• VHIMEM memory limit calculation (calculate_memory_limit_kb): for SCORE_VHIMEM jobs, the allowed RSS is (queuedata.maxrss / queuedata.corecount) * job.corecount — i.e. the per-queue max RSS (defined for the queue's nominal core count) is divided down to a per-core value and re-scaled to the job's actual core count.
• Work directory size limit (pilot/util/monitoring.py, get_max_allowed_work_dir_size): queuedata.maxwdir is defined assuming an MCORE job (i.e. queuedata.corecount cores). If a job on that queue is single-core (job.corecount == 1), the pilot divides the max work directory size by queuedata.corecount to get a fair single-core allowance.

In short: queuedata.corecount is the queue's reference core count, used to scale memory and disk limits so single-core and multi-core jobs on the same (UCORE) queue get proportionate resource allowances.

2. job.corecount (from the job definition)

JobData (pilot/info/jobdata.py) models the individual job sent by the PanDA server. The field:

corecount = 1   # Number of cores as requested by the task

is populated from the job definition's coreCount key via the key map:

kmap = {
    ...
    'corecount': 'coreCount',
    ...
}

and validated by:

def clean__corecount(self, raw, value):
    # Overwrite the corecount value with ATHENA_PROC_NUMBER if it is set
    athena_corecount = os.environ.get('ATHENA_PROC_NUMBER')
    if athena_corecount:
        try:
            value = int(athena_corecount)
        except Exception:
            logger.info(f"ATHENA_PROC_NUMBER is not properly set.. ignored, data={athena_corecount}")

    return value if value else 1

Two things happen here:

• Default to 1 if the job definition didn't set coreCount.
• If the environment variable ATHENA_PROC_NUMBER is already set at parse time (e.g. by a batch system or site wrapper script), it overrides the job-definition value.

job.corecount is the operative value used to actually run the payload. It is used, among other places, in pilot/user/atlas/common.py:

• verify_ncores(job.corecount) sets ATHENA_PROC_NUMBER_JOB and ATHENA_CORE_NUMBER from job.corecount, unless ATHENA_PROC_NUMBER is already set in the environment (in which case that wins and is never overwritten — it's treated as an authoritative external setting, e.g. from the batch system).
• add_makeflags(job.corecount, cmd) sets MAKEFLAGS=-j<corecount> for build-type payloads, so parallel builds use the right number of cores (important specifically when coreCount=1, to prevent the build from over-subscribing the node).
• For event-service jobs, ATHENA_PROC_NUMBER/ATHENA_CORE_NUMBER are exported directly from job.corecount (defaulting to 1 if unset).
• get_core_count() in pilot/user/atlas/cpu.py re-derives job.corecount, again always preferring a set ATHENA_PROC_NUMBER over the job-definition value, with a special case: on HPC queues (catchall contains HPC_HPC), an unset corecount becomes 0 rather than 1.
• The job definition also carries maxCpuCount, minRamCount, etc. mapped similarly, but these are separate quantities (CPU time and RAM, not core count).

job.corecount is also what gets reported back to PanDA in job updates/heartbeats (pilot/control/job.py):

if job.corecount and job.corecount != 'null' and job.corecount != 'NULL':
    data['core_count'] = job.corecount

Precedence summary for job.corecount

From highest to lowest priority:

1. ATHENA_PROC_NUMBER environment variable, if already set when the job data is parsed or when get_core_count() is called (treated as authoritative — typically set by the batch system/site).
2. The job definition's coreCount field (from the PanDA server).
3. Fallback default of 1 (or 0 on HPC queues if still unset).

3. job.actualcorecount (pilot-measured)

actualcorecount = 0   # number of cores actually used by the payload
corecounts = []        # keep track of all actual core count measurements

This is the pilot's own estimate of how many cores the payload is really using at runtime, independent of what was requested. It's computed periodically during payload monitoring (pilot/util/monitoring.py: set_number_used_cores() → experiment-specific cpu.set_core_counts()), and the method differs by experiment plugin:

• ATLAS (pilot/user/atlas/cpu.py): uses the memory monitor (prmon) output, computing actualcorecount = (stime + utime) / walltime — i.e. total CPU time (system + user) divided by elapsed wall time. This is a CPU-efficiency-based estimate of core-equivalents used, not a literal count of distinct CPUs. It's rounded to 2 decimal places and can be fractional (e.g. 3.87). ATLAS does not overwrite job.corecount with this value — it's tracked and reported separately.
• Generic / Rubin / DarkSide / SKA / EPIC / sPHENIX (pilot/user/<experiment>/cpu.py): uses ps axo pgid,psr to list the distinct CPU cores (psr) that processes in the payload's process group (job.pgrp) have run on, and counts the unique cores. This is a literal (integer) count of distinct cores touched. These plugins additionally overwrite job.corecount with the measured value and append it to job.corecounts.

Each measurement is appended to job.corecounts (a running list) via add_core_count(). When the pilot sends job updates, it computes the mean of that list:

if job.corecounts:
    _mean = mean(job.corecounts)
    data['mean_core_count'] = int(_mean)

job.actualcorecount is also surfaced in ATLAS job metrics (pilot/user/atlas/jobmetrics.py):

if job.actualcorecount:
    job_metrics += get_job_metrics_entry("actualCoreCount", job.actualcorecount)

(reported alongside coreCount=<job.corecount> in the jobMetrics string built elsewhere in pilot/user/atlas/common.py.)

Putting it together

Value	Source	Set when	Purpose
queuedata.corecount	CRIC (queuedata)	Pilot startup, when queuedata is loaded	Reference core count for the queue; used to scale memory and work-directory limits for jobs with a different core count than the queue's nominal value (UCORE queues)
job.corecount	Job definition (coreCount), possibly overridden by ATHENA_PROC_NUMBER env var	Job data parsing, and re-checked in get_core_count()/verify_ncores()	The core count the payload is actually launched with (ATHENA_PROC_NUMBER/ATHENA_CORE_NUMBER, MAKEFLAGS); reported to PanDA as core_count
job.actualcorecount	Pilot's own runtime measurement (prmon CPU-time ratio for ATLAS, or distinct-psr count via ps for other experiments)	Periodically during payload monitoring	Sanity-check / accounting of actual core usage vs. requested; reported as actualCoreCount job metric and mean_core_count in job updates; for non-ATLAS plugins, also fed back into job.corecount

The key distinction to keep in mind when explaining this to others: queuedata.corecount answers "what's this queue normally sized for?", job.corecount answers "how many cores should/does this job run with?", and job.actualcorecount answers "how many cores did the pilot observe the payload actually using?" The first is a scaling reference from CRIC, the second drives execution and is echoed back to PanDA, and the third is an independent, empirical check computed from process/CPU-time data during the run.

---

Based on a review of the pilot3 source: pilot/info/queuedata.py, pilot/info/jobdata.py, pilot/user/atlas/cpu.py, pilot/user/atlas/memory.py, pilot/user/atlas/common.py, pilot/user/atlas/jobmetrics.py, pilot/user/generic/cpu.py (and equivalents for other experiment plugins), pilot/util/monitoring.py, pilot/control/job.py, pilot/resource/jobdescription.py.