Machine and job features

The pilot supports the HEPiX Machine/Job Features standard, which provides a site-agnostic mechanism for batch systems to expose resource and scheduling information to running jobs. The pilot reads these values at runtime from two directories pointed to by environment variables.

How the pilot reads features

The implementation is in pilot/util/features.py. On construction, MachineFeatures and JobFeatures each read from a directory whose path comes from an environment variable:

Class	Environment variable
MachineFeatures	$MACHINEFEATURES
JobFeatures	$JOBFEATURES

Each attribute is read from a separate file within that directory, named after the attribute itself (e.g. $MACHINEFEATURES/shutdowntime). Files that are absent are silently skipped and the attribute remains an empty string. If the environment variable is unset or the directory does not exist, all attributes remain empty.

Sites that conform to the HEPiX standard will typically set these environment variables and populate the directories automatically. On sites that do not, the files can be written manually — see below.

Machine features ($MACHINEFEATURES)

The following attributes are read from the $MACHINEFEATURES directory:

File	Type	Description
hs06	float	HS06 benchmark score for the whole machine
total_cpu	int	Total number of CPUs on the machine
shutdowntime	int (Unix epoch)	Time at which the machine will be shut down
grace_secs	int	Grace period in seconds before shutdown (defined but not currently read by the pilot — see note below)

hs06 and total_cpu

These two values are used together in job metrics reporting. When both are present, the pilot scales the machine-level hs06 score to a per-job estimate based on the job's corecount:

job_hs06 = hs06 * corecount / total_cpu

The result is included in the job metrics string sent back to the server (e.g. hs06=4.25).

shutdowntime

Used by the monitor to trigger a graceful pilot abort before the machine is taken down. See Shutdown time below for full details.

grace_secs

This field is defined in the MachineFeatures class and will be populated if the file exists, but the pilot does not currently read it at runtime. The grace period before a shutdown abort is hardcoded at 10 minutes (see Shutdown time).

Job features ($JOBFEATURES)

The following attributes are read from the $JOBFEATURES directory:

File	Type	Description
allocated_cpu	int	Number of CPUs allocated to this job
hs06_job	float	HS06 benchmark score for this job's allocation
shutdowntime_job	int (Unix epoch)	Time at which this job slot will be shut down
grace_secs_job	int	Grace period in seconds before job shutdown
jobstart_secs	int (Unix epoch)	Time at which the job started
job_id	str	Batch system job identifier
wall_limit_secs	int	Wall time limit in seconds
cpu_limit_secs	int	CPU time limit in seconds
max_rss_bytes	int	RSS memory limit in bytes
max_swap_bytes	int	Swap memory limit in bytes
scratch_limit_bytes	int	Scratch disk limit in bytes

Any non-empty job feature values are appended to the job metrics string reported to the server at the end of the job.

Shutdown time

The monitor thread checks $MACHINEFEATURES/shutdowntime once per minute. If the current time is within 10 minutes of the shutdown time, the pilot logs a fatal message and initiates a graceful abort, allowing the current job to be re-queued before the machine is taken down.

shutdowntime minus grace time has been exceeded - time to abort pilot

The check can be disabled per queue via the is_pilot_check mechanism (check='machinefeatures').

Value format

The value must be a Unix epoch timestamp (integer seconds since 1970-01-01 00:00:00 UTC). Human-readable date strings will cause a conversion error and the check will be skipped. Convert to epoch before writing:

date -d "2026-05-18T18:00:00" +%s

Ignored conditions

The pilot will silently skip the shutdown check if:

• $MACHINEFEATURES is unset or the directory does not exist
• The shutdowntime file is absent or empty
• The timestamp predates the pilot's own start time (i.e. it was set before the pilot launched and was never updated)

The third condition exists to avoid false aborts caused by stale values from a previous job occupying the same directory. Ensure the file is written after the pilot process starts, or that its value is in the future at the time the pilot starts.

Grace period

The grace period is hardcoded at 10 minutes. The grace_secs file in $MACHINEFEATURES is not currently used. Plan your job walltime so that the pilot has at least 10 minutes of runway between the shutdown trigger and the hard kill by the batch system.

Configuring features on sites without native support

If your site does not set $MACHINEFEATURES or $JOBFEATURES automatically, you can populate them in your job prologue or wrapper script before launching the pilot. There are two approaches depending on whether the environment variables are already set by the site.

Option 1 — Create the directory yourself (no site support)

Create a dedicated directory, write the relevant feature files into it, and export the environment variable before launching the pilot:

export MACHINEFEATURES=/tmp/machine_features_${JOB_ID}
mkdir -p "$MACHINEFEATURES"

# Write individual feature files, e.g.:
echo "<unix_epoch_timestamp>" > "$MACHINEFEATURES/shutdowntime"
echo "16.0"                   > "$MACHINEFEATURES/hs06"
echo "64"                     > "$MACHINEFEATURES/total_cpu"

The exact method for obtaining values such as the job end time depends on your batch system. For example, with SLURM:

export MACHINEFEATURES=/tmp/machine_features_${SLURM_JOB_ID}
mkdir -p "$MACHINEFEATURES"

squeue -j "$SLURM_JOB_ID" -h --format=%e \
    | xargs -I{} date -d "{}" +%s \
    > "$MACHINEFEATURES/shutdowntime"

Option 2 — Write into an existing directory (partial site support)

If $MACHINEFEATURES is already set by the site but the specific files you need are not populated, write the missing files directly into the existing directory:

echo "<unix_epoch_timestamp>" > "$MACHINEFEATURES/shutdowntime"

Do not overwrite files the site has already populated unless you intend to replace their values.

Timestamp format

All time-valued feature files (shutdowntime, shutdowntime_job, jobstart_secs) must contain a Unix epoch timestamp — an integer number of seconds since 1970-01-01 00:00:00 UTC. Human-readable date strings (e.g. 2026-05-18T18:00:00) will cause a conversion error at runtime and the value will be silently ignored.

To convert a date string to an epoch timestamp:

date -d "2026-05-18T18:00:00" +%s
# or, to convert directly from a SLURM-formatted end time:
squeue -j "$SLURM_JOB_ID" -h --format=%e | xargs -I{} date -d "{}" +%s

Timing requirement

All feature files must be in place and the environment variables exported before the pilot process starts. The pilot reads the features directory on construction of MachineFeatures / JobFeatures — it does not watch for changes after that point (except for shutdowntime, which is re-read once per minute by the monitor thread).

Verifying the configuration

With a correct setup, the per-minute monitor log will show non-empty values:

MachineFeatures().get()={'hs06': '16.0', 'shutdowntime': '1747587600', 'total_cpu': '64', 'grace_secs': ''}

Empty strings indicate the corresponding file was not found. A fully empty dict (all fields '') means the directory was not found or $MACHINEFEATURES was not set.