Job metrics

Job metrics are a space-separated string of key=value pairs reported to the PanDA server during updateJob calls throughout a job's lifetime. They are assembled by each experiment plugin's get_job_metrics() function, which is dispatched from pilot/util/jobmetrics.py.

The string is capped at 500 characters. If it exceeds this limit the pilot truncates it and drops the last (potentially partial) entry with a warning.

Format

key1=value1 key2=value2 key3=value3 ...

Values must not contain spaces. Where a value is derived from an OS string or similar free-form source, any spaces are replaced with underscores before inclusion.

---

Plugin support summary

Plugin	Reports job metrics	Notes
ATLAS	Yes	Most complete; described in full below
ePIC	Yes	Near-identical to ATLAS; differences noted below
Rubin	Yes	Minimal; only cpuFrequency and extra fields
DarkSide	No	Stub returns empty string
sPHENIX	No	Raises NotImplementedError
SKA	No	Raises NotImplementedError
Generic	No	Raises NotImplementedError

---

ATLAS fields

These fields are reported by pilot/user/atlas/jobmetrics.py. All fields are conditional — they are omitted from the string if the underlying value is zero, empty, or unavailable.

Event counts

Key	Type	Description
nEvents	int	Number of events read (processed) by the payload. Set from the job report or Athena summary files.
nEventsW	int	Number of events written by the payload. May differ from nEvents for filter or merge jobs.
eventnumber	int	Most recent event number from eventLoopHeartBeat.txt, updated at each heartbeat interval.
resimevents	int	Number of re-simulation events, extracted from the job report metadata. ATLAS-specific.

CPU and core usage

Key	Type	Description
actualCoreCount	int	Number of CPU cores actually used by the payload, as measured during execution.

Disk usage

Key	Type	Description
workDirSize	int	Maximum size of the job work directory in bytes, measured at job completion (finished/failed/holding).

Performance scaling

Key	Type	Description
hs06	float	HS06 benchmark score for the worker node, corrected for the number of cores allocated to this job: hs06_total × corecount / total_cpu. Sourced from machine features.

Database usage (ATLAS production jobs)

These fields are populated from the job report and are specific to ATLAS production transforms that interact with conditions or geometry databases.

Key	Type	Description
dbTime	float	Total time spent on database queries, in seconds.
dbData	float	Total data volume read from databases, in bytes.

Memory analytics

These fields are derived by fitting a linear model to the memory monitor time series (memory_monitor_output.txt). They are reported during job execution as well as at completion.

Key	Type	Description
leak	float	Slope of the linear fit to RSS memory usage over time (bytes/s). A positive value indicates a leak.
chi2	float	Chi-squared of the linear fit. Indicates how well the linear model describes the memory growth.
intersect	float	Y-intercept of the linear fit (initial RSS in bytes).

Rucio tracing

Key	Type	Description
rucioTraceError	string	curl exit code from a failed Rucio trace submission. Read from the RUCIO_TRACE_ERROR environment variable or from a file written by the middleware container. Omitted when the value is 0.

Alternative stage-out

Key	Type	Description
altTransferred	string	Comma-separated list of LFNs that were successfully staged out to an alternative destination (e.g. a backup endpoint). Only present when at least one output file used alternative stage-out.

DASK distributed computing

These fields are only present for DASK-based jobs.

Key	Type	Description
schedulerIP	string	IP address of the DASK scheduler.
sessionIP	string	IP address of the Jupyter notebook session.

OS and Python environment (pending server-side support)

These fields are implemented in the pilot but the reporting call is currently out-commented, pending confirmation that the PanDA server can receive and store them. Once enabled they will allow Grafana dashboards to track OS and Python version distributions across grid sites. Spaces in values are replaced with underscores.

Key	Type	Description
osId	string	OS identifier read from /etc/os-release (ID field), e.g. almalinux, centos, ubuntu. Falls back to platform.system().lower() if the file is absent.
osVersion	string	OS version read from /etc/os-release (VERSION_ID field), e.g. 9.4, 7, 24.04. Falls back to platform.release().
pythonVersion	string	Python interpreter version as major.minor.micro, e.g. 3.11.9.

Extra fields

Any key/value pairs passed in the extra dict argument to get_job_metrics() are appended verbatim. This mechanism is used by callers to inject one-off or job-type-specific metrics without modifying the jobmetrics module.

---

ePIC fields

pilot/user/epic/jobmetrics.py is structurally identical to the ATLAS module. It reports the following subset:

Key	Reported	Notes vs ATLAS
actualCoreCount	Yes	Same
nEvents	Yes	Same
nEventsW	Yes	Same
workDirSize	Yes	Same
rucioTraceError	Yes	Same
hs06	Yes	Same (corrected for corecount)
leak	Yes	Same
chi2	Yes	Same
intersect	Yes	Same
eventnumber	Yes	Same
schedulerIP	Yes	Same
sessionIP	Yes	Same
altTransferred	Yes	Same
dbTime	No	Not reported (no job report DB parsing)
dbData	No	Not reported (no job report DB parsing)
resimevents	No	Not reported
cpuFrequency	No	Collected but out-commented

---

Rubin fields

pilot/user/rubin/jobmetrics.py is a minimal implementation.

Key	Type	Description
cpuFrequency	int	Mean CPU frequency in MHz across all samples collected during the job.

Extra fields passed via the extra argument are also appended.

---

Plugins that do not report job metrics

The following plugins have stub implementations and report no metrics.

Plugin	Behaviour
DarkSide	Returns empty string silently
sPHENIX	Raises NotImplementedError
SKA	Raises NotImplementedError
Generic	Raises NotImplementedError

pilot/util/jobmetrics.py catches AttributeError and NotImplementedError from the plugin dispatch and returns None in those cases, so a missing or unimplemented plugin does not cause the pilot to crash.

---

Source files

File	Role
pilot/util/jobmetrics.py	Entry point; dispatches to the active experiment plugin
pilot/user/atlas/jobmetrics.py	ATLAS implementation (most complete)
pilot/user/epic/jobmetrics.py	ePIC implementation (near-identical to ATLAS)
pilot/user/rubin/jobmetrics.py	Rubin implementation (minimal)
pilot/user/darkside/jobmetrics.py	DarkSide stub (returns empty string)
pilot/user/sphenix/jobmetrics.py	sPHENIX stub (not implemented)
pilot/user/ska/jobmetrics.py	SKA stub (not implemented)
pilot/user/generic/jobmetrics.py	Generic stub (not implemented)