Skip to content

Latest commit

 

History

History
1480 lines (1071 loc) · 66.3 KB

system-metrics.md

File metadata and controls

1480 lines (1071 loc) · 66.3 KB
Raw

Semantic Conventions for System Metrics

Status: Experimental

This document describes instruments and attributes for common system level metrics in OpenTelemetry. Consider the general metric semantic conventions when creating instruments not explicitly defined in the specification.

The system.* namespace SHOULD be exclusively used to report hosts' metrics. The system.* namespace SHOULD only be used when the metrics are collected from within the target system. (physical servers, virtual machines etc). Metrics collected from technology-specific, well-defined APIs (e.g. Kubelet's API or container runtimes) should be reported under their respective namespace (e.g. k8s., container.). Resource attributes related to a host, SHOULD be reported under the host.* namespace.

Warning Existing instrumentations and collector that are using v1.21.0 of this document (or prior):

  • SHOULD NOT adopt any breaking changes from document until the system semantic conventions are marked stable. Conventions include, but are not limited to, attributes, metric names, and unit of measure.
  • SHOULD introduce a control mechanism to allow users to opt-in to the new conventions once the migration plan is finalized.

Processor Metrics

Description: System level processor metrics captured under the namespace system.cpu.

Metric: system.cpu.time

This metric is recommended.

Name Instrument Type Unit (UCUM) Description Stability
system.cpu.time Counter s Seconds each logical CPU spent on each mode Experimental
Attribute Type Description Examples Requirement Level Stability
cpu.mode string The CPU mode for this data point. A system's CPU SHOULD be characterized either by data points with no mode labels, or only data points with mode labels. [1] user; system Recommended Experimental
system.cpu.logical_number int The logical CPU number [0..n-1] 1 Recommended Experimental

[1]: Following states SHOULD be used: user, system, nice, idle, iowait, interrupt, steal

cpu.mode has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value Description Stability
idle idle Experimental
interrupt interrupt Experimental
iowait iowait Experimental
kernel kernel Experimental
nice nice Experimental
steal steal Experimental
system system Experimental
user user Experimental

Metric: system.cpu.utilization

This metric is recommended.

Name Instrument Type Unit (UCUM) Description Stability
system.cpu.utilization Gauge 1 Difference in system.cpu.time since the last measurement, divided by the elapsed time and number of logical CPUs Experimental
Attribute Type Description Examples Requirement Level Stability
cpu.mode string The CPU mode for this data point. A system's CPU SHOULD be characterized either by data points with no mode labels, or only data points with mode labels. [1] user; system Recommended Experimental
system.cpu.logical_number int The logical CPU number [0..n-1] 1 Recommended Experimental

[1]: Following modes SHOULD be used: user, system, nice, idle, iowait, interrupt, steal

cpu.mode has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value Description Stability
idle idle Experimental
interrupt interrupt Experimental
iowait iowait Experimental
kernel kernel Experimental
nice nice Experimental
steal steal Experimental
system system Experimental
user user Experimental

Metric: system.cpu.physical.count

This metric is recommended.

Name Instrument Type Unit (UCUM) Description Stability
system.cpu.physical.count UpDownCounter {cpu} Reports the number of actual physical processor cores on the hardware Experimental

Metric: system.cpu.logical.count

This metric is recommended.

Name Instrument Type Unit (UCUM) Description Stability
system.cpu.logical.count UpDownCounter {cpu} Reports the number of logical (virtual) processor cores created by the operating system to manage multitasking Experimental

Metric: system.cpu.frequency

This metric is recommended.

Name Instrument Type Unit (UCUM) Description Stability
system.cpu.frequency Gauge {Hz} Reports the current frequency of the CPU in Hz Experimental
Attribute Type Description Examples Requirement Level Stability
system.cpu.logical_number int The logical CPU number [0..n-1] 1 Recommended Experimental

Memory Metrics

Description: System level memory metrics capture under the namespace system.memory. This does not include paging/swap memory.

Metric: system.memory.usage

This metric is recommended.

Name Instrument Type Unit (UCUM) Description Stability
system.memory.usage UpDownCounter By Reports memory in use by state. [1] Experimental

[1]: The sum over all system.memory.state values SHOULD equal the total memory available on the system, that is system.memory.limit.

Attribute Type Description Examples Requirement Level Stability
system.memory.state string The memory state free; cached Recommended Experimental

system.memory.state has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value Description Stability
buffers buffers Experimental
cached cached Experimental
free free Experimental
used used Experimental

Metric: system.memory.limit

This metric is opt-in.

Name Instrument Type Unit (UCUM) Description Stability
system.memory.limit UpDownCounter By Total memory available in the system. [1] Experimental

[1]: Its value SHOULD equal the sum of system.memory.state over all states.

Metric: system.memory.shared

This metric is opt-in.

Name Instrument Type Unit (UCUM) Description Stability
system.memory.shared UpDownCounter By Shared memory used (mostly by tmpfs). [1] Experimental

[1]: Equivalent of shared from free command or Shmem from /proc/meminfo"

Metric: system.memory.utilization

This metric is recommended.

Name Instrument Type Unit (UCUM) Description Stability
system.memory.utilization Gauge 1 Experimental
Attribute Type Description Examples Requirement Level Stability
system.memory.state string The memory state free; cached Recommended Experimental

system.memory.state has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value Description Stability
buffers buffers Experimental
cached cached Experimental
free free Experimental
used used Experimental

Paging/Swap Metrics

Description: System level paging/swap memory metrics captured under the namespace system.paging.

Metric: system.paging.usage

This metric is recommended.

Name Instrument Type Unit (UCUM) Description Stability
system.paging.usage UpDownCounter By Unix swap or windows pagefile usage Experimental
Attribute Type Description Examples Requirement Level Stability
system.paging.state string The memory paging state free Recommended Experimental

system.paging.state has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value Description Stability
free free Experimental
used used Experimental

Metric: system.paging.utilization

This metric is recommended.

Name Instrument Type Unit (UCUM) Description Stability
system.paging.utilization Gauge 1 Experimental
Attribute Type Description Examples Requirement Level Stability
system.paging.state string The memory paging state free Recommended Experimental

system.paging.state has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value Description Stability
free free Experimental
used used Experimental

Metric: system.paging.faults

This metric is recommended.

Name Instrument Type Unit (UCUM) Description Stability
system.paging.faults Counter {fault} Experimental
Attribute Type Description Examples Requirement Level Stability
system.paging.type string The memory paging type minor Recommended Experimental

system.paging.type has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value Description Stability
major major Experimental
minor minor Experimental

Metric: system.paging.operations

This metric is recommended.

Name Instrument Type Unit (UCUM) Description Stability
system.paging.operations Counter {operation} Experimental
Attribute Type Description Examples Requirement Level Stability
system.paging.direction string The paging access direction in Recommended Experimental
system.paging.type string The memory paging type minor Recommended Experimental

system.paging.direction has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value Description Stability
in in Experimental
out out Experimental

system.paging.type has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value Description Stability
major major Experimental
minor minor Experimental

Disk Controller Metrics

Description: System level disk performance metrics captured under the namespace system.disk.

Metric: system.disk.io

This metric is recommended.

Name Instrument Type Unit (UCUM) Description Stability
system.disk.io Counter By Experimental
Attribute Type Description Examples Requirement Level Stability
disk.io.direction string The disk IO operation direction. read Recommended Experimental
system.device string The device identifier (identifier) Recommended Experimental

disk.io.direction has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value Description Stability
read read Experimental
write write Experimental

Metric: system.disk.operations

This metric is recommended.

Name Instrument Type Unit (UCUM) Description Stability
system.disk.operations Counter {operation} Experimental
Attribute Type Description Examples Requirement Level Stability
disk.io.direction string The disk IO operation direction. read Recommended Experimental
system.device string The device identifier (identifier) Recommended Experimental

disk.io.direction has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value Description Stability
read read Experimental
write write Experimental

Metric: system.disk.io_time

This metric is recommended.

Name Instrument Type Unit (UCUM) Description Stability
system.disk.io_time Counter s Time disk spent activated [1] Experimental

[1]: The real elapsed time ("wall clock") used in the I/O path (time from operations running in parallel are not counted). Measured as:

Attribute Type Description Examples Requirement Level Stability
system.device string The device identifier (identifier) Recommended Experimental

Metric: system.disk.operation_time

This metric is recommended.

Name Instrument Type Unit (UCUM) Description Stability
system.disk.operation_time Counter s Sum of the time each operation took to complete [1] Experimental

[1]: Because it is the sum of time each request took, parallel-issued requests each contribute to make the count grow. Measured as:

  • Linux: Fields 7 & 11 from procfs-diskstats
  • Windows: "Avg. Disk sec/Read" perf counter multiplied by "Disk Reads/sec" perf counter (similar for Writes)
Attribute Type Description Examples Requirement Level Stability
disk.io.direction string The disk IO operation direction. read Recommended Experimental
system.device string The device identifier (identifier) Recommended Experimental

disk.io.direction has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value Description Stability
read read Experimental
write write Experimental

Metric: system.disk.merged

This metric is recommended.

Name Instrument Type Unit (UCUM) Description Stability
system.disk.merged Counter {operation} Experimental
Attribute Type Description Examples Requirement Level Stability
disk.io.direction string The disk IO operation direction. read Recommended Experimental
system.device string The device identifier (identifier) Recommended Experimental

disk.io.direction has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value Description Stability
read read Experimental
write write Experimental

Filesystem Metrics

Description: System level filesystem metrics captured under the namespace system.filesystem.

Metric: system.filesystem.usage

This metric is recommended.

Name Instrument Type Unit (UCUM) Description Stability
system.filesystem.usage UpDownCounter By Experimental
Attribute Type Description Examples Requirement Level Stability
system.device string The device identifier (identifier) Recommended Experimental
system.filesystem.mode string The filesystem mode rw, ro Recommended Experimental
system.filesystem.mountpoint string The filesystem mount path /mnt/data Recommended Experimental
system.filesystem.state string The filesystem state used Recommended Experimental
system.filesystem.type string The filesystem type ext4 Recommended Experimental

system.filesystem.state has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value Description Stability
free free Experimental
reserved reserved Experimental
used used Experimental

system.filesystem.type has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value Description Stability
exfat exfat Experimental
ext4 ext4 Experimental
fat32 fat32 Experimental
hfsplus hfsplus Experimental
ntfs ntfs Experimental
refs refs Experimental

Metric: system.filesystem.utilization

This metric is recommended.

Name Instrument Type Unit (UCUM) Description Stability
system.filesystem.utilization Gauge 1 Experimental
Attribute Type Description Examples Requirement Level Stability
system.device string The device identifier (identifier) Recommended Experimental
system.filesystem.mode string The filesystem mode rw, ro Recommended Experimental
system.filesystem.mountpoint string The filesystem mount path /mnt/data Recommended Experimental
system.filesystem.state string The filesystem state used Recommended Experimental
system.filesystem.type string The filesystem type ext4 Recommended Experimental

system.filesystem.state has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value Description Stability
free free Experimental
reserved reserved Experimental
used used Experimental

system.filesystem.type has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value Description Stability
exfat exfat Experimental
ext4 ext4 Experimental
fat32 fat32 Experimental
hfsplus hfsplus Experimental
ntfs ntfs Experimental
refs refs Experimental

Network Metrics

Description: System level network metrics captured under the namespace system.network.

Metric: system.network.dropped

This metric is recommended.

Name Instrument Type Unit (UCUM) Description Stability
system.network.dropped Counter {packet} Count of packets that are dropped or discarded even though there was no error [1] Experimental

[1]: Measured as:

Attribute Type Description Examples Requirement Level Stability
network.io.direction string The network IO operation direction. transmit Recommended Experimental
system.device string The device identifier (identifier) Recommended Experimental

network.io.direction has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value Description Stability
receive receive Experimental
transmit transmit Experimental

Metric: system.network.packets

This metric is recommended.

Name Instrument Type Unit (UCUM) Description Stability
system.network.packets Counter {packet} Experimental
Attribute Type Description Examples Requirement Level Stability
network.io.direction string The network IO operation direction. transmit Recommended Experimental
system.device string The device identifier (identifier) Recommended Experimental

network.io.direction has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value Description Stability
receive receive Experimental
transmit transmit Experimental

Metric: system.network.errors

This metric is recommended.

Name Instrument Type Unit (UCUM) Description Stability
system.network.errors Counter {error} Count of network errors detected [1] Experimental

[1]: Measured as:

Attribute Type Description Examples Requirement Level Stability
network.io.direction string The network IO operation direction. transmit Recommended Experimental
system.device string The device identifier (identifier) Recommended Experimental

network.io.direction has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value Description Stability
receive receive Experimental
transmit transmit Experimental

Metric: system.network.io

This metric is recommended.

Name Instrument Type Unit (UCUM) Description Stability
system.network.io Counter By Experimental
Attribute Type Description Examples Requirement Level Stability
network.io.direction string The network IO operation direction. transmit Recommended Experimental
system.device string The device identifier (identifier) Recommended Experimental

network.io.direction has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value Description Stability
receive receive Experimental
transmit transmit Experimental

Metric: system.network.connections

This metric is recommended.

Name Instrument Type Unit (UCUM) Description Stability
system.network.connections UpDownCounter {connection} Experimental
Attribute Type Description Examples Requirement Level Stability
network.transport string OSI transport layer or inter-process communication method. [1] tcp; udp Recommended Stable
system.device string The device identifier (identifier) Recommended Experimental
system.network.state string A stateless protocol MUST NOT set this attribute close_wait Recommended Experimental

[1]: The value SHOULD be normalized to lowercase.

Consider always setting the transport when setting a port number, since a port number is ambiguous without knowing the transport. For example different processes could be listening on TCP port 12345 and UDP port 12345.

network.transport has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value Description Stability
pipe Named or anonymous pipe. Stable
tcp TCP Stable
udp UDP Stable
unix Unix domain socket Stable

system.network.state has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value Description Stability
close close Experimental
close_wait close_wait Experimental
closing closing Experimental
delete delete Experimental
established established Experimental
fin_wait_1 fin_wait_1 Experimental
fin_wait_2 fin_wait_2 Experimental
last_ack last_ack Experimental
listen listen Experimental
syn_recv syn_recv Experimental
syn_sent syn_sent Experimental
time_wait time_wait Experimental

Aggregate System Process Metrics

Description: System level aggregate process metrics captured under the namespace system.process. For metrics at the individual process level, see process metrics.

Metric: system.process.count

This metric is recommended.

Name Instrument Type Unit (UCUM) Description Stability
system.process.count UpDownCounter {process} Total number of processes in each state Experimental
Attribute Type Description Examples Requirement Level Stability
system.process.status string The process state, e.g., Linux Process State Codes running Recommended Experimental

system.process.status has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value Description Stability
defunct defunct Experimental
running running Experimental
sleeping sleeping Experimental
stopped stopped Experimental

Metric: system.process.created

This metric is recommended.

Name Instrument Type Unit (UCUM) Description Stability
system.process.created Counter {process} Total number of processes created over uptime of the host Experimental

system.{os}. - OS Specific System Metrics

Instrument names for system level metrics that have different and conflicting meaning across multiple OSes should be prefixed with system.{os}. and follow the hierarchies listed above for different entities like CPU, memory, and network.

For example, UNIX load average over a given interval is not well standardized and its value across different UNIX like OSes may vary despite being under similar load:

Without getting into the vagaries of every Unix-like operating system in existence, the load average more or less represents the average number of processes that are in the running (using the CPU) or runnable (waiting for the CPU) states. One notable exception exists: Linux includes processes in uninterruptible sleep states, typically waiting for some I/O activity to complete. This can markedly increase the load average on Linux systems.

(source of quote, linux source code)

An instrument for load average over 1 minute on Linux could be named system.linux.cpu.load_1m, reusing the cpu name proposed above and having an {os} prefix to split this metric across OSes.

Metric: system.linux.memory.available

Name Instrument Type Unit (UCUM) Description Stability
system.linux.memory.available UpDownCounter By An estimate of how much memory is available for starting new applications, without causing swapping [1] Experimental

[1]: This is an alternative to system.memory.usage metric with state=free. Linux starting from 3.14 exports "available" memory. It takes "free" memory as a baseline, and then factors in kernel-specific values. This is supposed to be more accurate than just "free" memory. For reference, see the calculations here. See also MemAvailable in /proc/meminfo.