Error with unsupported new metrics on V100 GPU's

[hassanbabaie]

Running 2.4.6-2.6.9 and if I enable the following metrics:

DCGM_FI_PROF_PIPE_TENSOR_IMMA_ACTIVE
DCGM_FI_PROF_PIPE_TENSOR_HMMA_ACTIVE

The DaemonSet set works and reports metrics for nodes with A100 GPU's

For nodes with V100-16GB GPUs the DaemonSet fails with the following message:

setting up csv
/etc/dcgm-exporter/dcp-metrics-bolt.csv
done
time="2022-07-20T01:54:11Z" level=info msg="Starting dcgm-exporter"
time="2022-07-20T01:54:11Z" level=info msg="DCGM successfully initialized!"
time="2022-07-20T01:54:11Z" level=info msg="Collecting DCP Metrics"
time="2022-07-20T01:54:11Z" level=info msg="No configmap data specified, falling back to metric file /etc/dcgm-exporter/dcp-metrics-bolt.csv"
time="2022-07-20T01:54:12Z" level=fatal msg="Error watching fields: Feature not supported"
running

[nikkon-dev]

@hassanbabaie,

The 1013 and 1014 metrics are not supported on pre-Ampere GPUs.

WBR,
Nik

[hassanbabaie]

Thanks @nikkon-dev however it would be better if there was some form of graceful skip, if they are not supported log it and don't report any metrics for those.

For example when using profiling metrics with an unsupported driver version the metrics are just not provided/skipped but we can still use the same DCGM DaemonSet for all nodes.

[glowkey]

Agreed. We'll modify that behavior.

[hassanbabaie]

thank you!