volcano vgpu metrics not update properly

[archlitchi]

If you submit a vgpu job, you can see the corresponding metrics by using scheduler metrics, as follows:

task yaml:

apiVersion: v1
kind: Pod
metadata:
  name: pod1
spec:
  restartPolicy: OnFailure
  schedulerName: volcano
  containers:
  - image: ubuntu:20.04
    name: pod1-ctr
    command: ["sleep"]
    args: ["100000"]
    resources:
      limits:
        volcano.sh/vgpu-memory: 1024
        volcano.sh/vgpu-number: 1

then by visiting scheduler metrics, you can get the vgpu overview of vc-scheduler

curl {vc-scheduler}:8080/metrics

# HELP volcano_vgpu_device_allocated_cores The percentage of gpu compute cores allocated in this card
# TYPE volcano_vgpu_device_allocated_cores gauge
volcano_vgpu_device_allocated_cores{devID="GPU-00552014-5c87-89ac-b1a6-7b53aa24b0ec"} 0
volcano_vgpu_device_allocated_cores{devID="GPU-0fc3eda5-e98b-a25b-5b0d-cf5c855d1448"} 0
# HELP volcano_vgpu_device_allocated_memory The number of vgpu memory allocated in this card
# TYPE volcano_vgpu_device_allocated_memory gauge
volcano_vgpu_device_allocated_memory{devID="GPU-00552014-5c87-89ac-b1a6-7b53aa24b0ec"} 0
volcano_vgpu_device_allocated_memory{devID="GPU-0fc3eda5-e98b-a25b-5b0d-cf5c855d1448"} 1024
# HELP volcano_vgpu_device_memory_limit The number of total device memory allocated in this card
# TYPE volcano_vgpu_device_memory_limit gauge
volcano_vgpu_device_memory_limit{devID="GPU-00552014-5c87-89ac-b1a6-7b53aa24b0ec"} 32768
volcano_vgpu_device_memory_limit{devID="GPU-0fc3eda5-e98b-a25b-5b0d-cf5c855d1448"} 32768
# HELP volcano_vgpu_device_shared_number The number of vgpu tasks sharing this card
# TYPE volcano_vgpu_device_shared_number gauge
volcano_vgpu_device_shared_number{devID="GPU-00552014-5c87-89ac-b1a6-7b53aa24b0ec"} 0
volcano_vgpu_device_shared_number{devID="GPU-0fc3eda5-e98b-a25b-5b0d-cf5c855d1448"} 1

But these metrics are not cleaned up after the pod ends, these metrics are still there, even if we delete this pod.

[Monokaix]

/good-first-issue

[volcano-sh-bot]

@Monokaix:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

/good-first-issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[googs1025]

/assign

[googs1025]

Let me understand the problem first. When our pod is updated, the indicator is not updated accordingly, right? @archlitchi

[Monokaix]

he percentage of gpu compute c

I think the problem is that metric is not updated when pod deleted: ) An core codes in file pkg/scheduler/api/devices/nvidia/vgpu/metrics.go & pkg/scheduler/api/devices/nvidia/vgpu/device_info.go

[archlitchi]

Let me understand the problem first. When our pod is updated, the indicator is not updated accordingly, right? @archlitchi

yes, i will submit a patch to fix that