Repository navigation
Replies: 1 comment
|
Yes. The node's glibc 2.38 is basically irrelevant to HAMi's isolation, because HAMi-core ( The device plugin bind-mounts What matters is that the lib was built against a glibc no newer than the one in your images:
To check that isolation is really active in one of your pods: kubectl exec -n algorithm-test <pod> -- cat /etc/ld.so.preload # should point at .../vgpu/libvgpu.so
kubectl exec -n algorithm-test <pod> -- nvidia-smi # total memory should be ~1000MiB, not 16376MiBThen allocate more than 1 GB in that pod (e.g. a small PyTorch Drafted with AI assistance; I checked the code references above. |
Uh oh!
There was an error while loading. Please reload this page.
Does the Hami v2.9.0 version support the glibc version for GPU nodes being 2.38?
Are there any potential risks or problems?
Currently, hami 2.9.0 has been deployed, and all pods are running normally. One GPU server can host multiple pods that require GPU resources, and the isolation should have taken effect. GPU model: NVIDIA RTX A400
k8s version:v1.30.0
k8s runc:containerd
hami:v2.9.0
[root@sv-cz-alg-test-master-01 hami]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
hami-device-plugin-sjc2r 2/2 Running 0 140m
hami-scheduler-7d67d5b44b-q5p49 2/2 Running 0 140m
The operating systems of all Kubernetes nodes (including GPU nodes) info:
[root@alg-gpu-k8s-test-work01 ~]# cat /etc/os-release
NAME="openEuler"
VERSION="24.03 (LTS-SP1)"
ID="openEuler"
VERSION_ID="24.03"
PRETTY_NAME="openEuler 24.03 (LTS-SP1)"
ANSI_COLOR="0;31"
[root@alg-gpu-k8s-test-work01 ~]# uname -a
Linux alg-gpu-k8s-test-work01 6.6.0-129.0.0.127.oe2403sp1.x86_64 #1 SMP Wed Dec 17 18:45:06 CST 2025 x86_64 x86_64 x86_64 GNU/Linux
[root@alg-gpu-k8s-test-work01 ~]#
[root@alg-gpu-k8s-test-work01 ~]# ldd --version
ldd (GNU libc) 2.38
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.
[root@sv-wx-alg-gpu-k8s-test-work01 ~]# nvidia-smi
Thu Jun 11 12:04:55 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.126.09 Driver Version: 580.126.09 CUDA Version: 13.0 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA RTX A4000 Off | 00000000:18:00.0 Off | Off |
| 41% 50C P8 17W / 140W | 33MiB / 16376MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 2948 G /usr/libexec/Xorg 9MiB |
| 0 N/A N/A 6133 G /usr/bin/gnome-shell 4MiB |
+-----------------------------------------------------------------------------------------+
下面这个deployment能运行起来,整个集群只有1个gpu节点,没有安装hami之前,只能运行1个pod,该pod独占1张显卡,另外1个pod会被pending,现在安装hami之后,两个pod都能Running了;
The following deployment can be run. In this entire cluster, there is only 1 GPU node. Before installing HAMI, only 1 pod could run, and that pod exclusively used 1 graphics card. The other pod would be in a pending state. Now after installing HAMI, both of the two pods can run:
apiVersion: apps/v1
kind: Deployment
metadata:
name: ubuntu2204-svai-runai-hami
namespace: algorithm-test # 可以修改为你的命名空间
labels:
app: ubuntu2204-svai-runai-hami
os: ubuntu2204-svai-runai-hami
spec:
strategy:
type: Recreate
replicas: 2 # 可以根据需要调整副本数量
revisionHistoryLimit: 3 # 保留3个旧的ReplicaSet用于回滚
selector:
matchLabels:
app: ubuntu2204-svai-runai-hami
template:
metadata:
labels:
app: ubuntu2204-svai-runai-hami
os: ubuntu2204-svai-runai-hami
spec:
# 重启策略:Always, OnFailure, Never
restartPolicy: Always
# 容忍度配置(如果节点有污点)
tolerations:
- key: "node.kubernetes.io/not-ready"
operator: "Exists"
effect: "NoExecute"
tolerationSeconds: 300
- key: "node.kubernetes.io/unreachable"
operator: "Exists"
effect: "NoExecute"
tolerationSeconds: 300
All reactions