New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Insufficient GPU on Node: xxx #23

Open

lyon-v opened this issue Nov 7, 2023 · 5 comments

lyon-v commented Nov 7, 2023

第一次挂载成功了，后面卸载再次deploy 显示这个 Insufficient GPU on Node: yigou-dev-102-46，gpu 实际空闲

Owner

pokerfaceSad commented Nov 7, 2023

describe node看下GPU资源是否空闲

Author

lyon-v commented Nov 8, 2023 •

edited

Loading

1.master日志：
[root@yigou-dev-102-45 examples]# kubectl logs gpu-mounter-master-bc547448d-t5nkl -n kube-system
2023-11-07T14:32:53.960Z INFO GPUMounter-master/main.go:239 Start gpu mounter master on :8080
2023-11-08T01:04:59.791Z INFO GPUMounter-master/main.go:25 access add gpu service
2023-11-08T01:04:59.791Z INFO GPUMounter-master/main.go:30 Pod: gpu-pod-1 Namespace: default GPU Num: 1 Is entire mount: false
2023-11-08T01:04:59.812Z INFO GPUMounter-master/main.go:66 Found Pod: gpu-pod-1 in Namespace: default on Node: yigou-dev-102-46
2023-11-08T01:04:59.822Z INFO GPUMounter-master/main.go:265 Worker: gpu-mounter-workers-s8j2f Node: yigou-dev-102-46
2023-11-08T01:04:59.925Z ERROR GPUMounter-master/main.go:109 Insufficient GPU on Node: yigou-dev-102-46

############################################################################################
2.worker 日志：
[root@yigou-dev-102-45 elastic-jupyter]# kubectl logs gpu-mounter-workers-s8j2f -n kube-system
2023-11-07T14:30:26.648Z INFO GPUMounter-worker/main.go:15 Service Starting...
2023-11-07T14:30:26.648Z INFO gpu-mount/server.go:22 Creating gpu mounter
2023-11-07T14:30:26.648Z INFO allocator/allocator.go:28 Creating gpu allocator
2023-11-07T14:30:26.648Z INFO collector/collector.go:24 Creating gpu collector
2023-11-07T14:30:26.648Z INFO collector/collector.go:42 Start get gpu info
2023-11-07T14:30:26.652Z INFO collector/collector.go:53 GPU Num: 2
2023-11-07T14:30:26.664Z INFO collector/collector.go:91 Updating GPU status
2023-11-07T14:30:26.667Z INFO collector/collector.go:136 GPU status update successfully
2023-11-07T14:30:26.667Z INFO collector/collector.go:36 Successfully update gpu status
2023-11-07T14:30:26.667Z INFO allocator/allocator.go:35 Successfully created gpu collector
2023-11-07T14:30:26.667Z INFO gpu-mount/server.go:29 Successfully created gpu allocator
2023-11-07T14:30:26.667Z INFO GPUMounter-worker/main.go:22 Successfully created gpu mounter
2023-11-08T01:04:59.825Z INFO gpu-mount/server.go:35 AddGPU Service Called
2023-11-08T01:04:59.825Z INFO gpu-mount/server.go:36 request: pod_name:"gpu-pod-1" namespace:"default" gpu_num:1
2023-11-08T01:04:59.848Z INFO gpu-mount/server.go:55 Successfully get Pod: default in cluster
2023-11-08T01:04:59.848Z INFO allocator/allocator.go:159 Get pod default/gpu-pod-1 mount type
2023-11-08T01:04:59.848Z INFO collector/collector.go:91 Updating GPU status
2023-11-08T01:04:59.851Z INFO collector/collector.go:136 GPU status update successfully
2023-11-08T01:04:59.880Z INFO allocator/allocator.go:59 Creating GPU Slave Pod: gpu-pod-1-slave-pod-1d3148 for Owner Pod: gpu-pod-1
2023-11-08T01:04:59.880Z INFO allocator/allocator.go:238 Checking Pods: gpu-pod-1-slave-pod-1d3148 state
2023-11-08T01:04:59.882Z INFO allocator/allocator.go:264 Pod: gpu-pod-1-slave-pod-1d3148 creating
2023-11-08T01:04:59.886Z INFO allocator/allocator.go:264 Pod: gpu-pod-1-slave-pod-1d3148 creating
2023-11-08T01:04:59.917Z INFO allocator/allocator.go:268 No enough gpu for Pod: gpu-pod-1-slave-pod-1d3148
2023-11-08T01:04:59.925Z ERROR gpu-mount/server.go:70 Insufficient gpu for Pod: gpu-pod-1 Namespace: default

#######################################################################################
3.node 资源：
[root@yigou-dev-102-45 yamls]# kubectl describe node yigou-dev-102-46
Name: yigou-dev-102-46
Roles: cpu,training
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
fluid.io/dataset-num=1
fluid.io/f-fluid-fluiddataset=true
fluid.io/s-alluxio-fluid-fluiddataset=true
fluid.io/s-fluid-fluiddataset=true
fluid.io/s-h-alluxio-d-fluid-fluiddataset=5GiB
fluid.io/s-h-alluxio-t-fluid-fluiddataset=5GiB
gpu-mounter-enable=enable
kubernetes.io/arch=amd64
kubernetes.io/hostname=yigou-dev-102-46
kubernetes.io/os=linux
node-role.kubernetes.io/cpu=true
node-role.kubernetes.io/training=true
Annotations: csi.volume.kubernetes.io/nodeid: {"csi.tigera.io":"yigou-dev-102-46","fuse.csi.fluid.io":"yigou-dev-102-46"}
kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/cri-dockerd.sock
node.alpha.kubernetes.io/ttl: 0
projectcalico.org/IPv4Address: 10.0.102.46/24
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Mon, 28 Aug 2023 17:11:40 +0800
Taints:
Unschedulable: false
Lease:
HolderIdentity: yigou-dev-102-46
AcquireTime:
RenewTime: Wed, 08 Nov 2023 10:58:29 +0800
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message

NetworkUnavailable False Tue, 07 Nov 2023 18:13:30 +0800 Tue, 07 Nov 2023 18:13:30 +0800 CalicoIsUp Calico is running on this node
MemoryPressure False Wed, 08 Nov 2023 10:58:23 +0800 Fri, 03 Nov 2023 10:39:46 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Wed, 08 Nov 2023 10:58:23 +0800 Fri, 03 Nov 2023 10:39:46 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Wed, 08 Nov 2023 10:58:23 +0800 Fri, 03 Nov 2023 10:39:46 +0800 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Wed, 08 Nov 2023 10:58:23 +0800 Tue, 07 Nov 2023 18:13:01 +0800 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 10.0.102.46
Hostname: yigou-dev-102-46
Capacity:
cpu: 32
ephemeral-storage: 1018975Mi
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 65801852Ki
nvidia.com/gpu: 0
nvidia.com/nvidia-rtx-3090: 2
pods: 110
Allocatable:
cpu: 32
ephemeral-storage: 961625455048
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 65699452Ki
nvidia.com/gpu: 0
nvidia.com/nvidia-rtx-3090: 2
pods: 110
System Info:
Machine ID: 2a5da2a3fe9d480f97ca66b4d8f4287b
System UUID: 32B53042-47EB-9349-E452-0B470FA25211
Boot ID: 87d25649-faa9-45da-a0ce-5b4b102fd56e
Kernel Version: 3.10.0-1160.99.1.el7.x86_64
OS Image: CentOS Linux 7 (Core)
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://24.0.6
Kubelet Version: v0.0.0-master+5244794d27b4cc68290bc496b00e248857ac8b47
Kube-Proxy Version: v0.0.0-master+5244794d27b4cc68290bc496b00e248857ac8b47
PodCIDR: 100.64.1.0/24
PodCIDRs: 100.64.1.0/24
Non-terminated Pods: (30 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age

calico-system calico-node-gfzjw 0 (0%) 0 (0%) 0 (0%) 0 (0%) 28d
calico-system calico-typha-856c6c9c4c-bzzbn 0 (0%) 0 (0%) 0 (0%) 0 (0%) 71d
calico-system csi-node-driver-vlxt9 0 (0%) 0 (0%) 0 (0%) 0 (0%) 71d
default gpu-pod-1 0 (0%) 0 (0%) 0 (0%) 0 (0%) 42s
elastic-jupyter-operator-system elastic-jupyter-operator-controller-manager-5d559bbbb8-77frj 100m (0%) 100m (0%) 20Mi (0%) 30Mi (0%) 23h
fluid-system alluxioruntime-controller-5b4fd8d788-56c4k 100m (0%) 100m (0%) 200Mi (0%) 1536Mi (2%) 49d
fluid-system csi-nodeplugin-fluid-4dw44 0 (0%) 0 (0%) 0 (0%) 0 (0%) 15d
fluid-system dataset-controller-665ff849b7-cnrh9 100m (0%) 100m (0%) 200Mi (0%) 1536Mi (2%) 49d
fluid-system dataset-controller-665ff849b7-xrvm7 100m (0%) 100m (0%) 200Mi (0%) 1536Mi (2%) 29d
fluid-system fluid-webhook-8689694b95-jsn49 0 (0%) 0 (0%) 0 (0%) 0 (0%) 49d
fluid-system fluidapp-controller-698b685d4f-z84r6 100m (0%) 100m (0%) 200Mi (0%) 1536Mi (2%) 49d
fluid-system thinruntime-controller-674bb4784b-qvcks 100m (0%) 100m (0%) 200Mi (0%) 1536Mi (2%) 49d
fluid fluiddataset-fuse-264x4 0 (0%) 0 (0%) 0 (0%) 0 (0%) 16d
fluid fluiddataset-master-0 0 (0%) 0 (0%) 0 (0%) 0 (0%) 5d
fluid fluiddataset-worker-0 0 (0%) 0 (0%) 0 (0%) 0 (0%) 5d
heros-controllers-system hero-controllers-controller-manager-cbdb77cf6-l7bjl 15m (0%) 1 (3%) 128Mi (0%) 256Mi (0%) 38h
heros-system file-proxy-5b5f76cf8d-xnj72 0 (0%) 0 (0%) 0 (0%) 0 (0%) 18h
kube-system gpu-mounter-master-bc547448d-x77th 0 (0%) 0 (0%) 0 (0%) 0 (0%) 42m
kube-system gpu-mounter-workers-7lf7k 0 (0%) 0 (0%) 0 (0%) 0 (0%) 42m
kube-system kube-proxy-rh2vt 0 (0%) 0 (0%) 0 (0%) 0 (0%) 28d
kube-system kube-sealos-lvscare-yigou-dev-102-46 0 (0%) 0 (0%) 0 (0%) 0 (0%) 71d
kube-system tigera-operator-66fd59dc66-tn24j 0 (0%) 0 (0%) 0 (0%) 0 (0%) 71d
monitoring alertmanager-main-0 104m (0%) 200m (0%) 150Mi (0%) 150Mi (0%) 5d
monitoring cadvisor-8cwmf 400m (1%) 800m (2%) 400Mi (0%) 2000Mi (3%) 12d
monitoring dcgm-exporter-7rxkt 0 (0%) 0 (0%) 0 (0%) 0 (0%) 18h
monitoring loki-stack-0 0 (0%) 0 (0%) 0 (0%) 0 (0%) 5d
monitoring loki-stack-promtail-x9zwg 0 (0%) 0 (0%) 0 (0%) 0 (0%) 16d
monitoring node-exporter-ppdvb 112m (0%) 270m (0%) 200Mi (0%) 220Mi (0%) 63d
monitoring prometheus-k8s-0 100m (0%) 100m (0%) 450Mi (0%) 50Mi (0%) 5d
nvidia-device-plugin nvdp-nvidia-device-plugin-r4zjr 0 (0%) 0 (0%) 0 (0%) 0 (0%) 18h
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits

cpu 1331m (4%) 2970m (9%)
memory 2348Mi (3%) 10386Mi (16%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
nvidia.com/gpu 0 0
nvidia.com/nvidia-rtx-3090 0 0
Events:
Type Reason Age From Message

Normal RegisteredNode 32m node-controller Node yigou-dev-102-46 event: Registered Node yigou-dev-102-46 in Controller

Author

lyon-v commented Nov 8, 2023 •

edited

Loading

我可能说错了，第一次也没挂上gpu, 但是gpu是空闲的。上面的信息是master和worker的日志，和46节点的信息，gpu-pool 下面没有slave-pod.
[root@yigou-dev-102-45 ~]# kubelet --version
Kubernetes v1.25.13
这是k8s版本信息

Owner

pokerfaceSad commented Nov 8, 2023

在k8s 1.20+有一个已知问题，ownerReference不允许跨namespaces，因此slavePod会创建失败

参考下#19 (comment)

Author

lyon-v commented Nov 8, 2023 •

edited

Loading

[root@yigou-dev-102-45 ~]# cat /etc/docker/daemon.json
{
"data-root": "/var/lib/docker",
"exec-opts": [
"native.cgroupdriver=systemd"
],
"insecure-registries": [
"registry.bitahub.com:5000",
"registry.hub.com:5000",
"docker-user.cambricon.com:30080",
"10.10.8.100:5000",
"10.11.3.8:5000",
"112.31.12.176:5000",
"10.12.4.35:5000",
"10.0.0.12:5000"
],
"log-driver": "json-file",
"log-level": "warn",
"log-opts": {
"max-file": "3",
"max-size": "10m"
},
"max-concurrent-downloads": 20,
"registry-mirrors": [
"https://reg-mirror.qiniu.com/",
"https://pqs5j944.mirror.aliyuncs.com",
"https://7bezldxe.mirror.aliyuncs.com/",
"https://registry.docker-cn.com",
"http://hub-mirror.c.163.com",
"https://docker.mirrors.ustc.edu.cn/"
],
"default-runtime": "nvidia",
"runtimes": {
"nvidia": {
"args": [],
"path": "nvidia-container-runtime"
}
}
}
上面是我的/etc/docker/daemon.json,我试试将改个环境变量

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment