You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
With Nvidia GPU device plugin, the allocatable GPU is a fixed number, i.e., the number of GPUs plugged to the node. It won't update based on current usage. Therefore, when scheduler decide the target replica (gpu number) for the unified job, the available GPU needs to be calculated by listing existing job's gpu request, also should add some safe exit process (e.g., reduce replica size after fail scheduling N times, drop job after M times of failure) to avoid keeping create/delete unsuccessful pods.
The text was updated successfully, but these errors were encountered:
when describe node, at the end it shows allocated resources, which is the correct number. could call describe node command and extract that field. After search the describe code, they also implemented by scanning all the pods on the node. So could just use the old method to scan all the running pods and their requested GPU
Merged with pull request #90 .
Interesting thing about select running pods from a given node is you need to define the Index first with manager. the List MatchingField is not automatically available. Also don't use real path, may shallow with existing ones and cause irrelevant errors.
Kuberbuilder's jobOwnerKey is a reference. kubernetes-sigs/kubebuilder#1422
With Nvidia GPU device plugin, the allocatable GPU is a fixed number, i.e., the number of GPUs plugged to the node. It won't update based on current usage. Therefore, when scheduler decide the target replica (gpu number) for the unified job, the available GPU needs to be calculated by listing existing job's gpu request, also should add some safe exit process (e.g., reduce replica size after fail scheduling N times, drop job after M times of failure) to avoid keeping create/delete unsuccessful pods.
The text was updated successfully, but these errors were encountered: