Queue Capability bug

[silenceli]

Queue Capability maybe has a bug. In certain scenarios, queues cannot limit users' usage

How to reproduce

• set queue capability nvidia.com/gpu = 2
• create 4 job, each job need nvidia.com/gpu = 1
• at that time queue works fine, 2 running and 2 pending, see below

[root@k8s-master-1-15-1ef36b060 ~]# kubectl describe queue training-jarvis-dev-team-k8s-training-test
Name:         training-jarvis-dev-team-k8s-training-test
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  scheduling.volcano.sh/v1beta1
Kind:         Queue
Metadata:
  Creation Timestamp:  2020-04-23T08:09:56Z
  Generation:          20
  Resource Version:    75588153
  Self Link:           /apis/scheduling.volcano.sh/v1beta1/queues/training-jarvis-dev-team-k8s-training-test
  UID:                 5ac970cf-9894-4e74-94fb-21e89cd55b68
Spec:
  Capability:
    Cpu:             80
    Memory:          399360Mi
    nvidia.com/gpu:  2
    qiyi.com/fuse:   10000
  Weight:            1
Status:
  Pending:  2
  Running:  2
  State:    Open
Events:     <none>

• stop one running job
• then the two pending job will RUNNING

[root@k8s-master-1-15-1ef36b060 ~]# kubectl describe queue training-jarvis-dev-team-k8s-training-test
Name:         training-jarvis-dev-team-k8s-training-test
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  scheduling.volcano.sh/v1beta1
Kind:         Queue
Metadata:
  Creation Timestamp:  2020-04-23T08:09:56Z
  Generation:          20
  Resource Version:    75589682
  Self Link:           /apis/scheduling.volcano.sh/v1beta1/queues/training-jarvis-dev-team-k8s-training-test
  UID:                 5ac970cf-9894-4e74-94fb-21e89cd55b68
Spec:
  Capability:
    Cpu:             80
    Memory:          399360Mi
    nvidia.com/gpu:  2
    qiyi.com/fuse:   10000
  Weight:            1
Status:
  Running:  3
  State:    Open
Events:     <none>

[k82cn]

/cc @Thor-wl

[Thor-wl]

@silenceli The above status with 3 job running is the finally status? Can this phenomenon be reproduced stably？

[Thor-wl]

@silenceli please provide the volcano version you are using

[Thor-wl]

fix PR: #949

[hzxuzhonghu]

fixed by #959 and #966

@silenceli Would you please have a try?

@Thor-wl We need an e2e case to cover this case