Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

default queue with a pod has pvc will be overused #3271

Closed
lowang-bh opened this issue Dec 17, 2023 · 4 comments · Fixed by #3285
Closed

default queue with a pod has pvc will be overused #3271

lowang-bh opened this issue Dec 17, 2023 · 4 comments · Fixed by #3285
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@lowang-bh
Copy link
Member

lowang-bh commented Dec 17, 2023

What happened:
image

What you expected to happen:

pod scheduled succcessfully with pvc

How to reproduce it (as minimally and precisely as possible):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-volcano-pvc
spec:
  selector:
    matchLabels:
      app: deployment-volcano-pvc
  replicas: 1
  template:
    metadata:
      annotations:
        scheduling.k8s.io/group-name: pg-volcano-pvc
      labels:
        app: deployment-volcano-pvc
    spec:
      schedulerName: volcano
      terminationGracePeriodSeconds: 1
      containers:
        - name: nginx
          image: nginx:1.14.2
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: '100m'
              memory: 100Mi
            limits:
              cpu: '100m'
              memory: 100Mi
          volumeMounts:
            - mountPath: /mnt/claim
              name: local-vol
      volumes:
        - name: local-vol
          persistentVolumeClaim:
            claimName: claim-for-deployment-volcano-pvc
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: claim-for-deployment-volcano-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 100Mi
  # storageClassName: csi-lvm # null to use default storage class
  volumeMode: Filesystem
---
apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
  name: pg-volcano-pvc
  namespace: default
spec:
  minMember: 1
  queue: default
---
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: default
spec:
  guarantee:
    resource:
      cpu: 2
      memory: 1Gi
  reclaimable: true
  weight: 1

Anything else we need to know?:

Environment:

  • Volcano Version: master
  • Kubernetes version (use kubectl version): client v1.27.1, server: v1.28.0
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Others:
@lowang-bh lowang-bh added the kind/bug Categorizes issue or PR as related to a bug. label Dec 17, 2023
@lowang-bh lowang-bh changed the title pod with pvc in default queue will be overused default queue with a pod has pvc will be overused Dec 17, 2023
@Monokaix
Copy link
Member

Hi, please paste cisnode resource yaml output.

@lowang-bh
Copy link
Member Author

This is the storage class, and I checked it works on v1.7.0

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"},"name":"standard"},"provisioner":"rancher.io/local-path","reclaimPolicy":"Delete","volumeBindingMode":"WaitForFirstConsumer"}
    storageclass.kubernetes.io/is-default-class: "true"
  creationTimestamp: "2023-12-20T14:53:36Z"
  name: standard
  resourceVersion: "267"
  uid: fac2253f-22bb-4d32-a211-39896c8bb829
provisioner: rancher.io/local-path
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
image

@lowang-bh
Copy link
Member Author

v1.8.0 also works
image

@Monokaix
Copy link
Member

Monokaix commented Dec 21, 2023

When set csi resource, volcano scheduler will check spec.driver of csinode, if spec.driver is null, the csi resource will not be added to node.Allocatable, and the csi reource will be missing in cluster resources.

Then deserved resource of csi resource computed in proportion will be zero, and task's request is 1, so it's not allocatable because of not enough csi resource.

dongjoon-hyun added a commit to apache/spark that referenced this issue Jan 20, 2024
### What changes were proposed in this pull request?

This PR aims to upgrade `Volcano` to 1.8.1 in K8s integration test document and GitHub Action job.

### Why are the changes needed?

To bring the latest bug fixes in addition to the test coverage for Volcano scheduler 1.8.2.
- https://github.com/volcano-sh/volcano/releases/tag/v1.8.2

Note that the following `Volcano` bug fix reveals our test suite issue. So, it's disabled in Volcano testing.
- volcano-sh/volcano#3271

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass the CIs.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #44779 from dongjoon-hyun/SPARK-46755.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants