Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The pipelined task may cause cluster autoscaler scale up more nodes #3499

Closed
bibibox opened this issue May 28, 2024 · 3 comments
Closed

The pipelined task may cause cluster autoscaler scale up more nodes #3499

bibibox opened this issue May 28, 2024 · 3 comments
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@bibibox
Copy link

bibibox commented May 28, 2024

What happened:
When one pod preempts other pods on the node, volcano will mark the related task as pipelined status in the cache, and update pod condition to UnSchedulable. The k8s ClusterAutoScaler will consider this pod as unschedulable and scale up one node for it.

What you expected to happen:
The k8s ClusterAutoScaler should not scale up node in this scene.

How to reproduce it (as minimally and precisely as possible):

  1. create a cluster with volcano and k8s autoscaler, the cluster has a free 4U4G node, and the node pool is also 4U4G
  2. create a low priority deploy with 4 replica, make the node occupied
  3. create a high priority deploy with 4 replica, to preempt the low priority pods
  4. we can see the autoscaler scale up 2 nodes, but just need scale up 1 node indeed

Anything else we need to know?:

Environment:

  • Volcano Version: 1.9.0
  • Kubernetes version (use kubectl version):
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Others: k8s cluster autoscaler 1.29.13
@bibibox bibibox added the kind/bug Categorizes issue or PR as related to a bug. label May 28, 2024
@william-wang
Copy link
Member

@bibibox please update the How to reproduce it section to provide reproducing step info. Thanks.

@Monokaix
Copy link
Member

/close

@volcano-sh-bot
Copy link
Contributor

@Monokaix: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests

4 participants