Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VMAgent doesn't work with VPA #693

Open
sdomme opened this issue Jul 18, 2023 · 7 comments
Open

VMAgent doesn't work with VPA #693

sdomme opened this issue Jul 18, 2023 · 7 comments
Assignees
Labels
bug Something isn't working enhancement New feature or request

Comments

@sdomme
Copy link

sdomme commented Jul 18, 2023

Hello, we are trying to setup a VPA on top of a VMAgent. Unfortunately we get the following error from the VPA

E0718 13:54:51.559558 1 cluster_feeder.go:570] Cannot get target selector from VPA's targetRef. Reason: Unhandled targetRef operator.victoriametrics.com/v1beta1 / VMAgent / application, last error Resource victoria/application has an empty selector for scale sub-resource

Here the manifest of the VPA:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: vmagent-application-vpa
  namespace: victoria
spec:
  targetRef:
    apiVersion: operator.victoriametrics.com/v1beta1
    kind: VMAgent
    name: application
  updatePolicy:
    updateMode: "Auto"
  resourcePolicy:
    containerPolicies:
      - containerName: vmagent
        maxAllowed:
          cpu: 1

Looking at the CRD of the VMAgent I would expect it to work since

 subresources:
      scale:
        labelSelectorPath: .status.selector
        specReplicasPath: .spec.shardCount
        statusReplicasPath: .status.shards
      status: {}

is configured.

Adding another piece to the CRD

- additionalPrinterColumns:
    - description: current selector
      jsonPath: .status.selector
      name: Selector
      type: string

shows an empty string.

k get vmagent application -n victoria
NAME          SHARDS COUNT   REPLICA COUNT   SELECTOR
application   0              1

Any help is appreciated. Maybe we just do something wrong?

@Haleygo
Copy link
Contributor

Haleygo commented Jul 19, 2023

Hello!
Looks like vpa has some conflict with operator on scale subresource, so it's not working now.
What's your case to scale vmagent? Maybe horizontal scaling can help.

@Haleygo Haleygo added the question Further information is requested label Jul 19, 2023
@sdomme
Copy link
Author

sdomme commented Jul 20, 2023

The issue at vpa is from us as well. We can't pinpoint the root cause of this issue yet.
We have a fleet of EKS Clusters with the same configuration for metrics. Each cluster has different workloads so the VMAgent has different resource requirements. The compute resources are scaled very dynamically in dependency of the consumption. Therefore the VPA.

@Haleygo
Copy link
Contributor

Haleygo commented Jul 20, 2023

Oh, looks like operator needs to change something here to fit vpa then.

@Haleygo Haleygo added enhancement New feature or request and removed question Further information is requested labels Jul 20, 2023
@ajaykumarmandapati
Copy link

Hello! Are there any updates on this? seems to effect us

@monotek
Copy link

monotek commented Feb 12, 2024

We also have an issue with this as we have policies in place which checks for existence of a VPA but we cant define a working one.

@f41gh7
Copy link
Collaborator

f41gh7 commented Apr 17, 2024

Sorry for delay. I think, we should prioritize it.

@f41gh7 f41gh7 added the bug Something isn't working label Apr 17, 2024
@f41gh7 f41gh7 self-assigned this Apr 17, 2024
f41gh7 added a commit that referenced this issue Apr 26, 2024
* it allows to correctly use VPA controller with vmagent scale sub-resource
#693
@f41gh7
Copy link
Collaborator

f41gh7 commented Apr 26, 2024

Fix was added with linked PR. It'll be a part of the next release.

Also, I think, that we should add direct support for VPA and HPA resources as .spec field for:

  • vmalert - VPA
  • vmagent - VPA+HPA
  • vmcluster - VPA+HPA for select and insert components
  • vmauth - HPA + VPA
  • vmalertmanager - VPA.

Operator is responsible for setting targetRef field with corresponding value for reference (deployment/statefulset).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

5 participants