Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mservice.elbv2.k8s.aws mutating webhook blocks Helm rollbacks #3617

Closed
diversario opened this issue Mar 18, 2024 · 4 comments
Closed

mservice.elbv2.k8s.aws mutating webhook blocks Helm rollbacks #3617

diversario opened this issue Mar 18, 2024 · 4 comments
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@diversario
Copy link
Contributor

diversario commented Mar 18, 2024

Describe the bug
When using Helm to deploy a Service object (of type LoadBalancer) into a cluster, the installation and subsequent updates succeed, but a rollback with --force fails.

This is because the webhook only activates for CREATE and not for UPDATE operations - meaning, when Helm attempts to roll back and update the Service to its previous state, this previous state does not include the spec.loadBalancerClass field set by the webhook. This results in

[helm rollback - release: echoserver]  Error: failed to replace object:
 Service "echoserver-nlb" is invalid:
 spec.loadBalancerClass:
 Invalid value: "null": may not change once set

because (afaik) Helm sends the entire Service object to the API server due to the --force flag.

Steps to reproduce
Deploy any Helm chart with an LB service. I'll use AWS LBC chart as a placeholder for this example.

  1. Deploy the chart 2-3 times so there's a release history:
    helm upgrade --install --wait --wait-for-jobs --timeout 10m --atomic aws-lbc eks/aws-load-balancer-controller
  2. Run a rollback with --force:
    helm rollback aws-lbc 1 --force --wait --no-hooks --timeout 600s

Expected outcome
Not an error; I expect the webhook to modify the service objects appropriately, and the rollback to complete.

Environment

  • AWS Load Balancer controller version
    6.7.1
  • Kubernetes version
    1.28.3
  • Using EKS (yes/no), if so version?
    no

Additional Context:
Using Helm 3.10.3

@aravindsagar
Copy link

/kind bug

@k8s-ci-robot k8s-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Mar 18, 2024
@PowerSurj
Copy link

  • AWS Load Balancer controller version
    6.7.1

are you sure? do you mean Helm Chart version 2.7.1

@diversario
Copy link
Contributor Author

Yes, sorry. The AWS LBC version 2.7.1, chart version is the one that sets that appVersion.

@shraddhabang
Copy link
Collaborator

Delivered in v2.8.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests

5 participants