delete ingress object hangs

[dfang]

What happened:

delete ingress object hangs even with --force and --grace-period=0

$ k delete ingress ingress-2048 -n game-2048  --force --grace-period=0
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
ingress.extensions "ingress-2048" force deleted
.........

no events in game-2048 namespace

$ k get event -n game-2048
No resources found in game-2048 namespace.

What you expected to happen:
deleted

How to reproduce it (as minimally and precisely as possible):
it might be hard to reproduce

Anything else we need to know?:

on a eks cluster, the ingress is created from this doc,
after delete aws load balancer controller, i can't delete ingress object

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T18:49:28Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.9-eks-a84824", GitCommit:"a84824a1c7b79853d0c57143ea42c9de1788dbba", GitTreeState:"clean", BuildDate:"2020-08-05T01:08:51Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Network plugin and version (if this is a network-related bug):
• Others:

[k8s-ci-robot]

@dfang: There are no sig labels on this issue. Please add an appropriate label by using one of the following commands:

• /sig <group-name>
• /wg <group-name>
• /committee <group-name>

Please see the group list for a listing of the SIGs, working groups, and committees available.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

@dfang: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[liggitt]

Does the load balancer add a finalizer to the ingress object? (you can check with kubectl get ingress $name -o yaml). If so, the object will not be removed until that finalizer is removed.

You can make delete return without waiting by adding -wait false, but for the object to actually be deleted the finalizer must be removed.

[dfang]

yes, there is a finalizer, after read some docs and know what exactly finalizer is.

λ  k get ingress ingress-2048 -n game-2048 -o yaml | grep -A 5 finalizers
  finalizers:
  - ingress.k8s.aws/resources
  generation: 4
  name: ingress-2048
  namespace: game-2048
  resourceVersion: "43226"

i remember i deleted the alb resource in aws management console before, so it always failed. after kubectl edit and remove finalizers, i can delete it

so i'll close it.

thanks for your help !

[sabirmmoglad]

I used:

kubectl edit ingress my-ingress

and then I removed the finalizers but still the ingress is not deleted. This is in AWS EKS.

...
 managedFields:
  - apiVersion: networking.k8s.io/v1beta1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        f:loadBalancer:
          f:ingress: {}
    manager: controller
    operation: Update
...

Do I need to do anything else for the changes to be applied?

[sachincool]

Facing the same issue @sabirmmoglad, were you able to resolve this?

Kubernetes v1.19.7

I was playing around with 2048 game and trying aws ALB and kubernetes-nginx-ingress controllers, and achieved the same state somehow.
@dfang I researched about finalizers, is there an api for ingress where I can upload the empty finalize array ingress object so K8s can remove it?. during my research I could find NS finalize api in stackoverflow.

[sachincool]

update:
I was able to delete the resource, by reinstalling alb-ingress-controller again and then my k8 controller was able to successfully remove the ingress object. then I removed the alb-ingress-controlled again. 😄

[jmmen1]

I have also ended up in this situation, using the aws-load-balancer-controller. Sadly, deleting and then recreating the controller still causes kubectl to hang when deleting the ingress.

Anyone else have more ideas for me to try?

[sachincool]

@jmmen1 try

1. delete the ns
2. Delete the ing object
3. apply the controller again
4. Wait for a few seconds/minutes
5. check if ing object has been removed or not.

[brenwhyte]

I'm getting this issue too, AWS EKS 1.18
Can we reopen this one? What other info do you need?

[jmmen1]

@jmmen1 try

1. delete the ns
2. Delete the ing object
3. apply the controller again
4. Wait for a few seconds/minutes
5. check if ing object has been removed or not.

@sachincool - I am not able to delete the namespace, since I am using the default namespace.

I deleted the AWS LB controller deployment, then attempted to delete the ingress objects, but it's still hanging when I attempt to delete the ingresses.

[cmsmith7]

I believe this issue is the same as the issue highlighted here ..
kubernetes-sigs/aws-load-balancer-controller#1629

[narendra-cloudthat]

Use following command to patch and delete the ingress-2048

kubectl patch ingress $Ingressname -n $namespace -p '{"metadata":{"finalizers":[]}}' --type=merge

[malcolm061990]

Use following command to patch and delete the ingress-2048

kubectl patch ingress $Ingressname -n $namespace > -p '{"metadata":{"finalizers":[]}}' --type=merge

The correct command is without stdout sign:

kubectl patch ingress $Ingressname -n $namespace -p '{"metadata":{"finalizers":[]}}' --type=merge

[Badawekoo]

Hello, I faced the same issue, deleting the below sections from the ingress resource made me able to delete it.
Make sure that aws-load-balancer-controller and aws-webhook installed successfully in the cluster

  finalizers:
  - ingress.k8s.aws/resources
-------------------------------
        f:metadata:
        f:finalizers:
          .: {}
          v:"ingress.k8s.aws/resources": {}

[adecchi-2inno]

HI,
I am I am facing the same issue. The ALB was deleted, the ingress object is still alive:

$ kubectl patch ingress argocd-server -n argocd -p '{"metadata":{"finalizers":[]}}' --type=merge

Here the output of the above command:

Error from server (InternalError): Internal error occurred: failed calling webhook "vingress.elbv2.k8s.aws": the server could not find the requested resource

The Logs:

{"level":"debug","ts":44444444,"logger":"secrets-manager","msg":"Monitoring secrets","groupID":"argocd/argocd-server","secrets":null} {"level":"error","ts":44444444,"logger":"controller-runtime.manager.controller.ingress","msg":"Reconciler error","name":"argocd-server","namespace":"argocd","error":"Internal error occurred: failed calling webhook \"vingress.elbv2.k8s.aws\": the server could not find the requested resource"}

Someone, can give any tips ?

[igor-pinchuk]

It fails on the following error because of admission controller
Error from server (InternalError): Internal error occurred: failed calling webhook "vingress.elbv2.k8s.aws": the server could not find the requested resource

run:
kubectl delete ValidatingWebhookConfiguration aws-load-balancer-webhook

and then try to delete the ingress.

[dokuboyejo]

Indeed @igor-pinchuk would prevent finalizer patch

[bkalcho]

kubectl delete ValidatingWebhookConfiguration aws-load-balancer-webhook

tnx. This is life saver.

[leopard627]

Use following command to patch and delete the ingress-2048
kubectl patch ingress $Ingressname -n $namespace > -p '{"metadata":{"finalizers":[]}}' --type=merge

The correct command is without stdout sign:

kubectl patch ingress $Ingressname -n $namespace -p '{"metadata":{"finalizers":[]}}' --type=merge

thank u so much!!!

[parasinsync]

kubectl patch ingress $Ingressname -n $namespace -p '{"metadata":{"finalizers":[]}}' --type=merge

This worked for me too.. I was struggling with this dangling ingress.. Thanks..!!

[malshan1991]

so the correct sequence would be

1. delete ingres and send the process to background
2. run the patch