Openstack LBaaSv2 - Kubernetes fails to delete Load Balancer

[zioproto]

/sig openstack
Is this a BUG REPORT or FEATURE REQUEST?: BUG

Uncomment only one, leave it on its own line:

/kind bug

What happened:

I delete a service:

kubectl delete service my-nginx

Kubernetes is never able to delete the Loadbalancer from Openstack

$ kubectl get events
LASTSEEN   FIRSTSEEN   COUNT     NAME        KIND      SUBOBJECT   TYPE      REASON                       SOURCE               MESSAGE
3m         3m          1         my-nginx                   Service                                  Normal    CreatingLoadBalancer         service-controller       Creating load balancer
2m         2m          1         my-nginx                   Service                                  Normal    CreatedLoadBalancer          service-controller       Created load balancer
2s         1m          5         my-nginx                   Service                                  Normal    DeletingLoadBalancer         service-controller       Deleting load balancer
42s        1m          4         my-nginx                   Service                                  Warning   DeletingLoadBalancerFailed   service-controller       Error deleting load balancer (will retry): Resource not found

The problem is that there are no health monitors in Openstack, and Kubernetes refuses to go on and delete pools and loadbalancers and listeners.
On the neutron server I see DELETE requests without the health-monitor uuid, because the list of healthmonitors is empty.

What you expected to happen:
The loadbalancer is deleted.

In case the health monitors list is empty Kubernetes should go on and delete the rest of the Load Balancer.

How to reproduce it (as minimally and precisely as possible):

kubectl run my-nginx --image=nginx --replicas=2 --port=80
kubectl expose deployment my-nginx --port=80 --type=LoadBalancer
kubectl delete service my-nginx
kubectl get events

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.2", GitCommit:"477efc3cbe6a7effca06bd1452fa356e2201e1ee", GitTreeState:"clean", BuildDate:"2017-04-19T20:33:11Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.2", GitCommit:"477efc3cbe6a7effca06bd1452fa356e2201e1ee", GitTreeState:"clean", BuildDate:"2017-04-19T20:22:08Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration**:
  Openstack Newton
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
  https://github.com/zioproto/k8s-on-openstack
• Others:

[k8s-merge-robot]

@zioproto There are no sig labels on this issue. Please add a sig label by:
(1) mentioning a sig: @kubernetes/sig-<team-name>-misc
e.g., @kubernetes/sig-api-machinery-misc for API Machinery
(2) specifying the label manually: /sig <label>
e.g., /sig scalability for sig/scalability

Note: method (1) will trigger a notification to the team. You can find the team list here and label list here

[k8s-merge-robot]

@zioproto There are no sig labels on this issue. Please add a sig label by:
(1) mentioning a sig: @kubernetes/sig-<team-name>-misc
e.g., @kubernetes/sig-api-machinery-misc for API Machinery
(2) specifying the label manually: /sig <label>
e.g., /sig scalability for sig/scalability

Note: method (1) will trigger a notification to the team. You can find the team list here and label list here

[kargakis]

/sig openstack
/sig network
/remove-sig api-machinery

[kargakis]

/sig openstack
/sig network
/remove-sig api-machinery

[FengyunPan]

@zioproto From the "Error deleting load balancer (will retry): Resource not found" of event, I suspect that you have deleted the neutron resource of service's LB, is it?
Or give me more detail log of kube-controller-manager?

[FengyunPan]

@zioproto From the "Error deleting load balancer (will retry): Resource not found" of event, I suspect that you have deleted the neutron resource of service's LB, is it?
Or give me more detail log of kube-controller-manager?

[zioproto]

@FengyunPan I have not deleted anything in Openstack. The problem is that kubernetes never creates health monitors but then it tries to delete them forever.

this is the log you requested:

E0630 07:31:51.577371       1 servicecontroller.go:772] Failed to process service. Retrying in 5s: Resource not found
I0630 07:31:56.578075       1 servicecontroller.go:759] Service has been deleted default/my-nginx
E0630 07:31:59.069556       1 servicecontroller.go:772] Failed to process service. Retrying in 10s: Resource not found
I0630 07:32:09.069908       1 servicecontroller.go:759] Service has been deleted default/my-nginx
E0630 07:32:11.247040       1 servicecontroller.go:772] Failed to process service. Retrying in 20s: Resource not found```

[zioproto]

@FengyunPan I have not deleted anything in Openstack. The problem is that kubernetes never creates health monitors but then it tries to delete them forever.

this is the log you requested:

E0630 07:31:51.577371       1 servicecontroller.go:772] Failed to process service. Retrying in 5s: Resource not found
I0630 07:31:56.578075       1 servicecontroller.go:759] Service has been deleted default/my-nginx
E0630 07:31:59.069556       1 servicecontroller.go:772] Failed to process service. Retrying in 10s: Resource not found
I0630 07:32:09.069908       1 servicecontroller.go:759] Service has been deleted default/my-nginx
E0630 07:32:11.247040       1 servicecontroller.go:772] Failed to process service. Retrying in 20s: Resource not found```

[FengyunPan]

@zioproto Maybe your kubernetes version is incorrect. Can you try it on kubernetes v1.6.4+. My test as following:
-bash-4.3# kubectl get events | grep test
5m 5m 1 test Service Normal CreatingLoadBalancer service-controller Creating load balancer
4m 4m 1 test Service Normal CreatedLoadBalancer service-controller Created load balancer
15s 15s 1 test Service Normal DeletingLoadBalancer service-controller Deleting load balancer
9s 9s 1 test Service Normal DeletedLoadBalancer service-controller Deleted load balancer

[FengyunPan]

@zioproto Maybe your kubernetes version is incorrect. Can you try it on kubernetes v1.6.4+. My test as following:
-bash-4.3# kubectl get events | grep test
5m 5m 1 test Service Normal CreatingLoadBalancer service-controller Creating load balancer
4m 4m 1 test Service Normal CreatedLoadBalancer service-controller Created load balancer
15s 15s 1 test Service Normal DeletingLoadBalancer service-controller Deleting load balancer
9s 9s 1 test Service Normal DeletedLoadBalancer service-controller Deleted load balancer

[zioproto]

@FengyunPan I tested with v1.6.4 and I had the same problem as with version v1.6.2. I am running Openstack Newton. Are you running the same version of Openstack as well ? thank you

[zioproto]

@FengyunPan I tested with v1.6.4 and I had the same problem as with version v1.6.2. I am running Openstack Newton. Are you running the same version of Openstack as well ? thank you

[FengyunPan]

@zioproto Yes, it work fine.

[FengyunPan]

@zioproto Yes, it work fine.

[FengyunPan]

According to slack's discussion, @zioproto service's lb hasn't healthmonitor on his cluster, but controllermanager do a deletion for empty monitor.

Add pr #48336 to prevent the deletion of empty monitors.

[FengyunPan]

According to slack's discussion, @zioproto service's lb hasn't healthmonitor on his cluster, but controllermanager do a deletion for empty monitor.

Add pr #48336 to prevent the deletion of empty monitors.