Cluster shutdown "events"

[bprashanth]

I'd like to delete certain external resources (loadbalancers) only when the cluster is being torn down. Not when the user deletes the pod, or the scheduler preempts it. Currently scripts like kube-down aren't kubernetes aware, they just start nuking cluster resources using gcloud, meaning the following happens:

$ ./cluster/kube-up.sh

$ cat <<EOF | kubectl create -f -
> apiVersion: v1
> kind: Service
> metadata:
>   name: nginxsvc
>   labels:
>     app: nginx
> spec:
>   type: LoadBalancer
>   ports:
>   - port: 8080
>     targetPort: 80
>     protocol: TCP
>     name: http
>   - port: 443
>     protocol: TCP
>     name: https
>   selector:
>     app: nginx
> EOF

$ kubectl get svc
NAME         CLUSTER_IP    EXTERNAL_IP      PORT(S)            SELECTOR    AGE
kubernetes   10.0.0.1      <none>           443/TCP            <none>      2m
nginxsvc     10.0.104.32   104.154.47.133   8080/TCP,443/TCP   app=nginx   1m

$ gcloud compute forwarding-rules list 
NAME                        REGION      IP_ADDRESS      IP_PROTOCOL TARGET
a5653a6897c4311e5bf0242010af0000 us-central1 104.154.47.133  TCP         us-central1/targetPools/a5653a6897c4311e5bf0242010af0000

$ ./cluster/kube-down.sh
Done

$ gcloud compute forwarding-rules list
NAME                        REGION      IP_ADDRESS      IP_PROTOCOL TARGET
a5653a6897c4311e5bf0242010af0000 us-central1 104.154.47.133  TCP         us-central1/targetPools/a5653a6897c4311e5bf0242010af0000

Having at least the following guarantees when there's a pending shutdown will make life a little easier:

1. Deny new resource create requests
2. Notify existing services/controllers/pods

@thockin

[thockin]

@bgrant0607 @brendandburns @smarterclayton

On Mon, Oct 26, 2015 at 6:06 PM, Prashanth B notifications@github.com
wrote:

I'd like to delete certain external resources (loadbalancers) only when
the cluster is being torn down. Not when the user deletes the pod, or the
scheduler preempts it. Currently scripts like kube-down aren't kubernetes
aware, they just start nuking cluster resources using gcloud, meaning the
following happens:

$ ./cluster/kube-up.sh

$ cat <<EOF | kubectl create -f -

apiVersion: v1
kind: Service
metadata:
name: nginxsvc
labels:
app: nginx
spec:
type: LoadBalancer
ports:

• port: 8080
  targetPort: 80
  protocol: TCP
  name: http
• port: 443
  protocol: TCP
  name: https
  selector:
  app: nginx
  EOF

$ kubectl get svcNAME CLUSTER_IP EXTERNAL_IP PORT(S) SELECTOR AGEkubernetes 10.0.0.1 443/TCP 2mnginxsvc 10.0.104.32 104.154.47.133 8080/TCP,443/TCP app=nginx 1m

$ gcloud compute forwarding-rules list NAME REGION IP_ADDRESS IP_PROTOCOL TARGETa5653a6897c4311e5bf0242010af0000 us-central1 104.154.47.133 TCP us-central1/targetPools/a5653a6897c4311e5bf0242010af0000

$ ./cluster/kube-down.shDone

$ gcloud compute forwarding-rules listNAME REGION IP_ADDRESS IP_PROTOCOL TARGETa5653a6897c4311e5bf0242010af0000 us-central1 104.154.47.133 TCP us-central1/targetPools/a5653a6897c4311e5bf0242010af0000

Having at least the following guarantees when there's a pending shutdown
will make life a little easier:

1. Deny new resource create requests
2. Notify existing services/controllers/pods

@thockin https://github.com/thockin

—
Reply to this email directly or view it on GitHub
#16337.

[bgrant0607]

Related #4630

cc @davidopp @fgrzadkowski @roberthbailey @zmerlynn

[bgrant0607]

It's the pod (LB plugin?) that creates the forwarding rule?

[bgrant0607]

Ideally there would be some Kubernetes resources that represented the underlying infrastructure resources, such that deleting the former would cause the latter to be deleted.

[bgrant0607]

See also #13515

[bprashanth]

My example was with a HEAD kube cluster, so it's the service controller (that runs as part of kube-controller-manager) that creates the forwarding rule for service Type=LoadBalancer, not the L7 LB plugin pod.

[zmerlynn]

Yes, GKE still really wants this. It's a pretty big wart.

[bprashanth]

Ideally there would be some Kubernetes resources that represented the underlying infrastructure resources, such that deleting the former would cause the latter to be deleted.

Just receiving a delete is insufficient, I need to differentiate that SIGTERM from a preemption in the case of a pod controller. A new field as suggested in some of the other issues might help, because I can use the grace period to check it in the apiserver.

[bprashanth]

Another possible solution would be to map the "Steps" in the simple cluster setup proposal (https://docs.google.com/document/d/1v68yStV2O6aHuRuT3AlWnbe6vkUCtzyC5I7Elhgik3o/edit?ts=561ee618) to runlevels persisted through the apiserver, and add a shutdown level just like init systems, hence the title.

[thockin]

Naively, deleting the Ingress object should trigger the controller Pod to
clean up the backend resources. But that requires some heuristic like
"pods are deleted last" and even then it has to be across namespaces.

On Tue, Oct 27, 2015 at 9:53 AM, Prashanth B notifications@github.com
wrote:

Ideally there would be some Kubernetes resources that represented the
underlying infrastructure resources, such that deleting the former would
cause the latter to be deleted.

Just receiving a delete is insufficient, I need to differentiate that
SIGTERM from a preemption in the case of a pod controller. A new field as
suggested in some of the other issues might help, because I can use the
grace period to check it in the apiserver.

—
Reply to this email directly or view it on GitHub
#16337 (comment)
.

[bgrant0607]

Re. pods deleted last: Another use case for finalizers. #3585

[davidopp]

Also vaguely related to #7459

[davidopp]

@bprashanth I'm marking this P2 but feel free to upgrade to P1 if you think it's higher priority.

[bgrant0607]

Also related to #10179

[bgrant0607]

Note that new resources cannot be created in namespaces that are being deleted:

kubernetes/plugin/pkg/admission/namespace/lifecycle/admission.go

Line 162 in e02b73f

return admission.NewForbidden(a, fmt.Errorf("unable to create new content in namespace %s because it is being terminated.", a.GetNamespace()))

And we already treat the default and kube-system namespaces specially (and openshift-infra in OS):

kubernetes/plugin/pkg/admission/namespace/lifecycle/admission.go

Line 53 in e02b73f

return NewLifecycle(client, sets.NewString(api.NamespaceDefault, api.NamespaceSystem))

[bgrant0607]

For my own reference: https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/cluster-loadbalancing/glbc

[fejta-bot]

Issues go stale after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[tonglil]

Should this issue be consolidated with something else, or kept open for the tear down of GCLB resources?

[bgrant0607]

/remove-lifecycle stale
/lifecycle frozen

cc @roberthbailey

[roberthbailey]

/cc @k4leung4