Addon Manager Infinite Termination Loop

[rossedman]

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Not sure. I think its a bug unless I am misunderstanding something.

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm): Kubernetes v1.6.6

What happened: When I try to put new addons in the addon directory none of them work. They enter an infinite termination loop. Even attempting to add a simple NGINX deployment fails.

What you expected to happen: I expect the addons to spin up and complete deployment and have the addon manager reconcile them frequently. When creating new clusters I want to run Ansible to drop files in the addons directory to get storage classes, monitoring, etc configured. Whenever I try this on minikube it works fine.

How to reproduce it (as minimally and precisely as possible): Put anything in the /etc/kubernetes/addons folder. Add this metadata to the files:

apiVersion: v1
kind: ReplicationController
metadata:
  name: nginx
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  replicas: 2
  selector:
    app: nginx
  template:
    metadata:
      name: nginx
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80

Example output:

nginx-v98bd   0/1       Pending   0         0s
nginx-vv77t   0/1       Pending   0         0s
nginx-v98bd   0/1       Pending   0         0s
nginx-vv77t   0/1       Pending   0         0s
nginx-v98bd   0/1       ContainerCreating   0         1s
nginx-vv77t   0/1       ContainerCreating   0         1s
nginx-v98bd   0/1       Terminating   0         5s
nginx-vv77t   0/1       Terminating   0         5s
nginx-v98bd   0/1       Terminating   0         5s
nginx-vv77t   0/1       Terminating   0         5s
nginx-v98bd   0/1       Terminating   0         7s
nginx-v98bd   0/1       Terminating   0         7s
nginx-vv77t   0/1       Terminating   0         8s
nginx-vv77t   0/1       Terminating   0         8s
nginx-rqj2t   0/1       Pending   0         1s
nginx-ph6wm   0/1       Pending   0         1s
nginx-rqj2t   0/1       Pending   0         1s
nginx-ph6wm   0/1       Pending   0         1s
nginx-rqj2t   0/1       ContainerCreating   0         1s
nginx-ph6wm   0/1       ContainerCreating   0         1s
nginx-ph6wm   0/1       Terminating   0         5s
nginx-rqj2t   0/1       Terminating   0         5s
nginx-rqj2t   0/1       Terminating   0         5s
nginx-ph6wm   0/1       Terminating   0         5s
nginx-rqj2t   0/1       Terminating   0         7s
nginx-rqj2t   0/1       Terminating   0         8s
nginx-ph6wm   0/1       Terminating   0         8s
nginx-ph6wm   0/1       Terminating   0         8s
nginx-vcwx2   0/1       Pending   0         0s
nginx-frm9f   0/1       Pending   0         0s
nginx-vcwx2   0/1       Pending   0         0s
nginx-frm9f   0/1       Pending   0         0s
nginx-vcwx2   0/1       ContainerCreating   0         0s
nginx-frm9f   0/1       ContainerCreating   0         0s
nginx-frm9f   1/1       Running   0         2s
nginx-vcwx2   1/1       Running   0         3s

[rossedman]

Just tested this on Kubernetes 1.6.2 that was deployed with ACS Engine and the same problem exists.

[colemickens]

Are you doing this on an HA cluster? Are you sure the addons dir is in sync on all the masters?

[rossedman]

@colemickens I am. I am guessing they are not in sync which makes more sense. I will try to distribute files to all masters at once and see if that fixes it. Wasn't sure if addon manager distributed changes to all clusters but after reading through the script its more basic then I thought.

[rossedman]

@colemickens Tested on a cluster with one master and had no problems. Thanks