Kubernetes installation stuck in IN_PROGRESS

[uatuko]

Hi,

I've been trying to install kubernetes (1.0.3-1.9.7 & 1.0.2-1.9.6) but the installation gets stuck. Can't access kubernetes dashboard either (similar error to #77).

$ dcos kubernetes plan status deploy
deploy (serial strategy) (IN_PROGRESS)
├─ etcd (serial strategy) (COMPLETE)
│  └─ etcd-0:[peer] (COMPLETE)
├─ apiserver (parallel strategy) (COMPLETE)
│  └─ kube-apiserver-0:[instance] (COMPLETE)
├─ kubernetes-api-proxy (parallel strategy) (COMPLETE)
│  └─ kubernetes-api-proxy-0:[install] (COMPLETE)
├─ controller-manager (parallel strategy) (COMPLETE)
│  └─ kube-controller-manager-0:[instance] (COMPLETE)
├─ scheduler (parallel strategy) (COMPLETE)
│  └─ kube-scheduler-0:[instance] (COMPLETE)
├─ node (parallel strategy) (IN_PROGRESS)
│  ├─ kube-node-0:[kube-proxy] (PREPARED)
│  ├─ kube-node-0:[coredns] (PREPARED)
│  └─ kube-node-0:[kubelet] (PREPARED)
├─ public-node (parallel strategy) (COMPLETE)
└─ mandatory-addons (serial strategy) (PENDING)
   ├─ mandatory-addons-0:[kube-dns] (PENDING)
   ├─ mandatory-addons-0:[metrics-server] (PENDING)
   ├─ mandatory-addons-0:[dashboard] (PENDING)
   └─ mandatory-addons-0:[ark] (PENDING)

Any ideas why this is?

[hectorj2f]

@uditha-atukorala can you try to search for any FAIL offer in the output of https://<CLUSTER_IP>/service/kubernetes/v1/debug/offers ? I believe it is related to lack of enough available resources

[uatuko]

@hectorj2f, thanks for the pointers. It does look like it is not having enough resources.

e.g.

FAIL(ResourceEvaluationStage): Offer failed to satisfy: name: cpus, value: type: SCALAR scalar { value: 3.0 }, role: kubernetes-role, pre-reserved-role: *, principal: kubernetes-principal with resourceId: Optional.empty

FAIL(VolumeEvaluationStage): Offer failed to satisfy: name: disk, value: type: SCALAR scalar { value: 10240.0 }, role: kubernetes-role, pre-reserved-role: *, principal: kubernetes-principal, type: 'ROOT', container-path: 'var' with resourceId: Optional.empty

I'll take a look.

[hectorj2f]

No problem. Closing this issue!

[ravinani02]

Hey @hectorj2f ,
I came across the same issue, Could you please let me know the recommended resources that need to be configured for kube-node such that the kubernetes dashboard works. I tried to scale my resource both horizontally and vertically but still didn't find any luck.I'm trying to install using dcos-vagrant on my local machine.

Thanks in advance

[ravinani02]

Here are the logs from above mentioned url,

FAIL(VolumeEvaluationStage): Offer failed to satisfy: name: disk, value: type: SCALAR scalar { value: 10240.0 }, role: kubernetes-role, pre-reserved-role: *, principal: kubernetes-principal, type: 'ROOT', container-path: 'var' with resourceId: Optional.empty

FAIL(ResourceEvaluationStage): Offer failed to satisfy: name: cpus, value: type: SCALAR scalar { value: 0.1 }, role: kubernetes-role, pre-reserved-role: *, principal: kubernetes-principal with resourceId: Optional.empty

FAIL(ResourceEvaluationStage): Offer failed to satisfy: name: mem, value: type: SCALAR scalar { value: 32.0 }, role: kubernetes-role, pre-reserved-role: *, principal: kubernetes-principal with resourceId: Optional.empty

FAIL(ResourceEvaluationStage): Offer failed to satisfy: name: cpus, value: type: SCALAR scalar { value: 0.1 }, role: kubernetes-role, pre-reserved-role: *, principal: kubernetes-principal with resourceId: Optional.empty

FAIL(ResourceEvaluationStage): Offer failed to satisfy: name: mem, value: type: SCALAR scalar { value: 32.0 }, role: kubernetes-role, pre-reserved-role: *, principal: kubernetes-principal with resourceId: Optional.empty

FAIL(ResourceEvaluationStage): Offer failed to satisfy: name: disk, value: type: SCALAR scalar { value: 256.0 }, role: kubernetes-role, pre-reserved-role: *, principal: kubernetes-principal with resourceId: Optional.empty

PASS(LaunchEvaluationStage): Added launch information to offer requirement

FAIL(ResourceEvaluationStage): Offer failed to satisfy: name: cpus, value: type: SCALAR scalar { value: 0.1 }, role: kubernetes-role, pre-reserved-role: *, principal: kubernetes-principal with resourceId: Optional.empty

FAIL(ResourceEvaluationStage): Offer failed to satisfy: name: mem, value: type: SCALAR scalar { value: 512.0 }, role: kubernetes-role, pre-reserved-role: *, principal: kubernetes-principal with resourceId: Optional.empty

And also dcos kubernetes plan status deploy output:

deploy (serial strategy) (IN_PROGRESS)
├─ etcd (serial strategy) (COMPLETE)
│ └─ etcd-0:[peer] (COMPLETE)
├─ apiserver (parallel strategy) (COMPLETE)
│ └─ kube-apiserver-0:[instance] (COMPLETE)
├─ mandatory-addons (serial strategy) (COMPLETE)
│ ├─ mandatory-addons-0:[additional-cluster-role-bindings] (COMPLETE)
│ ├─ mandatory-addons-0:[kube-dns] (COMPLETE)
│ ├─ mandatory-addons-0:[metrics-server] (COMPLETE)
│ ├─ mandatory-addons-0:[dashboard] (COMPLETE)
│ └─ mandatory-addons-0:[ark] (COMPLETE)
├─ kubernetes-api-proxy (parallel strategy) (COMPLETE)
│ └─ kubernetes-api-proxy-0:[install] (COMPLETE)
├─ controller-manager (parallel strategy) (COMPLETE)
│ └─ kube-controller-manager-0:[instance] (COMPLETE)
├─ scheduler (parallel strategy) (COMPLETE)
│ └─ kube-scheduler-0:[instance] (COMPLETE)
├─ node (parallel strategy) (IN_PROGRESS)
│ └─ kube-node-0:[kube-proxy, coredns, kubelet] (PREPARED)
└─ public-node (parallel strategy) (COMPLETE)

[ravinani02]

@uditha-atukorala May be you can guide me on this since you already have a solution,

[hectorj2f]

@ravinani02 Did you have a look at the docs https://github.com/mesosphere/dcos-kubernetes/blob/master/docs/package/install.md#prerequisites ? Based on the FAIL logs, you shared here, it seems there aren't enough available resources in your cluster.

[uatuko]

@ravinani02, my issue was not having enough agent node resources. I ended up upgrading my nodes but you can also try scaling the k8s cluster down.