Deploying kubernetes-dashboard via helm chart fails #578

kennyjwilli · 2019-05-31T22:33:53Z

The below code will fail to deploy with the follow error.

const cluster = new eks.Cluster(`k8s-cluster`, {
    vpcId: vpc.id,
    subnetIds: vpc.publicSubnetIds,
    deployDashboard: false
});
const k8sProvider = cluster.provider;

const k8sDashboardChart = new k8s.helm.v2.Chart("kubernetes-dashboard", {
    repo: "stable",
    chart: "kubernetes-dashboard",
    version: "1.5.2",
    namespace: "kube-system"
}, {providers: {kubernetes: k8sProvider}});

error:

  kubernetes:core:Service (kubernetes-dashboard):
    error: Plan apply failed: 2 errors occurred:
    	* Timeout occurred for 'kubernetes-dashboard'
    	* Service does not target any Pods. Selected Pods may not be ready, or field '.spec.selector' may not match labels on any Pods

The text was updated successfully, but these errors were encountered:

metral · 2019-06-04T17:56:43Z

Closing in favor of pulumi/pulumi-eks#142

lblackstone · 2019-06-18T20:56:12Z

This was failing with the following error:

  Warning  FailedCreate  26s (x15 over 108s)  replicaset-controller  Error creating: pods "kubernetes-dashboard-6b64f75d74-" is forbidden: error looking up service account default/kubernetes-dashboard: serviceaccount "kubernetes-dashboard" not found

It works as expected if you leave off the namespace parameter.

lblackstone · 2019-06-18T20:57:50Z

Related #217

kennyjwilli · 2019-06-19T16:24:25Z

@lblackstone I have just tried the code here without the namespace parameter set on line 56. pulumi up fails with the following message:

Diagnostics:
  pulumi:pulumi:Stack (k8s-dash-helm-test1):
    error: update failed
 
  kubernetes:core:Service (kubernetes-dashboard):
    error: Plan apply failed: 2 errors occurred:
    	* Timeout occurred for 'kubernetes-dashboard'
    	* Service does not target any Pods. Selected Pods may not be ready, or field '.spec.selector' may not match labels on any Pods
 
  kubernetes:extensions:Deployment (kubernetes-dashboard):
    error: Plan apply failed: 3 errors occurred:
    	* Timeout occurred for 'kubernetes-dashboard'
    	* Minimum number of Pods to consider the application live was not attained
    	* 1 Pods failed to schedule because: [Unschedulable] no nodes available to schedule pods

metral · 2019-06-19T16:31:43Z

@kennyjwilli Are you sure there are nodes available in the cluster? That error says there are none: [Unschedulable] no nodes available to schedule pods.

What does kubectl get nodes return?

Ah - I think I know what your problem is in that linked example: you create a role & instanceProfile and pass the instanceProfile to the NodeGroup but do not set the given role on the cluster. You must do that for the NodeGroup to join the cluster as shown here.

See instanceRole and instanceRoles in the ref docs.

Try setting instanceRole: role0 in your eksCluster definition, and see if this works for you with @lblackstone's namespace omission suggestion.

kennyjwilli · 2019-06-19T16:48:04Z

Oh wow, totally missed that. You're probably right. I have added the instanceRole: role0 to my Cluster and run pulumi up. Note that I am updating my existing cluster, not creating a new one. The update fails with the same message:

Diagnostics:
  pulumi:pulumi:Stack (k8s-dash-helm-test1):
    error: update failed
 
  kubernetes:core:Service (kubernetes-dashboard):
    error: Plan apply failed: 2 errors occurred:
    	* Timeout occurred for 'kubernetes-dashboard'
    	* Service does not target any Pods. Selected Pods may not be ready, or field '.spec.selector' may not match labels on any Pods
 
  kubernetes:extensions:Deployment (kubernetes-dashboard):
    error: Plan apply failed: 3 errors occurred:
    	* Timeout occurred for 'kubernetes-dashboard'
    	* Minimum number of Pods to consider the application live was not attained
    	* 1 Pods failed to schedule because: [Unschedulable] 0/2 nodes are available: 2 node(s) had taints that the pod didn't tolerate.

This seems like something you should be able to recover from without needing to recreate the whole cluster. Perhaps this deserves a separate ticket to fix?

Further, it seems far too easy to trip over yourself by forgetting something like this. Getting into a state like this does not seem desirable. Is there a way Pulumi could catch this much earlier?

metral · 2019-06-19T17:11:27Z

This seems like something you should be able to recover from without needing to recreate the whole cluster. Perhaps this deserves a separate ticket to fix?

The roles are used to update the aws-auth configmap, which is how the Nodes join the cluster. Given how the cluster does not change, I don't imagine this should trigger replacing a cluster - @lukehoban thoughts?

Further, it seems far too easy to trip over yourself by forgetting something like this. Getting into a state like this does not seem desirable. Is there a way Pulumi could catch this much earlier?

I'm sure we can build in checks that make sense, but we should first identify if this may either be an issue in your stack, or in pulumi/eks that we should track in a separate issue from this one.

kennyjwilli · 2019-06-19T17:20:38Z

It sounds like a pulumi/eks issue. The stack I am using is the one linked previously here. It should be easy to replicate the issue.

metral · 2019-06-19T17:27:59Z

@kennyjwilli I've opened up pulumi/pulumi-eks#174 to track the role update errors. Thanks for your feedback!

kennyjwilli · 2019-06-19T19:23:33Z

@metral Just tried this with a fresh stack and I am still getting the same error as before. Here is the code I used: https://gist.github.com/kennyjwilli/8ef9540117351cc69dc2b98c473ce99e. Here is the full output from pulumi up:

Previewing update (test1):

     Type                                     Name                                      Plan       
 +   pulumi:pulumi:Stack                      k8s-dash-helm-test1                       create     
 +   ├─ kubernetes:helm.sh:Chart              kubernetes-dashboard                      create     
 +   ├─ eks:index:Cluster                     k8s-cluster                               create     
 +   │  ├─ eks:index:ServiceRole              k8s-cluster-eksRole                       create     
 +   │  │  ├─ aws:iam:Role                    k8s-cluster-eksRole-role                  create     
 +   │  │  ├─ aws:iam:Role                    k8s-cluster-eksRole-role                  create     
 +   │  │  ├─ aws:iam:Role                                   k8s-cluster-eksRole-role                  create     
 +   │  │  ├─ aws:iam:Role                                   k8s-cluster-eksRole-role                  create     
 +   │  │  ├─ aws:iam:Role                                   k8s-cluster-eksRole-role                  create     
 +   │  │  ├─ aws:iam:Role                                   k8s-cluster-eksRole-role                  create     
 +   │  │  ├─ aws:iam:Role                                   k8s-cluster-eksRole-role                  create     
 +   │  ├─ kubernetes:core:ServiceAccount                    default/kubernetes-dashboard              create     
 +   │  │  └─ aws:iam:RolePolicyAttachment                   k8s-cluster-eksRole-4b490823                   create     
 +   │  ├─ eks:index:NodeGroup                               k8s-ng                                         create     
 +   │  ├─ kubernetes:rbac.authorization.k8s.io:Role         default/kubernetes-dashboard                   create     
 +   │  ├─ aws:ec2:SecurityGroup                             k8s-cluster-eksClusterSecurityGroup            create     
 +   │  ├─ aws:ec2:SecurityGroup                             k8s-cluster-eksClusterSecurityGroup            create     
 +   │  ├─ aws:ec2:SecurityGroup                             k8s-cluster-eksClusterSecurityGroup            create     
 +   │  │  └─ aws:cloudformation:Stack                       k8s-ng-nodes                                   create     
 +   │  ├─ aws:eks:Cluster                                   k8s-cluster-eksCluster                         create     
 +   │  ├─ pulumi:providers:kubernetes                       k8s-cluster-eks-k8s                            create     
 +   │  ├─ pulumi:providers:kubernetes                       k8s-cluster-provider                           create     
 +   │  ├─ pulumi-nodejs:dynamic:Resource                    k8s-cluster-vpc-cni                            create     
 +   │  ├─ aws:ec2:SecurityGroup                             k8s-cluster-nodeSecurityGroup                  create     
 +   │  ├─ kubernetes:core:ConfigMap                         k8s-cluster-nodeAccess                         create     
 +   │  ├─ aws:ec2:SecurityGroupRule                         k8s-cluster-eksNodeIngressRule                 create     
 +   │  ├─ aws:ec2:SecurityGroupRule                         k8s-cluster-eksExtApiServerClusterIngressRule  create     
 +   │  ├─ aws:ec2:SecurityGroupRule                         k8s-cluster-eksNodeInternetEgressRule          create     
 +   │  ├─ aws:ec2:SecurityGroupRule                         k8s-cluster-eksClusterIngressRule              create     
 +   │  └─ aws:ec2:SecurityGroupRule                         k8s-cluster-eksNodeClusterIngressRule          create     
 +   ├─ awsx:x:ec2:Vpc                                       default-vpc                                    create     
 +   │  ├─ awsx:x:ec2:Subnet                                 default-vpc-public-0                           create     
 +   │  ├─ awsx:x:ec2:Subnet                                 default-vpc-public-1                           create     
 >   │  ├─ aws:ec2:Subnet                                    default-vpc-public-0                           read       
 >   │  └─ aws:ec2:Subnet                                    default-vpc-public-1                           read       
 +   ├─ aws:iam:Role                                         example-role0                                  create     
 +   ├─ aws:iam:RolePolicyAttachment                         example-role0-policy-0                         create     
 +   ├─ aws:iam:InstanceProfile                              example-instanceProfile0                       create     
 +   ├─ aws:iam:RolePolicyAttachment                         example-role0-policy-1                         create     
 +   ├─ aws:iam:RolePolicyAttachment                         example-role0-policy-2                         create     
 >   └─ aws:ec2:Vpc                                          default-vpc                                    read       
 
Resources:
    + 38 to create

Do you want to perform this update? yes
Updating (test1):

     Type                                     Name                                      Status       
 +   pulumi:pulumi:Stack                      k8s-dash-helm-test1                       created      
 +   ├─ awsx:x:ec2:Vpc                        default-vpc                               created      
 +   │  ├─ awsx:x:ec2:Subnet                  default-vpc-public-0                      created      
 +   │  ├─ awsx:x:ec2:Subnet                  default-vpc-public-1                      created      
 >   │  ├─ aws:ec2:Subnet                     default-vpc-public-0                      read         
 >   │  └─ aws:ec2:Subnet                     default-vpc-public-1                      read         
 +   ├─ kubernetes:helm.sh:Chart              kubernetes-dashboard                      creating..   
 +   │  ├─ kubernetes:core:ServiceAccount                    default/kubernetes-dashboard              creating     
 +   │  ├─ kubernetes:core:Secret                            default/kubernetes-dashboard              creating     
 +   │  ├─ kubernetes:rbac.authorization.k8s.io:RoleBinding  default/kubernetes-dashboard              creating     
 +   │  ├─ kubernetes:core:Service                           kubernetes-dashboard                      creating     
 +   │  ├─ kubernetes:rbac.authorization.k8s.io:Role         default/kubernetes-dashboard              creating     
 +   │  └─ kubernetes:extensions:Deployment                  kubernetes-dashboard                      creating     
 +   ├─ eks:index:Cluster                                    k8s-cluster                               created      
 +   │  ├─ eks:index:NodeGroup                               k8s-ng                                    created      
 +   │  │  └─ pulumi-nodejs:dynamic:Resource                 k8s-ng-cfnStackName                            created      
 +   │  │  └─ aws:ec2:LaunchConfiguration                    k8s-ng-nodeLaunchConfiguration                 creating..   
 +   │  │  └─ aws:cloudformation:Stack                       k8s-ng-nodes                                   creating     
 +   │  └─ kubernetes:extensions:Deployment                  kubernetes-dashboard                           **creating failed**     1 error
 +   │  │  ├─ aws:iam:Role                                   k8s-cluster-eksRole-role                       created                 
 +   │  │  ├─ aws:iam:RolePolicyAttachment                   k8s-cluster-eksRole-4b490823                   created                 
 +   │  │  └─ aws:iam:RolePolicyAttachment                   k8s-cluster-eksRole-90eb1c99                   created                 
 +   │  ├─ aws:ec2:SecurityGroup                             k8s-cluster-eksClusterSecurityGroup            created                 
 +   │  ├─ aws:ec2:SecurityGroupRule                         k8s-cluster-eksClusterInternetEgressRule       created                 
 +   │  ├─ aws:eks:Cluster                                   k8s-cluster-eksCluster                         created                 
 +   │  ├─ pulumi:providers:kubernetes                       k8s-cluster-eks-k8s                            created                 
 +   │  ├─ pulumi:providers:kubernetes                       k8s-cluster-provider                           created                 
 +   │  ├─ aws:ec2:SecurityGroup                             k8s-cluster-nodeSecurityGroup                  created                 
 +   │  ├─ pulumi-nodejs:dynamic:Resource                    k8s-cluster-vpc-cni                            created                 
 +   │  ├─ kubernetes:core:ConfigMap                         k8s-cluster-nodeAccess                         created                 
 +   │  ├─ aws:ec2:SecurityGroupRule                         k8s-cluster-eksNodeIngressRule                 created                 
 +   │  ├─ aws:ec2:SecurityGroupRule                         k8s-cluster-eksNodeInternetEgressRule          created                 
 +   │  ├─ aws:ec2:SecurityGroupRule                         k8s-cluster-eksClusterIngressRule              created                 
 +   │  ├─ aws:ec2:SecurityGroupRule                         k8s-cluster-eksNodeClusterIngressRule          created                 
 +   │  └─ aws:ec2:SecurityGroupRule                         k8s-cluster-eksExtApiServerClusterIngressRule  created                 
 +   ├─ aws:iam:Role                                         example-role0                                  created                 
 >   ├─ aws:ec2:Vpc                                          default-vpc                                    read                    
 +   ├─ aws:iam:RolePolicyAttachment                         example-role0-policy-2                         created                 
 +   ├─ aws:iam:InstanceProfile                              example-instanceProfile0                       created                 
 +   ├─ aws:iam:RolePolicyAttachment                         example-role0-policy-1                         created                 
 +   └─ aws:iam:RolePolicyAttachment                         example-role0-policy-0                         created                 
 
Diagnostics:
  kubernetes:core:Service (kubernetes-dashboard):
    error: Plan apply failed: 2 errors occurred:
    	* Timeout occurred for 'kubernetes-dashboard'
    	* Service does not target any Pods. Selected Pods may not be ready, or field '.spec.selector' may not match labels on any Pods
 
  kubernetes:extensions:Deployment (kubernetes-dashboard):
    error: Plan apply failed: 3 errors occurred:
    	* Timeout occurred for 'kubernetes-dashboard'
    	* Minimum number of Pods to consider the application live was not attained
    	* 1 Pods failed to schedule because: [Unschedulable] 0/2 nodes are available: 2 node(s) had taints that the pod didn't tolerate.
 
  pulumi:pulumi:Stack (k8s-dash-helm-test1):
    error: update failed
 
Resources:
    + 36 created

Duration: 19m43s

metral · 2019-06-19T23:30:58Z

@kennyjwilli I've copied your notes over to the new issue.

Let's continue any discussions there. Thanks!

lukehoban assigned metral May 31, 2019

metral mentioned this issue Jun 4, 2019

Expose createDashboard pulumi/pulumi-eks#142

Closed

metral closed this as completed Jun 4, 2019

metral mentioned this issue Jun 19, 2019

Adding in a missing role to a cluster, after its already up with a NodeGroup with the associated instanceProfile, does not properly update pulumi/pulumi-eks#174

Closed

pulumi locked and limited conversation to collaborators Jun 19, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deploying kubernetes-dashboard via helm chart fails #578

Deploying kubernetes-dashboard via helm chart fails #578

kennyjwilli commented May 31, 2019

metral commented Jun 4, 2019

lblackstone commented Jun 18, 2019

lblackstone commented Jun 18, 2019

kennyjwilli commented Jun 19, 2019

metral commented Jun 19, 2019 •

edited

kennyjwilli commented Jun 19, 2019 •

edited

metral commented Jun 19, 2019

kennyjwilli commented Jun 19, 2019

metral commented Jun 19, 2019

kennyjwilli commented Jun 19, 2019

metral commented Jun 19, 2019

Deploying kubernetes-dashboard via helm chart fails #578

Deploying kubernetes-dashboard via helm chart fails #578

Comments

kennyjwilli commented May 31, 2019

metral commented Jun 4, 2019

lblackstone commented Jun 18, 2019

lblackstone commented Jun 18, 2019

kennyjwilli commented Jun 19, 2019

metral commented Jun 19, 2019 • edited

kennyjwilli commented Jun 19, 2019 • edited

metral commented Jun 19, 2019

kennyjwilli commented Jun 19, 2019

metral commented Jun 19, 2019

kennyjwilli commented Jun 19, 2019

metral commented Jun 19, 2019

metral commented Jun 19, 2019 •

edited

kennyjwilli commented Jun 19, 2019 •

edited