Issue with configMap generation during EKS Managed Node Group implementation #293
Assignees
Labels
Milestone
Comments
|
@lukehoban tagging you per slack conversation |
|
We got a more complete isolated repro for this offline: // Bug repro showing that changing roleMappings on a cluster will remove the managed node group's role
// from the aws-config.yaml and cause the nodes to go offline.
// Run pulumi up (works), switch the comment from line 31 to 32 and run pulumi up again (nodes will go offline after a few minutes).
import * as aws from '@pulumi/aws';
import * as eks from '@pulumi/eks';
// IAM role for the node group.
const ngRole = createRole('example-ng-role1');
// IAM roles for role mapping
const role1 = createRole('example-role2');
const role2 = createRole('example-role3');
const roleMapping1: eks.RoleMapping = {
roleArn: role1.arn,
username: 'roleMapping1',
groups: ['system:masters']
};
const roleMapping2: eks.RoleMapping = {
roleArn: role2.arn,
username: 'roleMapping2',
groups: ['system:masters']
};
// Create an EKS cluster.
const cluster = new eks.Cluster('example-managed-nodegroups', {
skipDefaultNodeGroup: true,
deployDashboard: false,
// roleMappings: [roleMapping1, roleMapping2]
roleMappings: [roleMapping1]
});
// Export the cluster's kubeconfig.
export const kubeconfig = cluster.kubeconfig;
// Create a simple AWS managed node group using a cluster as input.
eks.createManagedNodeGroup(
'example-managed-ng1',
{
cluster: cluster,
nodeGroupName: 'aws-managed-ng1',
nodeRoleArn: ngRole.arn,
scalingConfig: {
desiredSize: 1,
minSize: 1,
maxSize: 1
}
},
cluster
);
// Creates a role and attches the EKS worker node IAM managed policies
function createRole(name: string): aws.iam.Role {
const managedPolicyArns: string[] = [
'arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy',
'arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy',
'arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly'
];
const role = new aws.iam.Role(name, {
assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal({
Service: 'ec2.amazonaws.com'
})
});
let counter = 0;
managedPolicyArns.forEach(policy => {
new aws.iam.RolePolicyAttachment(`${name}-policy-${counter++}`, {
policyArn: policy,
role: role
});
});
return role;
}This it the config map after the first run: This is the configmap after the second run: You can see that this role map disappeared: - groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::048653383995:role/example-ng-role1-902ed97
username: system:node:{{EC2PrivateDNSName}} |
|
A couple more notes from offline discussion:
And
|
metral
added
bug
customer/feedback
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Problem description
Deployed EKS Managed Node Groups to existing EKS cluster leveraing the new support in 0.18.18 of the eks library. As a result of this, the
aws-authconfigmap was updated with an auto-generated, managed node role. Custom node roles created in another stack and passed to the nodeGroup constructor APPEARED to not be added to theaws-authconfigMap. End result was all nodes going into a 'NotReady' state. The configMap appeared to contain only the auto-generated role (based on the diff), however, the custom role must have been still been present somehow and the nodes began using the AWS-generated role (rather than the custom roles passed to the constructor).Mitigation - Manually edited the configMap to add the custom roles and the nodes immediately went back to 'Ready' status.
Errors & Logs
Condensed version of change log
Key snippet showing the configmap change
Affected product version(s)
pulumi/eks 0.18.18
Reproducing the issue
Have not attempted repro yet. However, this same issue happened in all 4 environments updated with the same code in the same manner.
Cluster code
Suggestions for a fix
Manually fixed by truing up the configMap. Appears there is a conflict between defining roles and AWS generating a node role.
The text was updated successfully, but these errors were encountered: