NoCredentialProviders: no valid providers in chain

[maauso]

Hi guys,

I am trying to run Kiam in EKS, we are using version 2.8, but we have the same error using version 2.7 in server side.

I created kiam-ubuntu-test role, with s3 admin rights, and Trust Relationship

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::017491163532:role/{node}"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

When pod trys to run aws s3 ls s3://something we have the next error:

Kiam Server side:

{"level":"info","msg":"found role","pod.iam.role":"","pod.ip":"10.250.171.51","time":"2018-06-20T13:37:38Z"}{"level":"info","msg":"requesting credentials","pod.iam.role":"kiam-ubuntu-test","time":"2018-06-20T13:37:38Z"}
{"level":"error","msg":"error requesting credentials: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors","pod.iam.role":"kiam-ubuntu-test","time":"2018-06-20T13:37:38Z"}{"level":"error","msg":"error requesting credentials: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors","pod.iam.role":"kiam-ubuntu-test","time":"2018-06-20T13:37:38Z"}

Kiam agent side

{"addr":"10.250.170.125:38056","headers":{},"level":"info","method":"GET","msg":"processed request","path":"/latest/meta-data/iam/security-credentials/","status":200,"time":"2018-06-20T13:3
7:38Z"}{"level":"warning","msg":"error getting credentials for role: rpc error: code = Unknown desc = NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see a
ws.Config.CredentialsChainVerboseErrors","pod.iam.role":"kiam-ubuntu-test","time":"2018-06-20T13:37:38Z"}

Take in consideration that :
1- EKS use amazon-vpc-cni
2.- Kiam server/agent are runing in worker nodes not in master
3.- We created the EKS cluster and Kiam services following the official documentation.

[ewbankkit]

@maauso Are you ensuring that the server is running on separate nodes from the agent (e.g. via a nodeSelector or taints and tolerations)?
If both the server and agents are running on all worker nodes then you get into the situation where the agent proxies to the server which tries to contact the node's underlying metadata endpoint but this is redirected (via iptables) to the agent and no credentials can ever be obtained. As I remember (because I got into that situation initially) that NoCredentialProviders error is how this behavior exhibits.

[maauso]

@ewbankkit really?, so no, server and agent are running in the same nodes, because in EKS if I try to split them I found #94 problems.

I opened a issue with aws and they are not sure if you can have more that one goup nodes, so nice.

Thanks for you reply

[ewbankkit]

@maauso I was just reading #94. I think of any theoretical reason why you can't have > 1 node group with EKS but I haven't tried that yet - Hopefully I can get to testing soon.

[maauso]

@ewantg you are right, the problem is related to have both parts in the same node, so I close this issue. Thanks for all