Support: Issue with using secret-based credentials

[0987363]

/kind bug

What happened?

• 3 EC2 servers, with 3 K3S master nodes deployed on them.
• Pending when creating a volume.

How to reproduce it (as minimally and precisely as possible)?

1. Create IAM user, attach permissions: AmazonEBSCSIDriverPolicy
2. Create access credentials.
3. helm install aws-ebs-csi-driver with config：

helm upgrade --install aws-ebs-csi-driver --namespace kube-system -f aws-iam-csi-driver-config.yml aws-ebs-csi-driver/aws-ebs-csi-driver

# aws-iam-csi-driver-config.yml
awsAccessSecret:
  name: aws-secret
  keyId: REDACTED
  accessKey: REDACTED
controller:
  region: ap-northeast-1
  sdkDebugLog: true
  logLevel: 7
node:
  logLevel: 7

4. Use example: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/tree/master/examples/kubernetes/dynamic-provisioning
5. ebs-csi-controller keeps reporting errors:
   E0428 10:12:28.020560 1 driver.go:108] "GRPC error" err="rpc error: code = Internal desc = Could not create volume \"pvc-dcfaffc6-28de-4dab-aae8-4f5e1e71287e\": could not create volume in EC2: operation error EC2: CreateVolume, get identity: get credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, http response error StatusCode: 404, request to EC2 IMDS failed

Anything else we need to know?:

• I tried setting the EC2 hop limit to 2, but the problem persists.
• I tried setting the config:hostNetwork to true, but the problem persists.

awsAccessSecret:
  name: aws-secret
  keyId: REDACTED
  accessKey: REDACTED
controller:
  region: ap-northeast-1
  sdkDebugLog: true
  logLevel: 7
node:
  logLevel: 7
  hostNetwork: true

Environment

• Kubernetes version (use kubectl version):

Client Version: v1.32.3+k3s1
Kustomize Version: v5.5.0
Server Version: v1.32.3+k3s1

• Driver version:
  v1.42.0

[ConnorJC3]

@0987363 Hi, you accidentally exposed your IAM credentials in the issue. I have edited the issue and deleted your credentials, and removed the edit history, but they were visible for 5-6 hours before I did so.

I recommend running through these steps to secure your account in case any attacker noticed the credentials before I removed them from your issue: https://repost.aws/knowledge-center/potential-account-compromise

[0987363]

@0987363 Hi, you accidentally exposed your IAM credentials in the issue. I have edited the issue and deleted your credentials, and removed the edit history, but they were visible for 5-6 hours before I did so.

I recommend running through these steps to secure your account in case any attacker noticed the credentials before I removed them from your issue: https://repost.aws/knowledge-center/potential-account-compromise

Thank you very much, I have refreshed the key.

[0987363]

Supplement:

1. Attach a IAM role with permissions: AmazonEBSCSIDriverPolicy to EC2.
2. Helm installation without any extra configuration files.
3. Working properly.

[ConnorJC3]

/close

Given that it's now working properly for you, I'm going to close out this issue, but please submit a new issue or reopen this one if you need further assistance.

[k8s-ci-robot]

@ConnorJC3: Closing this issue.

In response to this:

/close

Given that it's now working properly for you, I'm going to close out this issue, but please submit a new issue or reopen this one if you need further assistance.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[0987363]

/close

Given that it's now working properly for you, I'm going to close out this issue, but please submit a new issue or reopen this one if you need further assistance.

Sorry, unable to completely resolve this issue, need reopen it.

The installation guide recommends using the AWS secret method instead of the IAM Instance Profile method.

The IAM Instance Profile is just a supplementary test.

[ConnorJC3]

/reopen

So it works for you with an IAM profile, but not with a secret? With the new installation, what happens with the secret, can you post the latest error logs after rotating the secret?

[k8s-ci-robot]

@ConnorJC3: Reopened this issue.

In response to this:

/reopen

So it works for you with an IAM profile, but not with a secret? With the new installation, what happens with the secret, can you post the latest error logs after rotating the secret?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ConnorJC3]

/retitle Support: Issue with using secret-based credentials
/kind support