Configuring external s3 blobstore with instance profile, missing required credential error returns in api

[eyett]

What is this issue about?

Configured the bosh director with instance iam profile, removing AWS access key and secret key. aws-iam-instance-profile
Configured IAM roles for both director and deployed vms, with pass role on the director role.
Deployed cf deployment, but job fails in api.

What version of cf-deployment are you using?

v30.6.0

bosh deploy... command:

bosh int cf-deployment.yml \
-o operations/use-external-blobstore.yml \
-o operations/use-s3-blobstore.yml \
-o logsearch-boshrelease/deployment/operations/cf/add-firehose-to-syslog-uaa-clients.yml \
-o logsearch-boshrelease/deployment/operations/cf/add-kibana-uaa-clients.yml \
-o prometheus-boshrelease/manifests/operators/cf/add-grafana-uaa-clients.yml \
-o prometheus-boshrelease/manifests/operators/cf/add-prometheus-uaa-clients.yml \
-o operations/override-app-domains.yml \
-o operations/scale-database-cluster.yml \
-o operations/use-compiled-releases.yml

Manually deleted blobstore credential from use-s3-blobstore.yml

 git diff operations/use-s3-blobstore.yml
 provider: AWS
 -  aws_access_key_id: ((blobstore_access_key_id))
 -  aws_secret_access_key: ((blobstore_secret_access_key))
 region: ((aws_region))`

Please provide output that helps describe the issue:

 ./cloud_controller_ng/cloud_controller_ng.stdout.log:Missing required arguments: aws_access_key_id, aws_secret_access_key
 ./cloud_controller_ng/cloud_controller_ng.log:{"timestamp":"2023-11-04T22:56:54.923519429Z","message":"Error with blobstore: ArgumentError - Missing required arguments: aws_access_ke_id, aws_secret_access_key","log_level":"error","source":"cc.error_handling_client","data":{},"thread_id":91280,"fiber_id":91300,"process_id":6,"file":"/var/vcap/data/packages/cloud_controller_ng/5bd60a698d8848da30efb047a9c9e6be3526c691/cloud_controller_ng/lib/cloud_controller/blobstore/fog/error_handling_client.rb","lineno":67,"method":"rescue in error_handling"}

It seems that the role is passed and attached to the deployed cf vms properly, and it should be able to access s3 blobstore, if it's not checking for the removed accesskey and secretkey credentials.
Is there anyone running cf with instance profile configuration?

What IaaS is this issue occurring on?

AWS

[jochenehret]

Hi eyett,

if you want to use an external S3 blobstore for the cloud controller with the use-s3-blobstore.yml ops file, you must provide an AWS access key and secret key. Otherwise you will receive the error shown above. The credentials are used to configure the Fog AWS client: https://github.com/fog/fog-aws

If you really want to use AWS IAM Instance Profiles, you can follow these instructions: https://docs.cloudfoundry.org/deploying/common/cc-blobstore-config.html#fog-aws-iam
You have to create your own ops files for this scenario, however.

Best Regards,

Jochen.

[eyett]

@jochenehret. Thank you for the information.
I thought the global configuration from the director should be applied over and never thought it could be configurable additionally. This shouldn't be an issue then if there's a way to configure..
I'm trying further with the Fog with AWS IAM Instance Profiles documentation so far to see if I can make all cf deployed vms to be successfully run.
Appreciate again for the sharing. Good day!