IAM Role for Service Account(IRSA) support for models stored in S3

[surajkota]

/kind feature

Currently, AWS users need to create a secret with long term/static IAM credentials for downloading models stored in S3. Security best practice is to use IAM role for service account(IRSA) which enables automatic credential rotation and fine grained access control

Describe the solution you'd like
Support annotating IAM role in a service account which can be used by storage initializer to download models from S3.

Anything else you would like to add:
It will be super helpful to know the customers/users interested in this feature for appropriate prioritization. Please upvote or comment on the issue if you are interested in this feature

related: awslabs/kubeflow-manifests#44

[yuzisun]

for reference https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html

[matty-rose]

Hi, this functionality currently exists but just isn't documented in the KServe guide. However there are some quirks required in order to get kserve to parse the annotations from the secret - an example and reasoning can be found in my comment here. I'm happy to contribute updates to the documentation, but wanted to check with @yuzisun whether it would be worth adding functionality to this code that makes this easier for end users.

A proposed solution could be to explicitly check for the eks role annotation on the service account and then either

1. also attempt to parse the relevant s3 annotations from the ServiceAccount instead of requiring an attached secret, so that users of IRSA functionality would not need to create the aws-secret resource
2. OR still require an attached secret for the s3 annotations, but add the required env variables to the storage-init container even if the AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY data fields are not present

If either of you @yuzisun @surajkota have any thoughts on this approach I'd love to hear them

[yuzisun]

@matty-rose option 1 sounds reasonable to me, also I think those s3 annotations(e.g s3-region, s3-endpoint/s3-usehttps) are more sensible to be added to the configmap so you do not need to configure for each service account.

In summary for s3 we first look for the IM Role annotation on service account and then check if it is attached with a secret of AWS credential data, the controller then inject the necessary environment variables based on global configurations or annotation overrides if secret presents.

[dimara]

We bumped into this issue as well with KServe 0.6 when trying to access a private bucket with SSE using IRSA only (without any secrets). storage-initialiazer failed with (the rather confusing):

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied

Once we removed the (recommended) bucket policy for allowing only HTTPS requests, storage-initializer failed with:

botocore.exceptions.ClientError: An error occurred (InvalidArgument) when calling the GetObject operation: Requests specifying Server Side Encryption with AWS KMS managed keys must be made over a secure connection.

This is because it ended up using the global HTTP endpoint for S3, that is http://s3.amazonaws.com:

kserve/python/kserve/kserve/storage.py

Line 155 in cbb93f6

endpoint_url=os.getenv("AWS_ENDPOINT_URL", "http://s3.amazonaws.com"),

Besides the HTTPS issue described above, storage-initializer will not work in an air-gapped env where we need to access the regional VPC endpoint.

So how about removing the hardcoded default value for the endpoint_url and let boto3 decide S3 endpoint based on its env?

In case of IRSA the pod will get AWS_REGION and AWS_DEFAULT_REGION so boto3 will use the regional endpoint. If the bucket is located in a different region boto3 will handle the redirect automatically. Finally, boto3 will use HTTPS by default. So things will work out of the box.

[surajkota]

@dimara thanks for reporting the issue. Are you able to use #2003 (comment) as a workaround to override just the s3 endpoint in the serving.kserve.io/s3-endpoint annotation? credential can still be pulled using IRSA

Letting AWS SDK decide the default S3 endpoint based on the env seems reasonable to me with the option to override using configmap/secret

[dimara]

@surajkota

as a workaround to override just the s3 endpoint in the serving.kserve.io/s3-endpoint annotation

Since we were using IRSA we wanted to avoid creating dummy secrets and attach them to service accounts.

Letting AWS SDK decide the default S3 endpoint based on the env seems reasonable to me with the option to override using configmap/secret

I agree. The said fix is orthogonal with the PR that @matty-rose opened recently (see #2373). Will try to send a PR for that.

[surajkota]

Feature was reviewed and merged in: #2373