Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SageMaker endpoints deployments only allow container images from the current account #986

Open
pjo74 opened this issue Aug 23, 2023 · 0 comments
Labels
bug Something isn't working needs-triage The issue or PR still needs to be triaged

Comments

@pjo74
Copy link

pjo74 commented Aug 23, 2023

When trying to deploy a SageMaker endpoint using aws-solutions-constructs, such as LambdaToSagemakerEndpoint, if the inference docker container image provided is hosted in a different account than the current one, it fails because of missing IAM permissions.
This prevents to use any of the Pre-built SageMaker Docker images that are hosted on various AWS-owned accounts depending on the regions.
See Amazon SageMaker ECR Paths

Reproduction Steps

const constructProps: LambdaToSagemakerEndpointProps = {
  modelProps: {
    primaryContainer: {
      image: '763104351884.dkr.ecr.eu-west-1.amazonaws.com/tensorflow-inference:2.8-cpu',
      modelDataUrl: 's3://jumpstart-cache-prod-eu-west-1/tensorflow-infer/infer-tensorflow-ic-swin-base-patch4-window7-224.tar.gz',
    },
  },
  lambdaFunctionProps: {
    runtime: lambda.Runtime.PYTHON_3_10,
    handler: 'index.handler',
    code: lambda.Code.fromInline(`
    def handler(event, context):
      print('Hello Lambda')
    `)
  },
};

const lambda_sagemaker = new LambdaToSagemakerEndpoint(this, 'LambdaToSagemakerEndpointPattern', constructProps);

Error Log

9:52:55 AM | CREATE_FAILED | AWS::SageMaker::Endpoint | LambdaToSagemakerE.../SagemakerEndpoint
The role 'arn:aws:iam::masked-account:role/IcServerlessInferenceStac-LambdaToSagemakerEndpoin-B318Y1J1FRYS' does not have BatchGetImage permission for the image: '763104351884.dkr.ecr.eu-west-1.amazonaws.com/tensorflow-inference:2.8-cpu'.

Environment

  • CDK CLI Version : 2.92.0
  • CDK Framework Version: 2
  • AWS Solutions Constructs Version : 2.42.0
  • OS : macOS Ventura 13.3
  • Language : English

Other

The permissions of the generated IAM role are restricted to the current account only.
This is what prevents the desired image to be retrieved from its ECR repository.

See aws-solutions-constructs/core/lib/sagemaker-helper.ts#L128C1-L128C1

  // To create a Sagemaker model using Bring-Your-Own-Model (BYOM) algorith image
  // The image URL is specified in the modelProps
  role.addToPolicy(
    new iam.PolicyStatement({
      resources: [`arn:${cdk.Aws.PARTITION}:ecr:${cdk.Aws.REGION}:${cdk.Aws.ACCOUNT_ID}:repository/*`],
      actions: [
        'ecr:BatchCheckLayerAvailability',
        'ecr:GetDownloadUrlForLayer',
        'ecr:DescribeRepositories',
        'ecr:DescribeImages',
        'ecr:BatchGetImage',
      ],
    })
  );

This is 🐛 Bug Report

@pjo74 pjo74 added bug Something isn't working needs-triage The issue or PR still needs to be triaged labels Aug 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working needs-triage The issue or PR still needs to be triaged
Projects
None yet
Development

No branches or pull requests

1 participant