No permissions for the repository that is obtained from image_uri

[daeho-ro]

When I use the image_uri from me-central-1 region for the forecasting-deepar image, then I got the error that

<user> is not authorized to perform: ecr:BatchGetImage on resource: arn:aws:ecr:me-central-1:272398656194:repository/forecasting-deepar because no resource-based policy allows the ecr:BatchGetImage action

I know the url is fine but there might be wrong permission on the ecr repository?

[CaoHoangTung]

Did you manage to solve it? I'm having the same issue

[daeho-ro]

Did you manage to solve it? I'm having the same issue

Nope.

[etothexipi]

Getting same issue trying to pull container for local development.

# Get the Data Wrangler container
image_uri = image_uris.retrieve(framework='data-wrangler', region='us-east-1', version='3.x')
log.info(f"Using Data Wrangler container: {image_uri}")

# Create a ScriptProcessor
script_processor = ScriptProcessor(
    command=['python3'],
    image_uri=image_uri,
    role=role,
    instance_count=1,
    # Use local mode for development
    instance_type="local"
)

DEBUG:botocore.hooks:Event needs-retry.ecr.GetAuthorizationToken: calling handler <botocore.retryhandler.RetryHandler object at 0x7f67a5394710>
DEBUG:botocore.retryhandler:No retry needed.
Login Succeeded
INFO:sagemaker.local.image:docker command: docker pull 663277389841.dkr.ecr.us-east-1.amazonaws.com/sagemaker-data-wrangler-container:3.x
Error response from daemon: pull access denied for 663277389841.dkr.ecr.us-east-1.amazonaws.com/sagemaker-data-wrangler-container, repository does not exist or may require 'docker login': denied: User: arn:aws:iam::*****:user/***** is not authorized to perform: ecr:BatchGetImage on resource: arn:aws:ecr:us-east-1:663277389841:repository/sagemaker-data-wrangler-container because no resource-based policy allows the ecr:BatchGetImage action

I am an admin in our account and gave my user full ECR access and still same issue.

Also, I can docker login to the ECR repo above from the same environment I am running my Pipeline locally.

[nov05]

🟢⚠️ I ran into a similar error when creating a SageMaker Canvas processing job. The code was automatically generated by Canvas. And my SageMaker execution role has the EC2InstanceProfileForImageBuilderECRContainerBuilds policy attached, which provides "ecr:BatchGetImage", "ecr:BatchCheckLayerAvailability", "ecr:GetDownloadUrlForLayer" permissions. However, my role doesn't have permissions for the ECR public registry, so it isn't able to pull the sagemaker-data-wrangler-container image. I believe this is the root cause of the issue.

UnexpectedStatusException: Error for Processing job data-wrangler-flow-processing-12-05-56-26-c7c7fe37: Failed. 
Reason: ClientError: ValidationException: Access denied for repository: sagemaker-data-wrangler-container in registry 
ID: 599662218115. Please check if your ECR repository and image exist and role arn:aws:iam::026211625715:role/service-
role/AmazonSageMaker-ExecutionRole-20241209T041445 has proper pull permissions for SageMaker: 
ecr:BatchCheckLayerAvailability, ecr:BatchGetImage, ecr:GetDownloadUrlForLayer
	status code: 400, request id: 1fc4d80b-db20-4875-a682-b9121ca44550

# Canvas Container URL.
container_uri = "599662218115.dkr.ecr.us-east-1.amazonaws.com/sagemaker-data-wrangler-container:5.x"
# Pinned Canvas Container URL.
container_uri_pinned = "599662218115.dkr.ecr.us-east-1.amazonaws.com/sagemaker-data-wrangler-container:5.0.6" 
...
from sagemaker.processing import Processor
processor = Processor(
    role=iam_role,
    image_uri=container_uri,
    instance_count=instance_count,
    instance_type=instance_type,
    volume_size_in_gb=volume_size_in_gb,
    network_config=network_config,
    sagemaker_session=sess,
    output_kms_key=kms_key,
    tags=user_tags
)
# Start Job
processor.run(
    inputs=[flow_input] + data_sources, 
    outputs=[processing_job_output],
    arguments=job_arguments,
    wait=False,
    logs=False,
    job_name=processing_job_name
)
sess.wait_for_processing_job(processing_job_name)

[mollyheamazon]

@nov05 It seems that you are using the container_uri automatically generated by Canvas. I have confirmed with Data Wrangler team that this is a bug that Data Wrangler didn't replace the ECR uri with correct account ids, but the fix is not prioritized. In this case, please replace the container_uri to those mentioned in this json file. Or, you can use retrieve function like image_uri = image_uris.retrieve(framework='data-wrangler', region='us-east-1', version='3.x') .

[mollyheamazon]

@etothexipi You mentioned that you are able to docker login, have you tried aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 663277389841.dkr.ecr.us-east-1.amazonaws.com? This should fix authentication issue with docker pull.

[mollyheamazon]

@daeho-ro @CaoHoangTung Can you provide more context when you try to use the image_uri from me-central-1 region for the forecasting-deepar image?
I'm able to execute image_name = sagemaker.image_uris.retrieve("forecasting-deepar", "me-central-1") in jupyter lab of sagemaker studio. Make sure you are calling from CreateModel or CreateTrainingJob function when you are using the image. The permissions policy do not support pulling ECR images directly, but only through SageMaker hosting or training call.

[mollyheamazon]

No response for 2 weeks. Closing this issue now. Please reopen if there is any issue.

[aramos-dino]

I also tried a local deployment and the pull image operation is returning forbidden
Error response from daemon: unknown: failed to resolve reference "522234722520.dkr.ecr.us-east-1.amazonaws.com/forecasting-deepar:1": unexpected status from HEAD request to https://522234722520.dkr.ecr.us-east-1.amazonaws.com/v2/forecasting-deepar/manifests/1: 403 Forbidden