Multi-Account/Multi-Tenant AWS ECR Replication not working with IAM Roles

[slacki123]

Harbor version
2.6.1

Chart Version
1.10.1

The use case
We have 24 AWS accounts and we would like to use harbor replication feature using AWS ECR using IAM roles for authentication instead of Access Keys

The problem
After configuring our harbor AWS-ECR type registries which we use for image replication, we find that despite specifying a registry with a specific AWS account ID endpoint (eg. 2222222222.dkr.ecr.ap-eu-west-1.amazonaws.com), the replication then ever only happens in the same account as our harbor application is running on (which is on account id 11111111111).

So basically it is replicating to account 11111111111 when it should be replicating to 2222222222

Similar issue has been previously referenced below:
#15388

It looks like someone released a fix in the PR here: #17533.
Although it looks like the url parameter has been removed in a further MR here: #17583, so it's unclear what is exactly happening there

We now need to confirm whether this needs a fix for when using IAM roles and not just access keys

Please let me know if I can add any more information

[Vad1mo]

@slacki123 is PR #17932 related to your issue?

[slacki123]

@Vad1mo I think that one is a bit different. The error message I am now getting when using 2.7.0-rc1 harbor version is

[ERROR] [/controller/replication/transfer/image/transfer.go:481]: failed to check the existence of the manifest of artifact testrepo/test:latest on the destination registry: http status code: 403, body: 

Perhaps something wrong with assuming role on the other account.

How does harbor actually assume a role with IAM so that it can access resources in another account?

[Vad1mo]

I don't know, I need to investigate that

[slacki123]

If we want to assume role using AWS CLI and retrieve the required keys, we use a command such as the below

aws sts assume-role --role-arn arn:aws:iam::${account_id}:role/${role_name} --role-session-name harbor-replication --region ${region}

The variables such as account_id and region can be extracted from the URL passed into the registry, but there is no field for role_name. Would be good to know if this is supported at all, and if there is a default role that users need to create, or if there's a way to pass a role that could be used

[caleblloyd]

Once #17932 is merged, you can assume a role using any of the configuration options that support assuming roles in the Go SDK including IAM Roles for Service Accounts on Kubernetes

You would just use 1 Role that has permissions to pull all of the images in all of your accounts

You would need to grant this role permissions in your other accounts by allowing secondary account access, except you could use the Role as the principal here instead of the entire AWS account.

[slacki123]

Thank you @caleblloyd

This would explain why I have been getting 403 errors when trying to access images in the ECR of other accounts - it's because the role that is being used is the EC2 instance role rather than my injected service account pod role

Do you know if the #17932 PR is merged, is this going to be targeted at 2.7.0 release or beyond at this point?

[github-actions]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[Vad1mo]

resolved with #17932