Private ECR repositories gets Failed to look up dependency

[jfoechsler]

How are you running Renovate?

Self-hosted

Please select which platform you are using if self-hosting.

github.com

If you're self-hosting Renovate, tell us what version of Renovate you run.

v25.76.1

Describe the bug

During the same run Renovate can find latest digest in ECR for one repository but fails to find tags for another in same registry. This is running in Kubernetes (EKS).
I was able to make it temporarily work by using hostRules matching ECR hostname with username and temporary password. With the workaround Renovate recognizes our versioning with x.y-<qualifier> and bumps up y version field.
Usually it is using IAM role to authorize to ECR just fine.

Relevant debug logs

Logs

DEBUG: getDigest(https://xxx.dkr.ecr.us-east-1.amazonaws.com, aws-kubernetes-cli, undefined) (repository=Company/kubernetes-platform)
DEBUG: getManifestResponse(https://xxx.dkr.ecr.us-east-1.amazonaws.com, aws-kubernetes-cli, latest) (repository=Company/kubernetes-platform)
DEBUG: Got docker digest (repository=Company/kubernetes-platform)
       "digest": "sha256:ac2e5f649ff69deec72f7fecb7016a22942d6a99adfa21e018aef0f7f45feffb"

DEBUG: Datasource unauthorized (repository=Company/ruby-project)
       "datasource": "docker",
       "lookupName": "xxx.dkr.ecr.us-east-1.amazonaws.com/passenger-ruby27",
       "url": "https://xxx.dkr.ecr.us-east-1.amazonaws.com/v2/passenger-ruby27/tags/list?n=1000"
DEBUG: Failed to look up dependency xxx.dkr.ecr.us-east-1.amazonaws.com/passenger-ruby27 (repository=Company/ruby-project, packageFile=Dockerfile, dependency=xxx.dkr.ecr.us-east-1.amazonaws.com/passenger-ruby27)

Have you created a minimal reproduction repository?

No reproduction repository

[HonkingGoose]

I think we don't have any support for ECR yet, at least according to this issue:

• Support for Amazon ECR (Public Gallery) #9268

---

EDIT: Turns out we have limited support for ECR via our Docker datasource: #11322 (comment)

[github-actions]

Hi there,

Help us by making a minimal reproduction repository.

Before we can start work on your issue we first need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on creating a minimal reproduction to understand what is needed.

We may close the issue if you (or someone else) have not provided a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team

[rarkins]

There is some explicit support for ECR already within the docker datasource. I have marked this as reproduction necessary but it's either that or it's extensive logs and troubleshooting. It's not possible to do anything with the data provided so far here.

[jfoechsler]

Thanks for the replies. I will see if I can do a setup. Is there any way you know off hand to configure things to use the clearly working docker datasource (working when authentication is password based) and at the same time get token from EC2 instance (same way as is working to find latest digest)?

[rarkins]

I don't think I understand what you mean by the last question

[jfoechsler]

I meant since Renovate already seems capable of using IAM role for private ECR authentication (finds digests with no hostrules in config) and is also capable of listing all tags in private ECR to find new versions, but seemingly only when using explicit hostrule with username/password. So wondering if it is possible to configure Renovate in a way that means I don't need to bother with password token (which expires in 12 hours) or access keys.

I suppose I could extend CronJob with init container doing a ecr get-login-password, but seems a bit overkill.

[viceice]

We are doing ecr test and auth here:

renovate/lib/datasource/docker/common.ts

Lines 73 to 79 in 3d4d252

	if (ecrRegex.test(registryHost)) {
	const [, region] = ecrRegex.exec(registryHost);
	const auth = await getECRAuthToken(region, opts);
	if (auth) {
	opts.headers = { authorization: `Basic ${auth}` };
	}
	} else if (opts.username && opts.password) {

renovate/lib/datasource/docker/common.ts

Lines 26 to 52 in 3d4d252

	async function getECRAuthToken(
	region: string,
	opts: HostRule
	): Promise<string | null> {
	const config: ECRClientConfig = { region };
	if (opts.username && opts.password) {
	config.credentials = {
	accessKeyId: opts.username,
	secretAccessKey: opts.password,
	};
	}
	const ecr = new ECR(config);
	try {
	const data = await ecr.getAuthorizationToken({});
	const authorizationToken = data?.authorizationData?.[0]?.authorizationToken;
	if (authorizationToken) {
	return authorizationToken;
	}
	logger.warn(
	'Could not extract authorizationToken from ECR getAuthorizationToken response'
	);
	} catch (err) {
	logger.trace({ err }, 'err');
	logger.debug('ECR getAuthorizationToken error');
	}
	return null;
	}

So somebody needs to workout, how to use IAM in getECRAuthToken

[viceice]

Did you tried to set access id as username and secret key as password as hostrule?

Are you using a custom dns alias for your ecr registry?

[jfoechsler]

Did you tried to set access id as username and secret key as password as hostrule?

Are you using a custom dns alias for your ecr registry?

No custom alias just the regular internal .dkr.ecr.us-east-1.amazonaws.com.

With access keys it works nicely, so that is of course better than expiring token. Best would obviously be that Renovate uses AWS credentials from instance profile or service account token like other Kubernetes Pods.

Logs with access keys:

DEBUG: getLabels(https://xxx.dkr.ecr.us-east-1.amazonaws.com, passenger-ruby27, 1-builder) (repository=Company/ruby-project)
DEBUG: getManifestResponse(https://xxx.dkr.ecr.us-east-1.amazonaws.com, passenger-ruby27, 1-builder) (repository=Company/ruby-project)
DEBUG: Package releases lookups complete (repository=Company/ruby-project)
...
"deps": [
               {
                 "depName": "xxx.dkr.ecr.us-east-1.amazonaws.com/passenger-ruby27",
                 "currentValue": "1.41-builder",
                 "replaceString": "xxx.dkr.ecr.us-east-1.amazonaws.com/passenger-ruby27:1.41-builder",
                 "autoReplaceStringTemplate": "{{depName}}{{#if newValue}}:{{newValue}}{{/if}}{{#if newDigest}}@{{newDigest}}{{/if}}",
                 "datasource": "docker",
                 "depType": "stage",
                 "depIndex": 0,
                 "updates": [
                   {
                     "bucket": "non-major",
                     "newVersion": "1.56",
                     "newValue": "1.56-builder",
                     "newMajor": 1,
                     "newMinor": 56,
                     "updateType": "minor",
                     "branchName": "renovate/stable-xxx.dkr.ecr.us-east-1.amazonaws.com-passenger-ruby27-1.x"
                   }
                 ],
                 "warnings": [],
                 "versioning": "docker",
                 "currentVersion": "1.41",
                 "isSingleVersion": true,
                 "fixedVersion": "1.41-builder"
               }

[viceice]

@jfoechsler We are suing @aws-sdk/client-ecr v3 for ecr auth, so any hints how to get the token passed to the sdk?

[jfoechsler]

@viceice I am not experienced in the code side of things, but reading docs on that one I think getauthorizationtokencommand describes the scenario well.

I found issue 8660 as well. I also added the RENOVATE_X_HARD_EXIT=true because I have the problem of cron job never exiting.

My nodes has this policy BTW: AmazonEC2ContainerRegistryReadOnly

[jfoechsler]

Hello. I think I tracked it down to the ECR permission policy only having a limited set of actions allowed for delegation to AWS account where cluster is running. It was seemingly just enough for pulling images, but not quite enough to use repository with Renovate. After adding ecr:* actions in the ECR policy Renovate seems to have no issues accessing ECR repository without any explicit password or access keys.
Sorry for the trouble in that case.

[viceice]

Can you please help us to update our docs to include required roles for ecr?

So others can benefit of your research.