ECR OCI Credentials for Docker and Helm repositories #19241

jmuddle-via · 2022-12-04T12:45:17Z

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us what version of Renovate you run.

34.47.1

If you're self-hosting Renovate, select which platform you are using.

github.com

If you're self-hosting Renovate, tell us what version of the platform you run.

No response

Was this something which used to work for you, and then stopped?

I never saw this working

Describe the bug

In #19239 I describe the setup and make a request for the necessary hostRules.

Recap: I have an ECR OCI helm chart with ECR OCI helm sub-charts as dependencies.

I now have a better understanding of the issue.

Looking at the helm manager code https://github.com/renovatebot/renovate/blob/main/lib/modules/manager/helmv3/artifacts.ts#L29
I can see that if the repository is an OCI repository, it will swap "oci://" for "https://", which matches the documentation here https://docs.renovatebot.com/modules/manager/helmv3/

Here https://github.com/renovatebot/renovate/blob/main/lib/modules/manager/helmv3/artifacts.ts#L57 uses the username and password from the host rules.
For ECR OCI repositories, this should match this

aws ecr get-login-password --region \<region> | helm registry login --username AWS --password-stdin \<account>.dkr.ecr.ca-central-1.amazonaws.com

If I create the following hostRule

           {
              "hostType": "docker",
              "matchHost": "https://<account-id>.dkr.ecr.ca-central-1.amazonaws.com",
              "username": "AWS",
              "password": "*********"
            }

I get an error related to finding the tags for the repository.
To get the tags, the docker data source is used.
This expects using the following authentication for ECR repositories

renovate/lib/modules/datasource/docker/index.ts

Line 232 in 8e4b523

async function getECRAuthToken(

This is expecting AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

So I need to pass the AWS credentials for ECR repositories but a token for Helm login.

My suggestion is that allow the token field to be used for dynamic authentication.
This is similar to this request #16912
and a bit like this https://docs.renovatebot.com/docker/#using-short-lived-access-tokens

Relevant debug logs

Logs

DEBUG: Updating Helm artifacts (repository=<github-org>/<repo name>, branch=renovate/<branch name>-0.x)
DEBUG: Setting CONTAINERBASE_CACHE_DIR to /tmp/renovate/cache/containerbase (repository=<github-org>/<repo name>, branch=renovate/<branch name>-0.x)
DEBUG: Using containerbase dynamic installs (repository=<github-org>/<repo name>, branch=renovate/<branch name>-0.x)
TRACE: Authorization already set (repository=<github-org>/<repo name>, branch=renovate/<branch name>-0.x)
       "url": "https://api.github.com/repos/helm/helm/releases?per_page=100"
TRACE: got request (repository=<github-org>/<repo name>, branch=renovate/<branch name>-0.x)
       "url": "https://api.github.com/repos/helm/helm/releases?per_page=100",
       "options": {
         "method": "get",
         "context": {"hostType": "github-releases"},
         "hostType": "github-releases",
         "baseUrl": "https://api.github.com/",
         "paginate": true,
         "responseType": "json",
         "headers": {
           "accept": "application/json, application/vnd.github.v3+json",
           "user-agent": "RenovateBot/34.47.1 (https://github.com/renovatebot/renovate)",
           "authorization": "***********"
         },
         "throwHttpErrors": true,
         "hooks": {"beforeRedirect": ["[function]"]},
         "timeout": 60000
       }
TRACE: Authorization already set (repository=<github-org>/<repo name>, branch=renovate/<branch name>-0.x)
       "url": "https://api.github.com/repositories/43723161/releases?per_page=100&page=2"
TRACE: got request (repository=<github-org>/<repo name>, branch=renovate/<branch name>-0.x)
       "url": "https://api.github.com/repositories/43723161/releases?per_page=100&page=2",
       "options": {
         "method": "get",
         "context": {"hostType": "github-releases"},
         "hostType": "github-releases",
         "baseUrl": "https://api.github.com/",
         "paginate": false,
         "responseType": "json",
         "headers": {
           "accept": "application/json, application/vnd.github.v3+json",
           "user-agent": "RenovateBot/34.47.1 (https://github.com/renovatebot/renovate)",
           "authorization": "***********"
         },
         "throwHttpErrors": true,
         "hooks": {"beforeRedirect": ["[function]"]},
         "timeout": 60000
       }
DEBUG: Resolved stable matching version (repository=<github-org>/<repo name>, branch=renovate/<branch name>-0.x)
       "toolName": "helm",
       "constraint": undefined,
       "resolvedVersion": "v3.10.2"
DEBUG: Executing command (repository=<github-org>/<repo name>, branch=renovate/<branch name>-0.x)
       "command": "install-tool helm v3.10.2"
TRACE: Command options (repository=<github-org>/<repo name>, branch=renovate/<branch name>-0.x)
       "commandOptions": {
         "cwd": "/tmp/renovate/repos/github/<github-org>/<repo name>",
         "encoding": "utf-8",
         "env": {
           "HELM_EXPERIMENTAL_OCI": "1",
           "HOME": "/home/ubuntu",
           "PATH": "/home/ubuntu/.local/bin:/go/bin:/home/ubuntu/bin:/opt/buildpack/tools/python/3.11.0/bin:/home/ubuntu/.npm-global/bin:/home/ubuntu/.cargo/bin:/home/ubuntu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
           "LC_ALL": "C.UTF-8",
           "LANG": "C.UTF-8",
           "BUILDPACK_CACHE_DIR": "/tmp/renovate/cache/containerbase",
           "CONTAINERBASE_CACHE_DIR": "/tmp/renovate/cache/containerbase"
         },
         "maxBuffer": 10485760,
         "timeout": 900000
       }
DEBUG: exec completed (repository=<github-org>/<repo name>, branch=renovate/<branch name>-0.x)
       "durationMs": 32,
       "stdout": "tool helm v3.10.2 is already installed\ntool is already linked: helm v3.10.2\nInstalled v2 /usr/local/buildpack/tools/v2/helm.sh in 0 seconds\nskip cleanup, not a docker build: log-5whbg\n",
       "stderr": ""
DEBUG: Executing command (repository=<github-org>/<repo name>, branch=renovate/<branch name>-0.x)
       "command": "helm repo add stable --registry-config /tmp/renovate/cache/__renovate-private-cache/registry.json --repository-config /tmp/renovate/cache/__renovate-private-cache/repositories.yaml --repository-cache /tmp/renovate/cache/__renovate-private-cache/repositories https://charts.helm.sh/stable"
TRACE: Command options (repository=<github-org>/<repo name>, branch=renovate/<branch name>-0.x)
       "commandOptions": {
         "cwd": "/tmp/renovate/repos/github/<github-org>/<repo name>",
         "encoding": "utf-8",
         "env": {
           "HELM_EXPERIMENTAL_OCI": "1",
           "HOME": "/home/ubuntu",
           "PATH": "/home/ubuntu/.local/bin:/go/bin:/home/ubuntu/bin:/opt/buildpack/tools/python/3.11.0/bin:/home/ubuntu/.npm-global/bin:/home/ubuntu/.cargo/bin:/home/ubuntu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
           "LC_ALL": "C.UTF-8",
           "LANG": "C.UTF-8",
           "BUILDPACK_CACHE_DIR": "/tmp/renovate/cache/containerbase",
           "CONTAINERBASE_CACHE_DIR": "/tmp/renovate/cache/containerbase"
         },
         "maxBuffer": 10485760,
         "timeout": 900000
       }
DEBUG: exec completed (repository=<github-org>/<repo name>, branch=renovate/<branch name>-0.x)
       "durationMs": 1326,
       "stdout": "\"stable\" has been added to your repositories\n",
       "stderr": ""
DEBUG: Executing command (repository=<github-org>/<repo name>, branch=renovate/<branch name>-0.x)
       "command": "helm dependency update --registry-config /tmp/renovate/cache/__renovate-private-cache/registry.json --repository-config /tmp/renovate/cache/__renovate-private-cache/repositories.yaml --repository-cache /tmp/renovate/cache/__renovate-private-cache/repositories <repo name>"
TRACE: Command options (repository=<github-org>/<repo name>, branch=renovate/<branch name>-0.x)
       "commandOptions": {
         "cwd": "/tmp/renovate/repos/github/<github-org>/<repo name>",
         "encoding": "utf-8",
         "env": {
           "HELM_EXPERIMENTAL_OCI": "1",
           "HOME": "/home/ubuntu",
           "PATH": "/home/ubuntu/.local/bin:/go/bin:/home/ubuntu/bin:/opt/buildpack/tools/python/3.11.0/bin:/home/ubuntu/.npm-global/bin:/home/ubuntu/.cargo/bin:/home/ubuntu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
           "LC_ALL": "C.UTF-8",
           "LANG": "C.UTF-8",
           "BUILDPACK_CACHE_DIR": "/tmp/renovate/cache/containerbase",
           "CONTAINERBASE_CACHE_DIR": "/tmp/renovate/cache/containerbase"
         },
         "maxBuffer": 10485760,
         "timeout": 900000
       }
DEBUG: rawExec err (repository=<github-org>/<repo name>, branch=renovate/<branch name>-0.x)
       "err": {
         "name": "ExecError",
         "cmd": "/bin/sh -c helm dependency update --registry-config /tmp/renovate/cache/__renovate-private-cache/registry.json --repository-config /tmp/renovate/cache/__renovate-private-cache/repositories.yaml --repository-cache /tmp/renovate/cache/__renovate-private-cache/repositories <repo name>",
         "stderr": "Error: could not download oci://<aws-account>.dkr.ecr.ca-central-1.amazonaws.com/<sub-chart name>: pulling from host <aws-account>.dkr.ecr.ca-central-1.amazonaws.com failed with status code [manifests 0.5.2]: 401 Unauthorized\n",
         "stdout": "Hang tight while we grab the latest from your chart repositories...\n...Successfully got an update from the \"stable\" chart repository\nUpdate Complete. ⎈Happy Helming!⎈\nSaving 2 charts\nDownloading <sub-chart name> from repo oci://<aws-account>.dkr.ecr.ca-central-1.amazonaws.com\nSave error occurred:  could not download oci://<aws-account>.dkr.ecr.ca-central-1.amazonaws.com/<sub-chart name>: pulling from host <aws-account>.dkr.ecr.ca-central-1.amazonaws.com failed with status code [manifests 0.5.2]: 401 Unauthorized\n",
         "options": {
           "cwd": "/tmp/renovate/repos/github/<github-org>/<repo name>",
           "encoding": "utf-8",
           "env": {
             "HELM_EXPERIMENTAL_OCI": "1",
             "HOME": "/home/ubuntu",
             "PATH": "/home/ubuntu/.local/bin:/go/bin:/home/ubuntu/bin:/opt/buildpack/tools/python/3.11.0/bin:/home/ubuntu/.npm-global/bin:/home/ubuntu/.cargo/bin:/home/ubuntu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
             "LC_ALL": "C.UTF-8",
             "LANG": "C.UTF-8",
             "BUILDPACK_CACHE_DIR": "/tmp/renovate/cache/containerbase",
             "CONTAINERBASE_CACHE_DIR": "/tmp/renovate/cache/containerbase"
           },
           "maxBuffer": 10485760,
           "timeout": 900000
         },
         "exitCode": 1,
         "message": "Command failed: helm dependency update --registry-config /tmp/renovate/cache/__renovate-private-cache/registry.json --repository-config /tmp/renovate/cache/__renovate-private-cache/repositories.yaml --repository-cache /tmp/renovate/cache/__renovate-private-cache/repositories <repo name>\nError: could not download oci://<aws-account>.dkr.ecr.ca-central-1.amazonaws.com/<sub-chart name>: pulling from host <aws-account>.dkr.ecr.ca-central-1.amazonaws.com failed with status code [manifests 0.5.2]: 401 Unauthorized\n",
         "stack": "ExecError: Command failed: helm dependency update --registry-config /tmp/renovate/cache/__renovate-private-cache/registry.json --repository-config /tmp/renovate/cache/__renovate-private-cache/repositories.yaml --repository-cache /tmp/renovate/cache/__renovate-private-cache/repositories <repo name>\nError: could not download oci://<aws-account>.dkr.ecr.ca-central-1.amazonaws.com/<sub-chart name>: pulling from host <aws-account>.dkr.ecr.ca-central-1.amazonaws.com failed with status code [manifests 0.5.2]: 401 Unauthorized\n\n    at ChildProcess.<anonymous> (/usr/src/app/node_modules/renovate/lib/util/exec/common.ts:99:11)\n    at ChildProcess.emit (node:events:525:35)\n    at ChildProcess.emit (node:domain:489:12)\n    at Process.ChildProcess._handle.onexit (node:internal/child_process:293:12)"
       }
DEBUG: Failed to update Helm lock file (repository=<github-org>/<repo name>, branch=renovate/<branch name>-0.x)
       "err": {
         "name": "ExecError",
         "cmd": "/bin/sh -c helm dependency update --registry-config /tmp/renovate/cache/__renovate-private-cache/registry.json --repository-config /tmp/renovate/cache/__renovate-private-cache/repositories.yaml --repository-cache /tmp/renovate/cache/__renovate-private-cache/repositories <repo name>",
         "stderr": "Error: could not download oci://<aws-account>.dkr.ecr.ca-central-1.amazonaws.com/<sub-chart name>: pulling from host <aws-account>.dkr.ecr.ca-central-1.amazonaws.com failed with status code [manifests 0.5.2]: 401 Unauthorized\n",
         "stdout": "Hang tight while we grab the latest from your chart repositories...\n...Successfully got an update from the \"stable\" chart repository\nUpdate Complete. ⎈Happy Helming!⎈\nSaving 2 charts\nDownloading <sub-chart name> from repo oci://<aws-account>.dkr.ecr.ca-central-1.amazonaws.com\nSave error occurred:  could not download oci://<aws-account>.dkr.ecr.ca-central-1.amazonaws.com/<sub-chart name>: pulling from host <aws-account>.dkr.ecr.ca-central-1.amazonaws.com failed with status code [manifests 0.5.2]: 401 Unauthorized\n",
         "options": {
           "cwd": "/tmp/renovate/repos/github/<github-org>/<repo name>",
           "encoding": "utf-8",
           "env": {
             "HELM_EXPERIMENTAL_OCI": "1",
             "HOME": "/home/ubuntu",
             "PATH": "/home/ubuntu/.local/bin:/go/bin:/home/ubuntu/bin:/opt/buildpack/tools/python/3.11.0/bin:/home/ubuntu/.npm-global/bin:/home/ubuntu/.cargo/bin:/home/ubuntu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
             "LC_ALL": "C.UTF-8",
             "LANG": "C.UTF-8",
             "BUILDPACK_CACHE_DIR": "/tmp/renovate/cache/containerbase",
             "CONTAINERBASE_CACHE_DIR": "/tmp/renovate/cache/containerbase"
           },
           "maxBuffer": 10485760,
           "timeout": 900000
         },
         "exitCode": 1,
         "message": "Command failed: helm dependency update --registry-config /tmp/renovate/cache/__renovate-private-cache/registry.json --repository-config /tmp/renovate/cache/__renovate-private-cache/repositories.yaml --repository-cache /tmp/renovate/cache/__renovate-private-cache/repositories <repo name>\nError: could not download oci://<aws-account>.dkr.ecr.ca-central-1.amazonaws.com/<sub-chart name>: pulling from host <aws-account>.dkr.ecr.ca-central-1.amazonaws.com failed with status code [manifests 0.5.2]: 401 Unauthorized\n",
         "stack": "ExecError: Command failed: helm dependency update --registry-config /tmp/renovate/cache/__renovate-private-cache/registry.json --repository-config /tmp/renovate/cache/__renovate-private-cache/repositories.yaml --repository-cache /tmp/renovate/cache/__renovate-private-cache/repositories <repo name>\nError: could not download oci://<aws-account>.dkr.ecr.ca-central-1.amazonaws.com/<sub-chart name>: pulling from host <aws-account>.dkr.ecr.ca-central-1.amazonaws.com failed with status code [manifests 0.5.2]: 401 Unauthorized\n\n    at ChildProcess.<anonymous> (/usr/src/app/node_modules/renovate/lib/util/exec/common.ts:99:11)\n    at ChildProcess.emit (node:events:525:35)\n    at ChildProcess.emit (node:domain:489:12)\n    at Process.ChildProcess._handle.onexit (node:internal/child_process:293:12)"
       }
DEBUG: Updated 1 package files (repository=<github-org>/<repo name>, branch=renovate/<branch name>-0.x)```

</details>


### Have you created a minimal reproduction repository?

No

The text was updated successfully, but these errors were encountered:

secustor · 2022-12-06T14:49:49Z

This requires a small refactor of the docker datasource, which breaks out the token request logic and exposes it for usage in other modules.
This function is then used by the helm manager for requesting credentials.

jmuddle-via · 2022-12-07T09:27:27Z

Hi @secustor thanks for the reply.

Thinking about this a little more, what I really want is the ability to do the following:

Set the following host rule that is only relevant for Helm charts.

{
  "hostType": "helm",
  "matchHost": "oci://<account-id>.dkr.ecr.ca-central-1.amazonaws.com",
  "username": "AWS",
  "password": <ecr-token>
}

It would be great if I could also do the above for Docker host rules.

{
  "hostType": "docker",
  "matchHost": "https://<account-id>.dkr.ecr.ca-central-1.amazonaws.com",
  "username": "AWS",
  "password": <ecr-token>
}

My preference here is that a token is used for authentication rather than an AWS access key.

secustor · 2022-12-07T11:58:09Z

If I'm understanding you correctly the second rule should work right now, at least if you supply a valid short lived token to password.

For OCI registries host-rules with hostType docker are used.

jmuddle-via · 2022-12-07T13:26:51Z

My understanding was that because these are ECR repositories, Renovate assumes that AWS access keys are passed.

This line seems to force ECR auth for ECR repositories

renovate/lib/modules/datasource/docker/index.ts

Line 103 in 8e4b523

if (ecrRegex.test(registryHost)) {

secustor · 2022-12-07T14:36:28Z

K, I see now what you mean.
That would be a different feature request as there is no opt out feature for ECR logins right now.

markushinz · 2023-08-31T09:38:40Z

Any updates on this? Right now all "helm login" commands fail as renovate tries to use AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY (which are configured as username and password in the hostRules) directly instead of using the aforementioned special treatment of ECR registries.

renovate/lib/modules/manager/helmv3/artifacts.ts

Lines 35 to 54 in 2f4c711

    
           const registries: RepositoryRule[] = repositories 
        
             .filter(isOCIRegistry) 
        
             .map((value) => { 
        
               return { 
        
                 ...value, 
        
                 repository: value.repository.replace('oci://', ''), 
        
                 hostRule: hostRules.find({ 
        
                   url: value.repository.replace('oci://', 'https://'), //TODO we need to replace this, as oci:// will not be accepted as protocol 
        
                   hostType: DockerDatasource.id, 
        
                 }), 
        
               }; 
        
             }); 
        
           // if credentials for the registry have been found, log into it 
        
           registries.forEach((value) => { 
        
             const loginCmd = generateLoginCmd(value, 'helm registry login'); 
        
             if (loginCmd) { 
        
               cmd.push(loginCmd); 
        
             } 
        
           });

renovate/lib/modules/manager/helmv3/common.ts

Lines 8 to 19 in 2f4c711

    
           export function generateLoginCmd( 
        
             repositoryRule: RepositoryRule, 
        
             loginCMD: string 
        
           ): string | null { 
        
             const { username, password } = repositoryRule.hostRule; 
        
             if (username && password) { 
        
               return `${loginCMD} --username ${quote(username)} --password ${quote( 
        
                 password 
        
               )} ${repositoryRule.repository}`; 
        
             } 
        
             return null; 
        
           }

secustor · 2023-08-31T09:48:21Z

@markushinz If there updates they are posted here.

Should you need this feature, feel free to provide a contribution as this is marked as status:ready.
Alternatively you can sponsor the feature.

ezzatron · 2023-09-15T04:10:19Z

I've submitted PR #24432 which should resolve ECR auth for Helm once merged. Feedback and/or help getting it merged would be welcome 🙏

renovate-release · 2023-09-23T06:44:58Z

🎉 This issue has been resolved in version 36.104.0 🎉

The release is available on:

GitHub release
36.104.0

Your semantic-release bot 📦🚀

jmuddle-via added priority-5-triage status:requirements Full requirements are not yet known, so implementation should not be started type:bug Bug fix of existing functionality labels Dec 4, 2022

secustor added type:feature Feature (new functionality) priority-3-medium Default priority, "should be done" but isn't prioritised ahead of others manager:helm Helm package manager datasource:docker and removed type:bug Bug fix of existing functionality priority-5-triage labels Dec 6, 2022

secustor added status:ready and removed status:requirements Full requirements are not yet known, so implementation should not be started labels Dec 6, 2022

ezzatron mentioned this issue Sep 15, 2023

feat(manager/helmv3): add support for ECR credentials #24432

Merged

6 tasks

rarkins closed this as completed in #24432 Sep 23, 2023

github-actions bot locked as resolved and limited conversation to collaborators Oct 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ECR OCI Credentials for Docker and Helm repositories #19241

ECR OCI Credentials for Docker and Helm repositories #19241

jmuddle-via commented Dec 4, 2022 •

edited

Loading

secustor commented Dec 6, 2022 •

edited

Loading

jmuddle-via commented Dec 7, 2022

secustor commented Dec 7, 2022 •

edited

Loading

jmuddle-via commented Dec 7, 2022 •

edited

Loading

secustor commented Dec 7, 2022

markushinz commented Aug 31, 2023

secustor commented Aug 31, 2023

ezzatron commented Sep 15, 2023 •

edited

Loading

renovate-release commented Sep 23, 2023

ECR OCI Credentials for Docker and Helm repositories #19241

ECR OCI Credentials for Docker and Helm repositories #19241

Comments

jmuddle-via commented Dec 4, 2022 • edited Loading

How are you running Renovate?

If you're self-hosting Renovate, tell us what version of Renovate you run.

If you're self-hosting Renovate, select which platform you are using.

If you're self-hosting Renovate, tell us what version of the platform you run.

Was this something which used to work for you, and then stopped?

Describe the bug

Relevant debug logs

secustor commented Dec 6, 2022 • edited Loading

jmuddle-via commented Dec 7, 2022

secustor commented Dec 7, 2022 • edited Loading

jmuddle-via commented Dec 7, 2022 • edited Loading

secustor commented Dec 7, 2022

markushinz commented Aug 31, 2023

secustor commented Aug 31, 2023

ezzatron commented Sep 15, 2023 • edited Loading

renovate-release commented Sep 23, 2023

jmuddle-via commented Dec 4, 2022 •

edited

Loading

secustor commented Dec 6, 2022 •

edited

Loading

secustor commented Dec 7, 2022 •

edited

Loading

jmuddle-via commented Dec 7, 2022 •

edited

Loading

ezzatron commented Sep 15, 2023 •

edited

Loading