Custom data source authentication #24377

akselleirv · 2023-09-12T11:28:54Z

akselleirv
Sep 12, 2023

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us what version of Renovate you run.

v39.0.3

If you're self-hosting Renovate, select which platform you are using.

github.com

Was this something which used to work for you, and then stopped?

I never saw this working

Wanted end result.

I want to download a file from a private GitHub repository.

What you tried so far.

I have been able to authenticate using a GitHub App which can read and write to the repository, for example creating PRs, but I'm not able to fetch a file using the GitHub API.

I have the following config of a custom datasource:

   ...
    customDatasources: {
        clusterAddons: {
            defaultRegistryUrlTemplate: "https://api.github.com/repos/my-org/my-repo/contents/charts/my-file.json",
            transformTemplates: [
                "{\"releases\": [{\"version\": $base64decode(content).addonVersionsIndex.{{ packageName }} }]}"
            ]
        }
    }

But it gets a 404 indicating that the authentication is not working.

I have been able to fetch the file using curl with a PAT sent as a bearer token.

Relevant debug logs

Logs

DEBUG: File config
       "config": {
         "username": "Renovate<MY_ORG>",
         "gitAuthor": "Renovate<MY_ORG> Bot <143614921+renovate<MY_ORG>[bot]@users.noreply.github.enterprise.com>",
         "onboarding": false,
         "requireConfig": "optional",
         "extends": ["config:recommended", ":disableDependencyDashboard"],
         "repositories": ["<MY_ORG>/<MY_REPO>"],
         "additionalBranchPrefix": "{{#if (equals parentDir 'clusters')}}{{{replace 'clusters/(.*)\\.cue' '$1' packageFile}}}-{{else}}{{parentDir}}-{{/if}}",
         "commitMessageExtra": "for {{#if (equals parentDir 'clusters')}}{{{replace 'clusters/(.*)\\.cue' '$1' packageFile}}}{{else}}{{parentDir}}{{/if}}",
         "enabledManagers": ["regex"],
         "packageRules": [{"matchManagers": ["regex"]}],
         "regexManagers": [
           {
             "customType": "regex",
             "fileMatch": ["^clusters/.*.cue$"],
             "matchStrings": [
               ":\\s(?<depName>[^\":]+).#Chart\\s&\\s{[\\s\\S]*?values:\\s\"(?<currentValue>[^\"]+)\":"
             ],
             "datasourceTemplate": "custom.clusterAddons"
           }
         ],
         "customDatasources": {
           "clusterAddons": {
             "defaultRegistryUrlTemplate": "https://api.github.com/repos/<MY_ORG>/<MY_REPO>/contents/charts/addon_versions_index.json",
             "transformTemplates": [
               "{\"releases\": [{\"version\": $base64decode(content).addonVersionsIndex.{{ packageName }} }]}"
             ]
           }
         }
       }
DEBUG: CLI config
       "config": {}
DEBUG: Env config
       "config": {"hostRules": [], "token": "***********"}
DEBUG: Combined config
       "config": {
         "username": "Renovate<MY_ORG>",
         "gitAuthor": "Renovate<MY_ORG> Bot <143614921+renovate<MY_ORG>[bot]@users.noreply.github.enterprise.com>",
         "onboarding": false,
         "requireConfig": "optional",
         "extends": ["config:recommended", ":disableDependencyDashboard"],
         "repositories": ["<MY_ORG>/<MY_REPO>"],
         "additionalBranchPrefix": "{{#if (equals parentDir 'clusters')}}{{{replace 'clusters/(.*)\\.cue' '$1' packageFile}}}-{{else}}{{parentDir}}-{{/if}}",
         "commitMessageExtra": "for {{#if (equals parentDir 'clusters')}}{{{replace 'clusters/(.*)\\.cue' '$1' packageFile}}}{{else}}{{parentDir}}{{/if}}",
         "enabledManagers": ["regex"],
         "packageRules": [{"matchManagers": ["regex"]}],
         "regexManagers": [
           {
             "customType": "regex",
             "fileMatch": ["^clusters/.*.cue$"],
             "matchStrings": [
               ":\\s(?<depName>[^\":]+).#Chart\\s&\\s{[\\s\\S]*?values:\\s\"(?<currentValue>[^\"]+)\":"
             ],
             "datasourceTemplate": "custom.clusterAddons"
           }
         ],
         "customDatasources": {
           "clusterAddons": {
             "defaultRegistryUrlTemplate": "https://api.github.com/repos/<MY_ORG>/<MY_REPO>/contents/charts/addon_versions_index.json",
             "transformTemplates": [
               "{\"releases\": [{\"version\": $base64decode(content).addonVersionsIndex.{{ packageName }} }]}"
             ]
           }
         },
         "hostRules": [],
         "token": "***********"
       }
DEBUG: Enabling forkProcessing while in non-autodiscover mode
DEBUG: Found valid git version: 2.42.0
DEBUG: Using default github endpoint: https://api.github.com/
DEBUG: Platform config
       "platformConfig": {
         "hostType": "github",
         "endpoint": "https://api.github.com/",
         "isGHApp": true,
         "isGhe": false
       },
       "renovateUsername": "Renovate<MY_ORG>"
DEBUG: Using configured gitAuthor (Renovate<MY_ORG> Bot <143614921+renovate<MY_ORG>[bot]@users.noreply.github.enterprise.com>)
DEBUG: Adding token authentication for api.github.com to hostRules
DEBUG: Using baseDir: /tmp/renovate
DEBUG: Using cacheDir: /tmp/renovate/cache
DEBUG: Using containerbaseDir: /tmp/renovate/cache/containerbase
DEBUG: Initializing Renovate internal cache into /tmp/renovate/cache/renovate/renovate-cache-v1
DEBUG: Commits limit = null
DEBUG: Setting global hostRules
DEBUG: Adding token authentication for api.github.com to hostRules
DEBUG: validatePresets()
DEBUG: Reinitializing hostRules for repo
DEBUG: Clearing hostRules
DEBUG: Adding token authentication for api.github.com to hostRules
 INFO: Repository started (repository=<MY_ORG>/<MY_REPO>)
       "renovateVersion": "36.93.1"
DEBUG: Using localDir: /tmp/renovate/repos/github/<MY_ORG>/<MY_REPO> (repository=<MY_ORG>/<MY_REPO>)
DEBUG: PackageFiles.clear() - Package files deleted (repository=<MY_ORG>/<MY_REPO>)
DEBUG: initRepo("<MY_ORG>/<MY_REPO>") (repository=<MY_ORG>/<MY_REPO>)
DEBUG: <MY_ORG>/<MY_REPO> default branch = main (repository=<MY_ORG>/<MY_REPO>)
DEBUG: Using app token for git init (repository=<MY_ORG>/<MY_REPO>)
DEBUG: Resetting npmrc (repository=<MY_ORG>/<MY_REPO>)
DEBUG: Resetting npmrc (repository=<MY_ORG>/<MY_REPO>)
DEBUG: checkOnboarding() (repository=<MY_ORG>/<MY_REPO>)
DEBUG: isOnboarded() (repository=<MY_ORG>/<MY_REPO>)
DEBUG: Repo is onboarded (repository=<MY_ORG>/<MY_REPO>)
DEBUG: Initializing git repository into /tmp/renovate/repos/github/<MY_ORG>/<MY_REPO> (repository=<MY_ORG>/<MY_REPO>)
DEBUG: Performing blobless clone (repository=<MY_ORG>/<MY_REPO>)
DEBUG: git clone completed (repository=<MY_ORG>/<MY_REPO>)
       "durationMs": 1330
DEBUG: latest repository commit (repository=<MY_ORG>/<MY_REPO>)
       "latestCommit": {
         "hash": "87fcfab470e7f739819cdc1cba50740470d5cc5b",
         "date": "2023-09-12T13:12:43+02:00",
         "message": "typo",
         "refs": "HEAD -> main, origin/main, origin/HEAD",
         "body": "",
         "author_name": "******",
         "author_email": "*****"
       }
DEBUG: No renovate config file found (repository=<MY_ORG>/<MY_REPO>)
DEBUG: migrateAndValidate() (repository=<MY_ORG>/<MY_REPO>)
DEBUG: No config migration necessary (repository=<MY_ORG>/<MY_REPO>)
DEBUG: Setting hostRules from config (repository=<MY_ORG>/<MY_REPO>)
DEBUG: Found repo ignorePaths (repository=<MY_ORG>/<MY_REPO>)
       "ignorePaths": [
         "**/node_modules/**",
         "**/bower_components/**",
         "**/vendor/**",
         "**/examples/**",
         "**/__tests__/**",
         "**/test/**",
         "**/tests/**",
         "**/__fixtures__/**"
       ]
DEBUG: No vulnerability alerts found (repository=<MY_ORG>/<MY_REPO>)
DEBUG: No vulnerability alerts found (repository=<MY_ORG>/<MY_REPO>)
DEBUG: No baseBranches (repository=<MY_ORG>/<MY_REPO>)
DEBUG: extract() (repository=<MY_ORG>/<MY_REPO>)
DEBUG: Setting current branch to main (repository=<MY_ORG>/<MY_REPO>)
DEBUG: latest commit (repository=<MY_ORG>/<MY_REPO>)
       "branchName": "main",
       "latestCommitDate": "2023-09-12T13:12:43+02:00"
DEBUG: Applying enabledManagers filtering (repository=<MY_ORG>/<MY_REPO>)
DEBUG: Using file match: ^clusters/.*.cue$ for manager regex (repository=<MY_ORG>/<MY_REPO>)
DEBUG: Matched 1 file(s) for manager regex: clusters/schema.cue (repository=<MY_ORG>/<MY_REPO>)
DEBUG: manager extract durations (ms) (repository=<MY_ORG>/<MY_REPO>)
       "managers": {"regex": 8}
DEBUG: Found regex package files (repository=<MY_ORG>/<MY_REPO>)
DEBUG: Found 1 package file(s) (repository=<MY_ORG>/<MY_REPO>)
 INFO: Dependency extraction complete (repository=<MY_ORG>/<MY_REPO>, baseBranch=main)
       "stats": {
         "managers": {"regex": {"fileCount": 1, "depCount": 1}},
         "total": {"fileCount": 1, "depCount": 1}
       }
DEBUG: GET https://api.github.com/repos/<MY_ORG>/<MY_REPO>/contents/charts/addon_versions_index.json = (code=ERR_NON_2XX_3XX_RESPONSE, statusCode=404 retryCount=0, duration=182) (repository=<MY_ORG>/<MY_REPO>)
DEBUG: Failed to look up custom.clusterAddons package grafana_agent_logs (repository=<MY_ORG>/<MY_REPO>, packageFile=clusters/schema.cue, dependency=grafana_agent_logs)
DEBUG: PackageFiles.add() - Package file saved for base branch (repository=<MY_ORG>/<MY_REPO>, baseBranch=main)
DEBUG: Package releases lookups complete (repository=<MY_ORG>/<MY_REPO>, baseBranch=main)
DEBUG: branchifyUpgrades (repository=<MY_ORG>/<MY_REPO>)
DEBUG: detectSemanticCommits() (repository=<MY_ORG>/<MY_REPO>)
DEBUG: getCommitMessages (repository=<MY_ORG>/<MY_REPO>)
DEBUG: semanticCommits: detected "unknown" (repository=<MY_ORG>/<MY_REPO>)
DEBUG: semanticCommits: disabled (repository=<MY_ORG>/<MY_REPO>)
DEBUG: 0 flattened updates found:  (repository=<MY_ORG>/<MY_REPO>)
DEBUG: Returning 0 branch(es) (repository=<MY_ORG>/<MY_REPO>)
DEBUG: config.repoIsOnboarded=true (repository=<MY_ORG>/<MY_REPO>)
DEBUG: packageFiles with updates (repository=<MY_ORG>/<MY_REPO>, baseBranch=main)
       "config": {
         "regex": [
           {
             "deps": [
               {
                 "depName": "grafana_agent_logs",
                 "currentValue": "0.0.24",
                 "datasource": "custom.clusterAddons",
                 "replaceString": ": grafana_agent_logs.#Chart & {\n\t\t\tvalues: \"0.0.24\":",
                 "updates": [],
                 "packageName": "grafana_agent_logs",
                 "versioning": "semver-coerced",
                 "warnings": [
                   {
                     "topic": "grafana_agent_logs",
                     "message": "Failed to look up custom.clusterAddons package grafana_agent_logs"
                   }
                 ]
               }
             ],
             "matchStrings": [
               ":\\s(?<depName>[^\":]+).#Chart\\s&\\s{[\\s\\S]*?values:\\s\"(?<currentValue>[^\"]+)\":"
             ],
             "datasourceTemplate": "custom.clusterAddons",
             "packageFile": "clusters/schema.cue"
           }
         ]
       }
DEBUG: detectSemanticCommits() (repository=<MY_ORG>/<MY_REPO>)
DEBUG: semanticCommits: returning "disabled" from cache (repository=<MY_ORG>/<MY_REPO>)
DEBUG: processRepo() (repository=<MY_ORG>/<MY_REPO>)
DEBUG: Processing 0 branches:  (repository=<MY_ORG>/<MY_REPO>)
DEBUG: Calculating hourly PRs remaining (repository=<MY_ORG>/<MY_REPO>)
DEBUG: getPrList success (repository=<MY_ORG>/<MY_REPO>)
       "pullsTotal": 0,
       "requestsTotal": 1,
       "apiQuotaAffected": true
DEBUG: currentHourStart=2023-09-12T11:00:00.000+00:00 (repository=<MY_ORG>/<MY_REPO>)
DEBUG: PR hourly limit remaining: 2 (repository=<MY_ORG>/<MY_REPO>)
DEBUG: Calculating prConcurrentLimit (10) (repository=<MY_ORG>/<MY_REPO>)
DEBUG: 0 PRs are currently open (repository=<MY_ORG>/<MY_REPO>)
DEBUG: PR concurrent limit remaining: 10 (repository=<MY_ORG>/<MY_REPO>)
DEBUG: Calculated maximum PRs remaining this run: 2 (repository=<MY_ORG>/<MY_REPO>)
DEBUG: PullRequests limit = 2 (repository=<MY_ORG>/<MY_REPO>)
DEBUG: Calculating hourly PRs remaining (repository=<MY_ORG>/<MY_REPO>)
DEBUG: currentHourStart=2023-09-12T11:00:00.000+00:00 (repository=<MY_ORG>/<MY_REPO>)
DEBUG: PR hourly limit remaining: 2 (repository=<MY_ORG>/<MY_REPO>)
DEBUG: Calculating branchConcurrentLimit (10) (repository=<MY_ORG>/<MY_REPO>)
DEBUG: 0 already existing branches found:  (repository=<MY_ORG>/<MY_REPO>)
DEBUG: Branch concurrent limit remaining: 10 (repository=<MY_ORG>/<MY_REPO>)
DEBUG: Calculated maximum branches remaining this run: 2 (repository=<MY_ORG>/<MY_REPO>)
DEBUG: Branches limit = 2 (repository=<MY_ORG>/<MY_REPO>)
DEBUG: ensureDependencyDashboard() (repository=<MY_ORG>/<MY_REPO>)
DEBUG: Closing Dependency Dashboard (repository=<MY_ORG>/<MY_REPO>)
DEBUG: Retrieving issueList (repository=<MY_ORG>/<MY_REPO>)
DEBUG: Retrieved 0 issues (repository=<MY_ORG>/<MY_REPO>)
DEBUG: Removing any stale branches (repository=<MY_ORG>/<MY_REPO>)
DEBUG: config.repoIsOnboarded=true (repository=<MY_ORG>/<MY_REPO>)
DEBUG: No renovate branches found (repository=<MY_ORG>/<MY_REPO>)
DEBUG: PackageFiles.clear() - Package files deleted (repository=<MY_ORG>/<MY_REPO>)
DEBUG: Branch summary (repository=<MY_ORG>/<MY_REPO>)
       "cacheModified": undefined,
       "baseBranches": [{"branchName": "main", "sha": "87fcfab470e7f739819cdc1cba50740470d5cc5b"}],
       "branches": [],
       "defaultBranch": "main",
       "inactiveBranches": []
DEBUG: Renovate repository PR statistics (repository=<MY_ORG>/<MY_REPO>)
       "stats": {"total": 0, "open": 0, "closed": 0, "merged": 0}
DEBUG: Repository result: done, status: onboarded, enabled: true, onboarded: true (repository=<MY_ORG>/<MY_REPO>)
DEBUG: Repository timing splits (milliseconds) (repository=<MY_ORG>/<MY_REPO>)
       "splits": {"init": 3468, "extract": 86, "lookup": 228, "onboarding": 0, "update": 876},
       "total": 4957
DEBUG: Package cache statistics (repository=<MY_ORG>/<MY_REPO>)
       "get": {"count": 0},
       "set": {"count": 0}
DEBUG: http statistics (repository=<MY_ORG>/<MY_REPO>)
       "urls": {
         "https://api.github.com/graphql (POST,200)": 2,
         "https://api.github.com/repos/<MY_ORG>/<MY_REPO>/contents/charts/addon_versions_index.json (GET,404)": 1,
         "https://api.github.com/repos/<MY_ORG>/<MY_REPO>/pulls (GET,200)": 1
       },
       "hostStats": {"api.github.com": {"requestCount": 4, "requestAvgMs": 419, "queueAvgMs": 0}},
       "totalRequests": 4
DEBUG: Package lookup durations (repository=<MY_ORG>/<MY_REPO>)
       "custom.clusterAddons": {"count": 1, "averageMs": 199, "totalMs": 199, "maximumMs": 199}
DEBUG: dns cache (repository=<MY_ORG>/<MY_REPO>)
       "hosts": []
 INFO: Repository finished (repository=<MY_ORG>/<MY_REPO>)
       "cloned": true,
       "durationMs": 4957
DEBUG: Checking file package cache for expired items
DEBUG: Deleted 0 of 0 file cached entries in 2ms
DEBUG: Renovate exiting

Answered by akselleirv

Sep 13, 2023

Fixed by fadf1f2 ! Thanks!

View full answer

rarkins · 2023-09-12T11:38:27Z

rarkins
Sep 12, 2023
Maintainer

Questions:

Is "my-org" the same org as the repo you are running Renovate against? i.e. the two repos are in the same org
Have you installed the app into "my-org/my-repo" too?

13 replies

akselleirv Sep 13, 2023
Author

Hello @rarkins, I saw that you released a new version which seems to be a fix for this issue.

Do you know when this will be pushed to your container registry?

rarkins Sep 13, 2023
Maintainer

All publishing is automated and in public repos, usually takes a few hours

akselleirv Sep 13, 2023
Author

Great, thanks for the quick answer!

akselleirv Sep 13, 2023
Author

Fixed by fadf1f2 ! Thanks!

Answer selected by akselleirv

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Custom data source authentication #24377

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 13 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Custom data source authentication #24377

akselleirv Sep 12, 2023

How are you running Renovate?

If you're self-hosting Renovate, tell us what version of Renovate you run.

If you're self-hosting Renovate, select which platform you are using.

Was this something which used to work for you, and then stopped?

Wanted end result.

What you tried so far.

Relevant debug logs

Replies: 1 comment · 13 replies

rarkins Sep 12, 2023 Maintainer

akselleirv Sep 13, 2023 Author

rarkins Sep 13, 2023 Maintainer

akselleirv Sep 13, 2023 Author

akselleirv Sep 13, 2023 Author

akselleirv
Sep 12, 2023

Replies: 1 comment 13 replies

rarkins
Sep 12, 2023
Maintainer

akselleirv Sep 13, 2023
Author

rarkins Sep 13, 2023
Maintainer

akselleirv Sep 13, 2023
Author

akselleirv Sep 13, 2023
Author