Ansible Galaxy Collections: using a git source doesn't work

[TheLastProject]

Type of discussion.

I'm proposing an idea

Tell us more.

When pointing renovate to an Ansible Galaxy collection managed through git (by having an entry in requirements.yml as such):

collections:
  - name: community.docker
    source: git@gitlab.<removed>:<removed>/mirrors/ansible-collections/community.docker
    type: git
    version: 2.7.5

Renovate incorrectly complains that "Custom registries are not allowed for this datasource and will be ignored"

{"name":"renovate","hostname":"<removed>","pid":8,"level":40,"logContext":"<removed>","repository":"<removed>","datasource":"git-tags","registryUrls":["git@gitlab.<removed>:<removed>/mirrors/ansible-collections/community.docker"],"defaultRegistryUrls":null,"msg":"Custom registries are not allowed for this datasource and will be ignored","time":"2023-04-18T08:53:54.136Z","v":0}

(Original source for this mirror is git@github.com:ansible-collections/community.docker.git, which will likely also trigger this issue)

The relevant renovate config is fairly default, so I doubt that to be relevant, but better more info than less info:

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": ["config:base"],
  "packageRules": [
    {
      "matchDatasources": ["pypi"],
      "registryUrls": ["https://<removed>/pypi"]
    },
    {
      "matchDatasources": ["docker"],
      "registryUrls": ["https://<removed>"]
    }
  ]
}

While this issue seems related to #14799 (as both the source value is set and the error message is the same), @viceice believes this is a different issue and suggested creating a new discussion instead in #14799 (comment).

[rarkins]

Are you able to reproduce this in a public repo?

[TheLastProject]

Yes, see https://github.com/TheLastProject/renovate_issue_21580_remake/pull/1.

The bot errors with

- Failed to look up git-tags package community.docker

On this very simple https://github.com/TheLastProject/renovate_issue_21580_remake/blob/main/requirements.yml.

It should instead offer an update to, as of writing, version 3.4.3.

Edit: Looks like I might have broken @renovate-bot by force-pushing a typo fix (which closed the PR) and re-creating the repo to try to get Renovate to make a new PR. Let's give it some time to hopefully recover properly...

Edit 2: I just made a new repo instead and updated the links 🤷‍♀️

[rarkins]

Please check app.renovatebot.com for the logs and copy paste here the result for packageFiles with updates

[TheLastProject]

There are no logs available on app.renovatebot.com whatsoever:

(My repo isn't even showing on the left side, only Nextcloud repos, so I followed the link on https://github.com/TheLastProject/renovate_issue_21580_remake/pull/1)

[rarkins]

Please use the Renovate app instead of Forking Renovate

[TheLastProject]

Okay, then see https://github.com/TheLastProject/renovate_issue_21580_remake/pull/2.

- Failed to look up git-tags package community.docker

Logs you asked for:

Select me to see logs

DEBUG: packageFiles with updates
{
  "baseBranch": "main",
  "config": {
    "ansible-galaxy": [
      {
        "deps": [
          {
            "depType": "galaxy-collection",
            "registryUrls": [
              "git@github.com:ansible-collections/community.docker"
            ],
            "currentValue": "2.7.5",
            "depName": "community.docker",
            "datasource": "git-tags",
            "updates": [],
            "packageName": "community.docker",
            "warnings": [
              {
                "topic": "community.docker",
                "message": "Failed to look up git-tags package community.docker"
              }
            ],
            "versioning": "semver"
          }
        ],
        "packageFile": "requirements.yml"
      }
    ]
  }
}

Logs that also seem meaningful:

Select me to see logs

DEBUG: manager extract durations (ms)
{
  "managers": {
    "ansible-galaxy": 2
  }
}
DEBUG: Found ansible-galaxy package files
DEBUG: Found 1 package file(s)
INFO: Dependency extraction complete
{
  "baseBranch": "main",
  "stats": {
    "managers": {
      "ansible-galaxy": {
        "fileCount": 1,
        "depCount": 1
      }
    },
    "total": {
      "fileCount": 1,
      "depCount": 1
    }
  }
}
WARN: Custom registries are not allowed for this datasource and will be ignored
{
  "datasource": "git-tags",
  "registryUrls": [
    "git@github.com:ansible-collections/community.docker"
  ],
  "defaultRegistryUrls": null
}
WARN: Attempting to use non-git url for git operations
{
  "url": "community.docker"
}
DEBUG: Datasource unknown error
{
  "datasource": "git-tags",
  "packageName": "community.docker",
  "err": {
    "task": {
      "commands": [
        "ls-remote",
        "community.docker"
      ],
      "format": "utf-8",
      "parser": "[function]"
    },
    "message": "fatal: 'community.docker' does not appear to be a git repository\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n",
    "stack": "Error: fatal: 'community.docker' does not appear to be a git repository\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n\n    at Object.action (/home/ubuntu/renovateapp/node_modules/simple-git/dist/cjs/index.js:1274:25)\n    at PluginStore.exec (/home/ubuntu/renovateapp/node_modules/simple-git/dist/cjs/index.js:1309:29)\n    at /home/ubuntu/renovateapp/node_modules/simple-git/dist/cjs/index.js:1676:43\n    at new Promise (<anonymous>)\n    at GitExecutorChain.handleTaskData (/home/ubuntu/renovateapp/node_modules/simple-git/dist/cjs/index.js:1674:16)\n    at GitExecutorChain.<anonymous> (/home/ubuntu/renovateapp/node_modules/simple-git/dist/cjs/index.js:1658:44)\n    at Generator.next (<anonymous>)\n    at fulfilled (/home/ubuntu/renovateapp/node_modules/simple-git/dist/cjs/index.js:55:24)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)"
  }
}
DEBUG: Failed to look up git-tags package community.docker (community.docker)(packageFile="requirements.yml", dependency="community.docker")

[github-actions[bot]]

Hi there,

We found a problem with the logs. Depending on which situation applies follow one, some or all of these instructions.

No logs at all

If you haven't posted any log yet, we need you to find and copy/paste the log into the issue template.

Finding logs on hosted app

Select me to read instructions

If you use the Renovate app (GitHub):

1. Go to the affected PR, and search for "View repository job log here"
2. Select the link to go to the "Mend Renovate Dashboard" and log in
3. You are now in the correct repository log overview screen
4. Copy/paste the correct log
5. Follow the steps in the formatting your logs section

Finding logs when self-hosting

Select me to read instructions

If you're running self-hosted, run with LOG_LEVEL=debug in your environment variables and search for whatever dependency/branch/PR that is causing the problem.

Insufficient logs

Select me to read instructions

If you already gave us a log, and the Renovate team said it's not enough, then follow the instructions from the No logs at all section.

Formatting your logs

Select me to read instructions

Please put your logs in a <details> and <summary> element like this:

<details><summary>Select me to see logs</summary>

```
Copy/paste your logs here, between the starting and ending backticks
```

</details>

Good luck,

The Renovate team

[rarkins]

Maybe fixed by #21640

FYI @secustor, I think this is your original code - please check

[secustor]

This is a missing feature, which should be implemented at:

renovate/lib/modules/manager/ansible-galaxy/collections.ts

Line 47 in 6341703

function handleGitDep(

Currently if the type is git and the git reference is not in the name it fails.

[mcaulifn]

For the case where the source starts with https such as below:

---
collections:
  - name: mynamespace.mycollection
    version: v1.1.0
    type: git
    source: https://github.com/mynamespace/mycollection-repo.git
roles: []

Would this need to be if git or https?

Requirements file reference

[secustor]

No, it should be:

-       if (value?.startsWith('git@')) {
+       if (value?.startsWith('git@') || value?.endsWith('.git')) {

[secustor]

#25568

[mcaulifn]

Thank you!