Rubygems with Artifactory doesn't seem to work

[henrysachs]

Which Renovate are you using?

Renovate Open Source CLI

Which platform are you using?

GitLab self-hosted

Have you checked the logs? Don't forget to include them if relevant

 DEBUG: Datasource 404 (repository=myrepo)
        "datasource": "rubygems",
        "lookupName": "sfn_my_dep1",
        "url": "https://hub.tech.my.domain.de/api/v1/gems/sfn_my_dep1.json"
 DEBUG: Failed to look up dependency sfn_my_dep1 (repository=myrepo, packageFile=Gemfile, dependency=sfn_my_dep1)

What would you like to do?

I wanted to scan ruby gems from a private artifactory. In the Gemfile i face 2 issues.

1. When using the source do syntax renovate cli doesnt detect those dependencies they arent even mentioned in the depcount

sample gemfile

gem 'sfn', :source => 'https://hub.tech.my.domain.de/artifactory/api/gems/rubygems-org-remote/'

gem 'sfn_my_dep1', '~> 1.1', :source => 'https://hub.tech.my.domain.de/artifactory/api/gems/my-gems-prod-local/'

source 'https://hub.tech.my.domain.de/artifactory/api/gems/my-gems-prod-local/' do
    gem 'sfn_my_dep1', "~> 1"
    gem 'sfn_my_dep2', "~> 1"

end

this is the output: "bundler": {"fileCount": 1, "depCount": 2}

when renovate checks these gems it tries to request them from the wrong url atleast this is what i think. But with debug logging enabled the config seems about right for me:

DEBUG: packageFiles with updates (repository=myrepo)
        "config": {
          "bundler": [
            {
              "packageFile": "Gemfile",
              "manager": "bundler",
              "registryUrls": [
                "https://hub.tech.my.domain.de/artifactory/api/gems/my-gems-prod-local/"
              ],
              "deps": [
                {
                  "depName": "sfn",
                  "managerData": {"lineNumber": 1},
                  "skipReason": "no-version",
                  "depIndex": 0,
                  "updates": []
                },
                {
                  "depName": "sfn_mydep1",
                  "managerData": {"lineNumber": 3},
                  "currentValue": "~> 1.1",
                  "datasource": "rubygems",
                  "depIndex": 1,
                  "updates": [],
                  "warnings": [
                    {
                      "depName": "sfn_mydep1",
                      "message": "Failed to look up dependency sfn_mydep1"
                    }
                  ]
                }
              ]
            }
          ],

thats currently my dead end where i don't know what i could further do to trace this.

i'm happy to gather more information if necessary. But i need to anonymize the data a bit so expect a typo.

[rarkins]

For the extraction problem, please raise that as a bug in the main repo, ideally with a simple reproduction repo.

For the lookup, can you confirm if the URL is correct by querying it manually with authentication? And how have you configured ruby gems authentication for Renovate?

[henrysachs]

Hi i the url is wrong but it seems to be coming from renovate because i found out artifactory as a rubygems repo has the following endpoint for rubygems:
https://hub.tech.my.domain.de/artifactory/api/gems/my-gems-prod-local/api/v1/gems/sfn_mydep1.json
is there a way i can tell renovate how to use this url structure for the rubygems datasource?

authentication isnt necessary for this repo because it allows anonymous installs but it is behind a vpn.

see also: https://www.jfrog.com/confluence/display/JFROG/RubyGems+Repositories

my config looks like this:

module.exports = {
  extends: [
    "docker:enableMajor",
    ":semanticCommits",
    "group:monorepos",
    "group:recommended",
    ":preserveSemverRanges"
  ],
  platform: "gitlab",
  endpoint: "https://git.tech.rz.db.de/api/v4",
  token: `${process.env.GIT_USER_API_TOKEN}`,
  logLevel: `debug`,
  labels: ["renovate", "seec-mop"],
  ignoreDeps: ["docker-hub-remote.bahnhub.tech.rz.db.de/renovate/renovate"],
  assigneesFromCodeOwners: true,
  assigneesSampleSize: 5,
  lockFileMaintenance: { enabled: true },
  recreateClosed: true, //probably debug config
  requireConfig: false,
  onboarding: false,
  requiredStatusChecks: null,
  packageRules: [
    {
      updateTypes: ["minor", "patch", "pin", "digest"],
      gitlabAutomerge: false,
      automerge: false,
    },
    {
      updateTypes: ["major"],
      labels: ["major","renovate", "seec-mop"],
    },
    {
      datasources: ["docker"],
      updateTypes: ["major","minor", "patch", "pin", "digest"],
      enabled: true
    }
  ],
  vulnerabilityAlerts: {
    labels: ["security"],
    enabled: true,
  },
  npmrc: `registry=https://bahnhub.tech.rz.db.de/artifactory/api/npm/default-npm-3rdparty/\nalways-auth=true\n_auth=${process.env.ARTIFACTORY_TOKEN}`,
  pinDigests: true,
  hostRules: [
    {
      hostType: "rubygems",
      username: `${process.env.ARTIFACTORY_USERNAME}`,
      password: `${process.env.ARTIFACTORY_PASSWORD}`,
      baseUrl:
        "https://bahnhub.tech.rz.db.de/artifactory/api/gems/seec-serviceteam-1-gems-prod-local",
    },
    {
      hostType: "rubygems",
      username: `${process.env.ARTIFACTORY_USERNAME}`,
      password: `${process.env.ARTIFACTORY_PASSWORD}`,
      baseUrl:
        "https://bahnhub.tech.rz.db.de/artifactory/api/gems/seec-mov-gems-prod-local",
    },
  ],
};

PS: data is now real because it seems to be fine with my company

[henrysachs]

https://github.com/renovatebot/renovate/blob/master/lib/datasource/rubygems/releases.ts
if i understand this correctly there needs to be an artifactory specific dependency checker. maybe i can help with that.

[rarkins]

Yes, I think it sounds like we need some code changes to our rubygems datasource, and any help on this would be greatly appreciated. I would hope that the code is not necessarily artifactory specific but it seems it's a URL structure we don't support yet. I'm guessing there's some overlap with other rubygems queries we've had: https://github.com/renovatebot/renovate/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+rubygems

I will rename and transfer this issue into the main repo

[rarkins]

Comparison of attempted (failing) vs proper URL:

Failing: https://hub.tech.my.domain.de/api/v1/gems/sfn_my_dep1.json
Working: https://hub.tech.my.domain.de/artifactory/api/gems/my-gems-prod-local/api/v1/gems/sfn_mydep1.json

The registryUrls entry is https://hub.tech.my.domain.de/artifactory/api/gems/my-gems-prod-local/ so maybe we are just doing a bad URL join here:

renovate/lib/datasource/rubygems/get.ts

Lines 17 to 30 in caad3cf

	export async function fetch(
	dependency: string,
	registry: string,
	path: string
	): Promise<any> {
	const headers = getHeaders();
	const name = `${path}/${dependency}.json`;
	const baseUrl = ensureTrailingSlash(registry);
	logger.trace({ dependency }, `RubyGems lookup request: ${baseUrl} ${name}`);
	const response = (await http.getJson(name, { baseUrl, headers })) || {
	body: undefined,
	};

?

I think we're calling http.getJson() with path set to /api/v1/gems/sfn_my_dep1.json and baseUrl set to https://hub.tech.my.domain.de/artifactory/api/gems/my-gems-prod-local/.

[henrysachs]

as the log states above the base url is set to https://hub.tech.my.domain.deif i understand that correctly.

[rarkins]

[henrysachs]

so can we do something like if path includes artifactory? baseurl + path?

[rarkins]

Actually I think it should be a fix in our HTTP layer, independent of artifactory or rubygems. PR in progress

[henrysachs]

that would be even better. Do you think this is a bigger problem to tackle? I would like to keep track of the PR :)

[henrysachs]

just to clear more things for artifactory (dont know about rubygems or other) when requesting the gem the url is like this: https://bahnhub.tech.rz.db.de:443/artifactory/seec-serviceteam-1-gems-prod-local/gems/sfn_seec_base-1.4.2.gem
so im not quite sure how the library handles the download when it received the json. Does it take the field gem_uri?

sample json:

{
  "name" : "sfn_seec_base",
  "version" : "1.4.2",
  "platform" : "ruby",
  "authors" : "",
  "info" : "",
  "gem_uri" : "https://bahnhub.tech.rz.db.de:443/artifactory/seec-serviceteam-1-gems-prod-local/gems/sfn_seec_base-1.4.2.gem",
  "homepage_uri" : "",
  "dependencies" : {
    "development" : [ ],
    "runtime" : [ {
      "name" : "sfn",
      "requirements" : ">= 3.0, < 4.0"
    } ]
  }
}

[rarkins]

I think we already should have the URL correct, it's just that we were joining the URL incorrectly. See #7316 for what is hopefully the only fix required

[renovate-release]

🎉 This issue has been resolved in version 23.31.1 🎉

The release is available on:

• GitHub release
• 23.31.1

Your semantic-release bot 📦🚀

[rarkins]

@henrysachs can you test out 23.31.1 ?

[henrysachs]

will do as far as i saw there is no docker container currently for this right?

[rarkins]

Yes, there's a multi-step build process so it will need to pop up in an hour or two

[henrysachs]

i'm getting an unauthorized now. So there seems to be progress need to dig deeper into that myself because im sure thats on my side. But the Url for the json is now correct. Thanks a lot!

[rarkins]

OK, please create a new issue if you think there's something needing changing from Renovate