Move registryUrls handling into datasource

[rarkins]

What would you like Renovate to be able to do?

Shift the complexity of trying multiple registries into the shared datasource logic instead of re-implemented into each datasource type.

Describe the solution you'd like

Default behavior for multiple datasources should be "hunt" - i.e. try the registryUrls in order and return as soon as there's one result. An alternative would be "merge" or "combine" - i.e. try them all and combine the results. If a datasource needs "combine" it would be an exported configuration option.

Describe alternatives you've considered

Today each datasource implements/re-implements hunt or merging logic on its own.

Additional context

If each datasource's getReleases() logic needs to take into consideration only one registryUrl then it will also improve our caching ability for times when there's a mix of public and private results.

[rarkins]

@JamieMagee does this conflict with any of the ideas you've had about Datasource classes?

[rarkins]

First step we need to do before moving this to ready is to survey which datasources support multiple registryUrls today, how they support them, and if there are any "spanners in the works" I haven't thought of that would make this too hard in their case.

[rarkins]

• docker: uses the first one only
• git-submodules: seems to abuse registryUrls into something it's not meant to be??
• gitlab-tags: uses the first one only
• gradle-version: merges multiple
• helm: uses the first one only
• maven: combines multiple
• nuget: combines multiple
• packagist: hunts multiple
• pod: seems to take the last one today (intentional?)
• pypi: also takes the last one?
• rubygems: hunts
• sbt-package: hunts
• sbt-plugin: hunts
• terraform-module: takes the first

[rarkins]

Datasources can optionally export a new field registryStrategy with values of:

• none (no registryUrl needed)
• first (only the first url is used)
• hunt (try registry urls in sequence)
• merge (try all registry urls and merge the results)

[rarkins]

This also gives us the possibility to record which registry/registries that results were found on, although I haven't thought of a use for that.

[JamieMagee]

I don't think my work with datasources should be affected by this. Rather, I think it might actively help us to achieve this.

I have been sitting on the conversion of the cndjs for over a week now. I should tidy it up and send it out for (draft) PR and get some feedback on the approach.

[rarkins]

Haha, I was also starting with cdnjs

[rarkins]

It's actually not the perfect one for me to start with though because it's one of the datasources that is best doing its own caching instead of delegating it to the main datasource.

[rarkins]

Waiting on #6538

[renovate-release]

🎉 This issue has been resolved in version 21.13.0 🎉

The release is available on:

• GitHub release
• 21.13.0

Your semantic-release bot 📦🚀