Feedback thread: customDatasources
#23286
Replies: 15 comments 35 replies
-
|
I love the feature! I've already been experimenting with this and it proves to be really useful. One thing, as far as I understand the implementation (see the remaining TODO), it expects the custom registry to return JSON document. In one of my use-cases, the API returns a plain text string that is supposed to represent latest version (GET on /custom api return body "1.0.0" or "123.123.asd-123.zxcase" or even "my-custom-versioning-scheme") In that case, parsing it fails as it's not a valid JSON (I guess because the body is unquoted so it's not treated as string literal). Is that something that can be worked around at a current state? Maybe we could have some kind of configuration option as to how to translate whatever comes from API to JSON literal? |
Beta Was this translation helpful? Give feedback.
-
|
Glad that you like it! Currently this is not possible, but should be rather easy to implement. Would you be interested to implement it, if I write up an feature request issue with the requirements? |
Beta Was this translation helpful? Give feedback.
-
|
sure thing, I think I'll be able to. We could discuss it further down the road :) |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
Idea: allow to run to external binaries to fetch data. The output is then fed into the current logic with Jsonata rules. My use case here is |
Beta Was this translation helpful? Give feedback.
-
|
We can't use aws as a library instead, in a repology like way? But for self-hosted I'm open to the idea of arbitrary CLI commands for datasource lookups |
Beta Was this translation helpful? Give feedback.
-
|
Should be possible, but I'm not sure if |
Beta Was this translation helpful? Give feedback.
-
|
for self-hosted ok, but should be disabled by default like post upgrade tasks |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
I'm new to renovate, I love it, it love the docs, but I feel there's a lot of concepts being thrown at me - I think I would really benefit from this custom data source, so some documentation more specific to my type of use case would help. I feel I just need some url generator, a json parser for the content - its a bit hard to know which of the matchers and templates are relevant. |
Beta Was this translation helpful? Give feedback.
-
|
I really like it. I'm using it to get releases of Hashicorp tools using their API and update Ansible playbooks. |
Beta Was this translation helpful? Give feedback.
-
|
@Demivan great - thank you! Much appreciated! |
Beta Was this translation helpful? Give feedback.
-
|
@SirUli PRs to extend the docs are always welcome. |
Beta Was this translation helpful? Give feedback.
-
|
@Demivan i assume your .yml file then has something along these lines? In my case, renovate detects that there is "something" in the .yml as it lists the file in the dependency dashboard but it doesn't list the dependencies. So apparently i'm doing something wrong there or simply overlook something. I also tried the following (remove "custom" from the datasource): but that didn't help. Any idea? Thanks in advance! |
Beta Was this translation helpful? Give feedback.
-
|
@SirUli With your yml format Here is my yml definition: # renovate: depName=nomad
nomad_version: 1.6.0
|
Beta Was this translation helpful? Give feedback.
-
|
Yehhhaaaawww - not it worked ;) You are right, it was a bit more picky than i anticipated. I made the double quotes to be optionally there and now it went through nicely. Also i had to use loose versioning for my usecase but that is a different story. The final setup for me i In the *.yml files of ansible: # renovate: datasource=custom.nexus_generic depName=something versioning=loose
something_version: "77"In Renovate: And in each of the versiontracker.json files the following content: [
{
"version": "77",
"filelink": "https://nexus.mycompany.com/repository/software/something-77.zip"
},
{
"version": "76",
"filelink": "https://nexus.mycompany.com/repository/software/something-76.zip"
},
]Awesome work by the renovate makers! |
Beta Was this translation helpful? Give feedback.
-
|
I used the custom datasource to extract supported kubernetes versions from Gardener and that worked great. Big thanks for adding this! Especially combining this as well with hostRules for authentication did work fully as expected. Only thing, which did cost me some time and might benefit from improving is the documentation about the Here e.g. the configuration we use to track available and supported kubernetes versions in Gardener: const config = {
hostRules: [
{
matchHost: "https://api.gardener.company.com",
token: "<gardenerToken>",
}
],
customDatasources: {
"gardener-k8s-supported": {
defaultRegistryUrlTemplate: 'https://api.gardener.company.com/apis/core.gardener.cloud/v1beta1/cloudprofiles/aws',
transformTemplates: [
`{"releases": $$.spec.kubernetes.versions[classification = 'supported'].{"version": $.version, "sourceUrl":"https://github.com/kubernetes/kubernetes", "changelogUrl":$join(["https://github.com/kubernetes/kubernetes/releases/tag/",$.version])},"sourceUrl": "https://github.com/kubernetes/kubernetes","homepage": "https://github.com/kubernetes/kubernetes"}`
],
},
"gardener-k8s-available": {
defaultRegistryUrlTemplate: 'https://api.gardener.company.com/apis/core.gardener.cloud/v1beta1/cloudprofiles/aws',
transformTemplates: [
`{"releases": $$.spec.kubernetes.versions.{"version": $.version, "sourceUrl":"https://github.com/kubernetes/kubernetes", "changelogUrl":$join(["https://github.com/kubernetes/kubernetes/releases/tag/",$.version])},"sourceUrl": "https://github.com/kubernetes/kubernetes","homepage": "https://github.com/kubernetes/kubernetes"}`
],
},
}
} |
Beta Was this translation helpful? Give feedback.
-
|
Great feature. I also love that it works with host rules which does not seem to be documented. That enables i.e. fetching data from the gitlab api. |
Beta Was this translation helpful? Give feedback.
-
|
I have copy paste the the same customdatasource between the configs with no success. |
Beta Was this translation helpful? Give feedback.
-
|
Can you share a reproduction repo? |
Beta Was this translation helpful? Give feedback.
-
|
I indeed havent escaped my transformTemplate properly. Thank you very much for giving me the hint. |
Beta Was this translation helpful? Give feedback.
-
|
Not sure why you get this message. A reproduction repo would be helpful here. |
Beta Was this translation helpful? Give feedback.
-
|
I have created very basic example since i cannot share the private repository. When I use a transformTemplate that does not match I get the following error message: |
Beta Was this translation helpful? Give feedback.
-
|
This feature is useful for me to lock aws-for-fluent-bit to stable. https://github.com/aws/aws-for-fluent-bit#guidance-on-consuming-versions
{
"packageRules": [
{
"matchDepNames": ["public.ecr.aws/aws-observability/aws-for-fluent-bit"],
"matchDatasources": ["docker"],
"matchManagers": ["helm-values"],
"enabled": false
}
],
"regexManagers": [
{
"customType": "regex",
"fileMatch": [
"values\\.yaml"
],
"matchStrings": [
"public\\.ecr\\.aws\\/aws-observability\\/aws-for-fluent-bit:(?<currentValue>.*)"
],
"datasourceTemplate": "custom.aws-for-fluent-bit-stable",
"depNameTemplate": "public.ecr.aws/aws-observability/aws-for-fluent-bit"
}
],
"customDatasources": {
"aws-for-fluent-bit-stable": {
"defaultRegistryUrlTemplate": "https://raw.githubusercontent.com/aws/aws-for-fluent-bit/mainline/AWS_FOR_FLUENT_BIT_STABLE_VERSION",
"format": "plain"
}
}
} |
Beta Was this translation helpful? Give feedback.
-
|
Is there a way to debug your datasources? I tried pulling in CoreOS updates via their stream JSON APIs. I validated the transformers against the data in https://try.jsonata.org/, but still Renovate insists that the currently used version is the latest one. "regexManagers": [
{
"fileMatch": [
"template-x86_64.pkr.hcl"
],
"matchStrings": [
"coreos_release = \"(?<currentValue>\\S+)\""
],
"depNameTemplate": "coreos",
"datasourceTemplate": "custom.coreos-stable"
}
],
"customDatasources": {
"coreos-stable": {
"defaultRegistryUrlTemplate": "https://builds.coreos.fedoraproject.org/streams/stable.json",
"transformTemplates": [
"{\"releases\": [{\"version\": $$.(architectures.x86_64.artifacts.metal.release),\"changelogUrl\": \"https://fedoraproject.org/coreos/release-notes/?arch=x86_64&stream=stable\"}]}"
]
}
} |
Beta Was this translation helpful? Give feedback.
-
|
The DEBUG logs confirm that the manager is picking up the dependency, and correctly prints its current version. The repository linked above reproduces the issue, but I recreated it in a separate repo: https://github.com/mhutter/renovate-sandbox |
Beta Was this translation helpful? Give feedback.
-
|
Transformer Expression: https://try.jsonata.org/Gkb9PSPeB |
Beta Was this translation helpful? Give feedback.
-
|
Reproduced: https://github.com/renovate-reproductions/23286-1 It is not a customDatasource issue. |
Beta Was this translation helpful? Give feedback.
-
|
Aha! Would be interesting to have this in the debug logs though. As it DOES output the current version, I just assumed that this was not the issue. Thanks to your input the fix was straight-forward: I love the feature, it is just what I needed right now 👍 |
Beta Was this translation helpful? Give feedback.
-
|
Glad that you like it. A debug log for this scenarios has been added with this PR: #24514 |
Beta Was this translation helpful? Give feedback.
-
|
Very glad for this feature! Thank you! |
Beta Was this translation helpful? Give feedback.
-
|
Yes, it should work the same as with other datasources. The best way to understand your problem would be a new discussion with a reproduction repo. |
Beta Was this translation helpful? Give feedback.
-
|
Not sure if related, but I noticed our custom datasources breaking since fadf1f2, which added e.g. to the renovate/lib/util/http/auth.ts Lines 31 to 36 in 69f7240
Edit: Created #24671 as follow up |
Beta Was this translation helpful? Give feedback.
-
|
I've returned and managed to get a lot further. A few things on my wish list:
To give an example, my latest struggle: All I had was: "Failed to look up custom.X package Y". |
Beta Was this translation helpful? Give feedback.
-
|
Config to update an (ivy) dependency, using the artifactory API, hope it helps someone |
Beta Was this translation helpful? Give feedback.
-
|
Regarding docs, you can find them for the custom manager here: https://github.com/renovatebot/renovate/tree/main/lib/modules/datasource/custom Currently we have only special logging in the custom datasource for schema issues. If you can give us some specific what information would have helped you, we can consider it.
|
Beta Was this translation helpful? Give feedback.
-
An Odyssey: Trying To Use Custom Datasources To Query APT-RepositoriesSo, we have two repositories on GitLab:
Renovate works perfectly fine for a large portion of our dependencies. We can update Docker image tags in Dockerfiles, Gitlab CI files, and Helm values.yml files. We can update Helm chart versions, Pipenv dependencies, and Terraform and Terragrunt dependencies. Great experience all around. However, we also had to setup a {
customType: "regex",
fileMatch: [
"(^|/|\\.)([Dd]ocker|[Cc]ontainer)file$",
"(^|/)([Dd]ocker|[Cc]ontainer)file[^/]*$"
],
matchStrings: [
"#[ \\t]+Renovate:[ \\t]+datasource=(?<datasource>[.-_a-z]+?)[ \\t]+depName=(?<depName>\\S+?)(?:[ \\t]+(lookupName|packageName)=(?<packageName>\\S+?))?(?:[ \\t]+versioning=(?<versioning>\\S+?|regex:\\/.*?\\/))?(?:[ \\t]+extractVersion=(?<extractVersion>\\S+?))?(?:[ \\t]+registryUrl=(?<registryUrl>\\S+?))?(\\r|\\n|\\r\\n)(?:ENV|ARG)[ \\t]+.+?_VERSION[ =]\"?(?<currentValue>\\S+?)\""
],
versioningTemplate: "{{#if versioning}}{{{versioning}}}{{else}}semver{{/if}}"
}This is basically the example from the docs with some adjustments. I.e., it allows for something like # Renovate: datasource=repology depName=ubuntu_22_04/git
ENV GIT_VERSION="2.34.1"in our Dockerfile. Again, where we can use built-in datasources, this works fine. Tools from custom APT repositories - Approach 1Here's the issue: We build a Docker image that contains tools which are installed from custom APT repositories (e.g., So my first workaround was this: Renovate can use GitHub releases and Git tags as a datasource. All the tools we install are available from accessible repositories so we can examine them and use the release names / Git tags as versions. This didn't work because the names of the releases / tags don't correspond directly to the version notation used in the APT repositories. We tried to map versions (which worked but was very messy) but then you run into the issue that versions published on GitHub take a while until they show up in the APT repositories. So Repology would propose an updated Dockerfile which then couldn't be built because the version wasn't present in the APT repository.
Tools from custom APT repositories - Approach 2So, since Renovate doesn't have such a datasource, I tried using the Right, so my approach is this:
Here come the issues with this approach: The only template variable that we have access to in # Renovate: datasource=custom.local_apt_versions_file depName=google-cloud packageName=https://packages.cloud.google.com/apt/cloud-sdk/main/google-cloud-cli versioning=deb
ENV GCLOUD_VERSION="462.0.1-0"Of course, the
Now, Renovate and But lets run with it, everything after the
So I put my files into the base folder, I verified that the file URL was correct given how it was handled, ran Renovate and... nothing. The files were gone. Well, turns out that Renovate will
Tools from custom APT repositories - Approach 3So we can't use Of course, accessing these artifacts requires an authentication header. Renovate already has a {
matchHost: `${process.env.RENOVATE_ENDPOINT}/projects/${process.env.PROJECT_ID}/jobs/${process.env.APT_VERSIONS_JOB_ID}/artifacts/${process.env.APT_VERSIONS_FOLDER}`,
token: `${process.env.TOKEN}`,
authType: "Bearer"
}I run Renovate and... nothing. I get a 404 for the URL which in this case indicates an invalid authentication header. If the header had the correct format but the token was wrong, I would get 401. By the way, the trace log shows "Adding Bearer authentication for URL" but this is not done.
Finally, I tried to work around this issue by adding the authentication header myself. I added allowedHeaders: [
"Authorization",
"X-*"
]and to the headers: {
Authorization: `Bearer ${process.env.TOKEN}`
}This workaround also doesn't work. I don't know why, I stopped investigating due to time constraints at this point. I think it's because the repository we are working with is also on GitLab and Renovate internally adds hostRules to handle authentication. The
Conclusion / TLDRI'd appreciate if you would take the time to look at this @secustor and tell me where / if I went wrong here. For now, we can't use Renovate to update APT backed dependencies. I think my approach should be doable but I am blocked by the issues that I described. My proposals for remediation are quoted. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the walk through and sry for coming back this late. Happy to review a PR for such a datasource. For such usecases I can see adding a Forwarding all capture groups is from an architectural point not so desired as managers and datasources are only lightly coupled. The custom datasource is currently gated to files inside of the same repository and references are always in reference to that root. Create a separate discussion with a reproduction for this, tough the OTEL integration may help you debug this. |
Beta Was this translation helpful? Give feedback.
-
|
Hi there, Please do not unnecessarily For example, never It's OK to comment in an issue or discussion after multiple days or weeks. But please, still don't Thanks, the Renovate team |
Beta Was this translation helpful? Give feedback.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
-
|
Found an additional interesting use case for k3s https://github.com/visualon/renovate-config/pull/10/files |
Beta Was this translation helpful? Give feedback.
-
|
Custom datasources have been very helpful to us. Some general remarks:
Some of our usecases:
|
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
Tell us more.
Tell us how you like the new
customDatasourcesexperimental feature. What works for you, what doesn't work for you? Do you see anything we could improve?Let us know what you think! 😉
Beta Was this translation helpful? Give feedback.
All reactions