Not all plugins are updated

[nre-ableton]

I would like to use the plugin-installation-manager-tool to update plugins on a Jenkins instance, which seems to kind of work, but not completely. I am running the tool from Ansible like so (forgive the Jinja2 templating):

java -jar /opt/jenkins-pimt/jenkins-plugin-manager-1.0.2.jar -w {{ jenkins_war }} -f {{ jenkins_home }}/plugins.yaml -d {{ jenkins_home }}/plugins --available-updates --latest

After running the above command, Jenkins is restarted (just to be safe!), and then the tool is run again with the --no-download --list arguments. The output gets parsed by a separate Python script which regenerates plugins.yaml. In my local git diff, I can see that a number of plugins were successfully updated, such as credentials from 2.3.1 to 2.3.3. However, when I go to Jenkins' plugin management page, I see a number of other plugins still want to be updated:

If I re-run my Ansible playbook, these plugins don't get updated. I can't seem to figure out why some plugins can be updated but others cannot. All of the above plugins are listed in the plugins.yaml file. Why don't they get updated?

[timja]

Can you supply the plugins you have installed?

Are you including the above plugins in your plugins.yaml?

I'm guessing you aren't and they are optional dependencies and so there's an issue picking those up,

but would need the plugins list to reproduce it

[nre-ableton]

I should note, I have also tried running the script with both --latest-specified, but the results are the same.

I've attached our plugins.yaml file. Sorry, I should have mentioned, that all of the plugins shown in the screenshot are listed there. We actually track all plugins in our plugins.yaml. The screenshot is also only a partial listing; there are about a dozen plugins which don't get updated.

jenkins-plugins.yml.txt

[timja]

@stopalopa does this issue ring any bells?

[stopalopa]

I think it's because the settings --latest and --latest-specified have to do with which version of the dependencies get installed. If you specify an exact version of a plugin, then the tool will try to download that one. For example, @nre-ableton specified the version of icon-shim as 1.0.3 in the jenkins-plugins.yml.txt file, so that is the version the tool downloaded.

Maybe it is confusing that the CLI parameters are only for dependencies? But if you want the newest-version of any plugin, you can opt to not put in a version in the yml file and the latest version of that listed plugin would be downloaded.

[timja]

I think it's because the settings --latest and --latest-specified have to do with which version of the dependencies get installed. If you specify an exact version of a plugin, then the tool will try to download that one. For example, @nre-ableton specified the version of icon-shim as 1.0.3 in the jenkins-plugins.yml.txt file, so that is the version the tool downloaded.

Maybe it is confusing that the CLI parameters are only for dependencies? But if you want the newest-version of any plugin, you can opt to not put in a version in the yml file and the latest version of that listed plugin would be downloaded.

right I think this goes back to some discussions awhile ago about a 'lock' file

[nre-ableton]

Maybe it is confusing that the CLI parameters are only for dependencies? But if you want the newest-version of any plugin, you can opt to not put in a version in the yml file and the latest version of that listed plugin would be downloaded.

Well, we definitely need to have the plugin version in the YAML file, so removing it is not an option for us. However, we can always write plugins.yaml to a temporary location and remove all of the version: attributes to update. This is a bit more work than I was imagining, but it's a good solution.

I'm still a bit confused, though... I thought that we were specifying all versions in the plugin.yaml file. For instance, we also specify the version of inedo-proget, but it doesn't get updated. icon-shim had a version specified, but it did get updated.

right I think this goes back to some discussions awhile ago about a 'lock' file

Yes, we are essentially treating plugins.yaml as a lockfile. This might not be the right way to do it, but in the past when we only specified top-level plugin versions, we ended up with weird incompatibilities and could not be 100% certain that our staging Jenkins instance had exactly the same plugins installed as what we would end up deploying to production. So we made the decision to track everything in plugins.yaml.

[stopalopa]

Well....plugin dependencies and versions are complicated. Without digging into it further, I'm not sure if you're seeing a bug or not. But suppose you list plugin A v1 and plugin B v3 in your yaml file and plugin B v3 depends on plugin A v2. In this case the expected behavior of the tool would be download plugin A v2 instead of plugin A v1 because it takes the highest required version, even if that version is higher than what you specified. So what you might be seeing is that some of the versions are upgraded because a higher version is required by some other plugin.

There have been some amount of discussions about how to replicate the downloaded plugins. There's some tension between reproducible results and downloading the latest versions, since the latest will change. By default the tool will download the dependency versions listed in the plugin meta data, which ensures reproducibility, but these versions are conservative and will likely result in out-of-date plugins, which is also not helpful.

We probably should look into a lock file or some better way to ensure reproducibility as @timja mentioned. Another thing I think would be beneficial for this project is somehow incorporating maven's dependency management, since dependency management from scratch ended up being far more complicated than I realized when I started this project last summer.

[nre-ableton]

Yeah, dependency management is tough, I understand that. I also appreciate that different users want different things: some (like myself) value reproducible results, others want the latest versions. I think that a lockfile is a great way to solve the problem, since you wouldn't need one if you always wanted the latest version.

BTW I'm ok if this issue gets closed, but is there another issue for the lockfile? If not, I'd be happy to create one. If it's something that's only been lightly discussed, I think that there should be a GH issue for interested persons to follow and comment upon.

[timja]

no issue that I'm aware of,

I think it would be better exposed as a feature / user story, (and yes please raise it)

I think the ask is:

I want to be able to update my plugins file with the latest versions of everything without having to do it by hand

[nre-ableton]

Great! Let's close this issue in favor of #88.