apt module, fix issue with packages not installing when pinned and cached

[jhooda]

SUMMARY

Currently when apt cache holds multiple versions of the same package, including newer versions and older versions, a specific installation of the package version (other than the latest version) fails with an error message:

no available installation candidate for pkgname=x.y.z

However, the command line such as

$ apt install pkgname=x.y.z

works just fine, the reason being some issue in apt_cache.Policy which fails to override the default priority.

When set_priority() is called before create_pin then it works as expected.

fixes #80813

ISSUE TYPE

• Bugfix Pull Request

COMPONENT NAME

modules/apt

[bcoca]

needs changelog and tests

[jhooda]

@bcoca I need some help regarding the unit tests for this issue. For the unit test, I need multiple versions of a test debian package (e.g., https://ci-files.testing.ansible.com/test/integration/targets/dpkg_selections/hello_2.6-1_amd64.deb). Since, I don't have access to https://ci-files.testing.ansible.com I am not able to see what's available there. Can you please provide URLs for some test debian package that has at least three versions and which will not be wiped out in future. Thanks.

[bcoca]

no, we have none, i would recommend adding them as part of the test .. also this is something i would expect in an integration test, not a unit one.

[bcoca]

cc @mattclay

[bcoca]

The apt tests use a setup_deb_repo which creates packages for foo and foobar and automatically sets up the repo so they can be installed from with these versions: foo-1.0.0 foo-1.0.1 foobar-1.0.0 foobar-1.0.1

[jhooda]

@bcoca I've tested foobar pkg (even with one additional version foobar-1.0.2) and that worked okay even without this pull request. So not sure how to reproduce the issue using foobar. I've added some tests to this pull request. My guess is the issue has something to do with non-std pkg versions such as '1.9.0-71-g9c1898a'. I'll try some more tests later.

[jhooda]

Also to explain the underlying issue, see lines:265-268 in
https://github.com/Debian/apt/blob/aa56836331870d975c212a5df2f13db9ce3914bf/apt-pkg/policy.cc#L265-L268

	    for (pkgCache::VerIterator Ver = Pkg.VersionList(); Ver.end() != true; ++Ver)
	    {
	       if (Match.VersionMatches(Ver)) {
		  Pin *VP = VerPins + Ver->ID;
		  if (VP->Type == pkgVersionMatch::None) {
		     *VP = P;
		      matched = true;
		  }
	       }
	    }

The priority pins are only updated if VP-Type is None, which in this case is 'Version'. When we manually invoke setPriority, it sets the pin Type to None, and which is why it works (lines:363-369)

void pkgPolicy::SetPriority(pkgCache::VerIterator const &Ver, signed short Priority)
{
   Pin pin;
   pin.Data = "pkgPolicy::SetPriority";
   pin.Priority = Priority;
   VerPins[Ver->ID] = pin;
}

[webknjaz]

@jhooda the CI failures look related to your patch: https://dev.azure.com/ansible/ansible/_build/results?buildId=86661&view=logs&j=26a6818c-7a55-5301-191d-87f9382f47e1&t=b498074a-badf-5058-76b3-ae0756883986&l=2586 — you need to fix them for the PR to be mergable.

[jhooda]

@webknjaz I reviewed the logs. They seem unrelated to my changes: my tests are further down from the failure point and the failure is in an unrelated "apt cache update" feature. I can try rebasing the pull request with the latest from the devel branch. Please suggest the way forward. Just for records "ansible [core 2.12.10]" does not suffer from this issue.

[webknjaz]

my tests are further down from the failure point and the failure is in an unrelated "apt cache update" feature.

This effectively means that your patch has side effects beyond what you expected, and it so happens that it breaks existing tests. Before this patch, policy.get_candidate_ver(pkg) was always called for = and now the control flow never reaches it in this case. This is a behavior change, and it does change how cache is used.
The failing test — https://github.com/ansible/ansible/blob/bd3ffbe/test/integration/targets/apt/tasks/apt.yml#L99-L104 — is a cache test, no wonder it got affected.

If you're sure the patch is correct, the changes necessary might be in the tests, not in the module.

[jhooda]

@webknjaz thanks for the suggestions. I have slightly changed apt.py and squashed commits too. Let's see if this helps. Thanks.

[jhooda]

@webknjaz I ran the test on my local Ubuntu22.04 LTS (devel branch - with last commit: f10d11b) and it failed exactly at the same place, so something else is causing the failure. The test which I ran

$ cat /etc/issue
Ubuntu 22.04 LTS

$ python3 ./bin/ansible-test integration --allow-destructive apt
...
TASK [apt : verify update_cache] ***********************************************
fatal: [testhost]: FAILED! => {
    "assertion": "apt_update_cache_1 is changed",
    "changed": false,
    "evaluated_to": false,
    "msg": "Assertion failed"
}

[webknjaz]

I ran the test on my local Ubuntu22.04 LTS

Try running with --docker to reproduce the exact CI env.

[webknjaz]

I wonder if this call modifies cache… Or maybe set_priority() does?

[webknjaz]

@jhooda could you rebase the PR branch, just in case the recent Launchpad API problems were related to the failure?

[Akasurde]

@jhooda Could you please rebase this branch? Thanks in advance.

[timon-michel-scopevisio]

@jhooda Any updates on the rebase? I just came across this issue while installing nodejs using nodesource, as it pins the repository origin:

Package: nodejs
Pin: origin deb.nodesource.com
Pin-Priority: 600