reproduce Testing Farm's package installation

[martinpitt]

I've been debugging a weird failure in cockpit-project/cockpit-machines#601. Long story short, it was because the packit/TF run ended up with the distro package version cockpit-machines-264-1, instead of the PR's COPR build (262.5.gblablabla). Yes, the distro version is newer -- long story, not important here.

The main issue is that it is totally not obvious how to install the tested package (locally built RPM or the automatically created COPR build) with tmt. The TF log makes it very obvious that TF spawns the machine, then has completely custom code to install the RPM from copr, and only then invokes tmt on that machine (supposedly with the provision --how connect?).

I find it rather hard to explain it to someone else how to reproduce a PR test failure locally with tmt run provision --how virtual. One has to jump through several hoops, like tmt run --until prepare, then figure out how to ssh in (tmt run login -l seems to destroy state, so one has to fish for the run-specific ssh command and the identity key), download/install the rpm, and then continue with --until report -l, then investigate the failure, and finally clean up with tmt run --since report -l.

I would like some way how to inject the package installation (or at least installing the copr repo) into the tmt command line, and Testing Farm to use that instead of its custom code. So that the whole run becomes reproducible locally, and the package installation becomes part of the tmt test log.

I don't particularly care about the "how" -- this could be something tailored such as --install-rpm, or something generic like running a shell script hook in between prepare and execute.

[martinpitt]

@TomasTomecek FYI, what we discussed on IRC.

[thrix]

@martinpitt yes, this is coming, @psss can point you to some tickets. We will unify how the artifacts are installed, tmt will takeover the responsibility, so it is reproducible in all environments (including your localhost)

[thrix]

@martinpitt for the particular run you have, it is now this:
https://artifacts.dev.testing-farm.io/246b3cdc-7f7e-46a1-b648-29855390e236/guest-setup-6b8652d0-3716-49d4-8df7-5402fec97df1/artifact-installation-6b8652d0-3716-49d4-8df7-5402fec97df1/

[martinpitt]

@thrix ! Glad to hear! I see, so currently there are some Ansible scripts which set up the VM, install the packages, and call tmt.

Cheers!

[thrix]

@martinpitt

there are some Ansible scripts which set up the VM

yes, they are here:

• https://gitlab.com/testing-farm/infrastructure/-/blob/testing-farm/ranch/public/citool-config/guest-setup/pre-artifact-installation/entrypoint.yml
• https://gitlab.com/testing-farm/infrastructure/-/blob/testing-farm/ranch/public/citool-config/guest-setup/post-artifact-installation/entrypoint.yml

install the packages, and call tmt

This is not done by Ansible, we have Python code doing that.

---

All these pieces will move under tmt, while the playbooks will be part of a tmt profile. And there will be different profile for Packit, Fedora CI, RHEL CI, CentOS Stream CI. We want to make the test environment prepartion well documented and available for contributions, so the CI communities can own and improve them.

[martinpitt]

Turns out that tmt can already do that. I just managed to reproduce a packit/COPR TF run from martinpitt/python-dbusmock#160 with

tmt run --verbose --all prepare --how install --copr packit/martinpitt-python-dbusmock-160 --package python3-dbusmock-0.28.4-1.20221005140834271905.pr160.11.g7b122ec.fc37.noarch provision --how virtual --image fedora-37

With the improved "show me all logs" UI, all this information can at least be pieced together without having to manually type URLs "that one needs to know" 😁

#1018 is the same thing, thus I am closing my issue as a duplicate. Indeed let's fix the tmt reproducer steps, but no need to track this in two places.