Extract "package manager" functionality into plugins

[happz]

More and more we see the need for primitives like "install a package". Plugins start to require packages to be present on guests, we already do have prepare/install plugin, and the upcoming all mighty install plugin. To support all these functions, tmt.package_managers would provide necessary primites, allowing implementations to be shipped as plugins.

The patch starts building the primitives, converting tmt.steps.prepare.install to use them in the process.

Pull Request Checklist

• implement the feature
• extend the test coverage
• include a release note

[happz]

So, adding a prototype support for apt to help me identify parts where rpm/yum/dnf are assumed. And it works :)

/plan
    discover
        how: fmf
        order: 50
        directory: /home/happz/git/tmt/tmp
        summary: 1 test selected
            /test
    provision
        how: local
        order: 50
        guest: localhost
        multihost name: default-0
        arch: x86_64
        distro: Ubuntu 22.04.3 LTS
        kernel: 5.10.102.1-microsoft-standard-WSL2
        package manager: apt
        selinux: no
        is superuser: no
        summary: 1 guest provisioned
    prepare
        queued push task #1: push on default-0
        
        push task #1: push on default-0
    
        queued prepare task #1: requires on default-0
        
        prepare task #1: requires on default-0
        how: install
        summary: Install required packages
        name: requires
        order: 70
        package: 1 package requested
            tree
            cmd: dpkg-query --show tree || sudo apt install -y  tree
            err: dpkg-query: no packages found matching tree
            err: 
            err: WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
            err: 
            out: Reading package lists...
            out: Building dependency tree...
            out: Reading state information...
            out: The following package was automatically installed and is no longer required:
            out:   podman-machine-cni
            out: Use 'sudo apt autoremove' to remove it.
            out: The following NEW packages will be installed:
            out:   tree
            out: 0 upgraded, 1 newly installed, 0 to remove and 2 not upgraded.
            err: dpkg-preconfigure: unable to re-open stdin: No such file or directory
            out: Need to get 0 B/47.9 kB of archives.
            out: After this operation, 116 kB of additional disk space will be used.
            out: Selecting previously unselected package tree.
            out:
                (Reading database ... 
                (Reading database ... 5%
                (Reading database ... 10%
                (Reading database ... 15%
                (Reading database ... 20%
                (Reading database ... 25%
                (Reading database ... 30%
                (Reading database ... 35%
                (Reading database ... 40%
                (Reading database ... 45%
                (Reading database ... 50%
                (Reading database ... 55%
                (Reading database ... 60%
                (Reading database ... 65%
                (Reading database ... 70%
                (Reading database ... 75%
                (Reading database ... 80%
                (Reading database ... 85%
                (Reading database ... 90%
                (Reading database ... 95%
                (Reading database ... 100%
                (Reading database ... 66655 files and directories currently installed.)
            out: Preparing to unpack .../tree_2.0.2-1_amd64.deb ...
            out: Unpacking tree (2.0.2-1) ...
            out: Setting up tree (2.0.2-1) ...
            out: Processing triggers for man-db (2.10.2-1) ...
            out: 
            out: Failed to retrieve available kernel versions.
            out: 
            out: Failed to check for processor microcode upgrades.
            out: 
            out: No services need to be restarted.
            out: 
            out: No containers need to be restarted.
            out: 
            out: User sessions running outdated binaries:
            out:  happz @ /dev/pts/0: bash[31121]
            out:  happz @ /dev/pts/1: node[1295,31204,31252,31293,31304,31394,31431], python[31333,31334,31489], sh[31194,31195,31200,31488]
            out:  happz @ /dev/pts/7: bash[31368]
            out: 
            out: No VM guests are running outdated hypervisor (qemu) binaries on this host.
    
        queued pull task #1: pull on default-0
        
        pull task #1: pull on default-0
    
        summary: 1 preparation applied
    execute
        queued execute task #1: default-0 on default-0
        
        execute task #1: default-0 on default-0
        how: tmt
        order: 50
        exit-first: false
            test: /test
                cmd: dpkg-query --show tree
                out: tree       2.0.2-1
                00:00:00 pass /test (on default-0) [1/1]

    
        summary: 1 test executed
    report
        how: display
        order: 50
            pass /test
                output.txt: /var/tmp/tmt/run-441/plan/execute/data/guest/default-0/test-1/output.txt
                content: tree   2.0.2-1

        summary: 1 test passed
    finish
    
    Prune '/plan' plan workdir '/var/tmp/tmt/run-441/plan'.
        summary: 0 tasks completed

total: 1 test passed

Unfortunately, I can't run tmt's tests because:

1. there's no tmt package for Ubuntu, and
2. there's no beakerlib package for Ubuntu :)

/plans/features/core
summary: Verify core functionality
    discover
        how: fmf
        directory: /home/happz/git/tmt
        hash: fb6235f1
        filter: tier: 0, 1
        summary: 1 test selected
            /tests/core/ls
    provision
        how: local
        guest: localhost
        multihost name: default-0
        arch: x86_64
        distro: Ubuntu 22.04.3 LTS
        kernel: 5.10.102.1-microsoft-standard-WSL2
        package manager: apt
        selinux: no
        is superuser: no
        summary: 1 guest provisioned
    prepare
        queued push task #1: push on default-0
        
        push task #1: push on default-0
    
        queued prepare task #1: default-0 on default-0
        queued prepare task #2: default-1 on default-0
        queued prepare task #3: requires on default-0
        
        prepare task #1: default-0 on default-0
        how: install
        package: 2 packages requested
            jq
            python3-pip
            cmd: dpkg-query --show jq python3-pip || sudo apt install -y  jq python3-pip
        
        prepare task #2: default-1 on default-0
        how: shell
        overview: 2 scripts found
        script: pip3 install --user yq || pip3 install yq
            cmd: /var/tmp/tmt/run-442/plans/features/core/tree/tmt-prepare-wrapper.sh-default-1-default-0
        script: yq --help
            cmd: /var/tmp/tmt/run-442/plans/features/core/tree/tmt-prepare-wrapper.sh-default-1-default-0
        
        prepare task #3: requires on default-0
        how: install
        summary: Install required packages
        name: requires
        package: 2 packages requested
            beakerlib
            tmt
            cmd: dpkg-query --show beakerlib tmt || sudo apt install -y  beakerlib tmt
        fail: Command 'dpkg-query --show beakerlib tmt || sudo apt install -y  beakerlib tmt' returned 100.
    finish
    
        summary: 0 tasks completed

plan failed

The exception was caused by 1 earlier exceptions

Cause number 1:

    prepare step failed

    The exception was caused by 1 earlier exceptions

    Cause number 1:

        Command 'dpkg-query --show beakerlib tmt || sudo apt install -y  beakerlib tmt' returned 100.

        stdout (3 lines)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        Reading package lists...
        Building dependency tree...
        Reading state information...
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

        stderr (7 lines)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        dpkg-query: no packages found matching beakerlib
        dpkg-query: no packages found matching tmt

        WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

        E: Unable to locate package beakerlib
        E: Unable to locate package tmt
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[LecrisUT]

Both beakerlib and tmt should be straightfoward packageable for debian/ubuntu. Are there any contacts that can help with that?

This PR would also help with tmt github action

[lukaszachy]

@happz Shouldn't be the package_manager property of Guest? For my use case I'd need to use it outside of prepare-install. For the rsync installation use case it can happen in provision/execute/finish as well.

[happz]

@happz Shouldn't be the package_manager property of Guest? For my use case I'd need to use it outside of prepare-install. For the rsync installation use case it can happen in provision/execute/finish as well.

I believe it is, see https://github.com/teemtee/tmt/pull/2557/files#diff-f684593942660856c3bdc810b597c3956b4107716a554eece2c2bf08344a9503R635. There's also Guest.facts.package_manager which is a string, name of the manager as discovered by fact collection, but once that one is known, Guest.package_manager will return a proper instance of tmt.package_managers.PackageManager. As you can see at https://github.com/teemtee/tmt/pull/2557/files#diff-f684593942660856c3bdc810b597c3956b4107716a554eece2c2bf08344a9503R1108, I think I even made it to the point where rsync installation would use this attribute.

[lukaszachy]

Ouch. I completely missed that I am looking on different file... Thus this PR is usable for my use case ;)

[lukaszachy]

Unfortunately, I can't run tmt's tests because:

1. there's no tmt package for Ubuntu, and
2. there's no beakerlib package for Ubuntu :)

I guess this asks for "I have all bits installed already use case" and/or better check (e.g command -v for binaries or existence of /usr/share/beakerlib/beakerlib.sh for beakerlib...)

[happz]

/packit test

[LecrisUT]

Unfortunately, I can't run tmt's tests because:

1. there's no tmt package for Ubuntu, and
2. there's no beakerlib package for Ubuntu :)

What about making an implementation for homebrew? Packaging there should be relatively fast.

[lukaszachy]

What about making an implementation for homebrew? Packaging there should be relatively fast.

Could you help with that? Not many macos users in the tmt devel audience.

[LecrisUT]

What about making an implementation for homebrew? Packaging there should be relatively fast.

Could you help with that? Not many macos users in the tmt devel audience.

Yep, I can help there. Granted I don't have macs anymore, but I still maintain some homebrew packages now and then

[psss]

What a weird failure in the c9s job:

:: [ 13:09:20 ] :: [  BEGIN   ] :: Running 'hatch -v run dev:pytest -vvv -ra --showlocals -n 0 -m 'not containers' tests/unit'
No config file found, creating one with default settings now...
Success! Please see `hatch config`.
───────────────────────────────────── dev ──────────────────────────────────────
Creating environment: dev
Installing project in development mode
ERROR: File "setup.py" or "setup.cfg" not found. Directory cannot be installed in editable mode: /var/ARTIFACTS/work-corewldzchyf/plans/features/core/discover/default-0/tests
(A "pyproject.toml" file was found, but editable mode currently requires a setuptools-based build.)
:: [ 13:09:21 ] :: [   FAIL   ] :: Command 'hatch -v run dev:pytest -vvv -ra --showlocals -n 0 -m 'not containers' tests/unit' (Expected 0, got 1)

[happz]

What a weird failure in the c9s job:

:: [ 13:09:20 ] :: [  BEGIN   ] :: Running 'hatch -v run dev:pytest -vvv -ra --showlocals -n 0 -m 'not containers' tests/unit'
No config file found, creating one with default settings now...
Success! Please see `hatch config`.
───────────────────────────────────── dev ──────────────────────────────────────
Creating environment: dev
Installing project in development mode
ERROR: File "setup.py" or "setup.cfg" not found. Directory cannot be installed in editable mode: /var/ARTIFACTS/work-corewldzchyf/plans/features/core/discover/default-0/tests
(A "pyproject.toml" file was found, but editable mode currently requires a setuptools-based build.)
:: [ 13:09:21 ] :: [   FAIL   ] :: Command 'hatch -v run dev:pytest -vvv -ra --showlocals -n 0 -m 'not containers' tests/unit' (Expected 0, got 1)

I believe we need a newer pip on CentOS. I have a patch ready, I'm just waiting for the remaining jobs to fail, to see whether there's more to fix.

[happz]

/packit build

[psss]

Nice! Tests are green \o/ Just two comments.

[psss]

The only failing test is /tests/prepare/multihost which looks like the race condition issue again:

• Possible race condition in the testcloud plugin #2687 (comment)

I've run the test manually and it's green, thus ready to merge.