Review tests + request for additional test ideas for Podman

[mairin]

Feature request description

Fedora's OpenQA runs some basic testing on Podman right now:
https://openqa.fedoraproject.org/tests/2024118/modules/podman/steps/1/src

These are the things it is testing. Are there other things we should be testing?

    # check to see if you can pull an image from the registry
    assert_script_run "podman pull registry.fedoraproject.org/fedora:latest", 300;
    # run hello-world to test
    validate_script_output "podman run -it registry.fedoraproject.org/fedora:latest echo Hello-World", sub { m/Hello-World/ };
    # create a Dockerfile
    assert_script_run 'printf \'FROM registry.fedoraproject.org/fedora:latest\nRUN /usr/bin/dnf install -y httpd\nEXPOSE 80\nCMD ["-D", "FOREGROUND"]\nENTRYPOINT ["/usr/sbin/httpd"]\n\' > Dockerfile';
    # Build an image
    assert_script_run 'podman build -t fedora-httpd $(pwd)', 180;
    # Verify the image
    validate_script_output "podman images", sub { m/fedora-httpd/ };
    # Run the container
    assert_script_run "podman run -d -p 80:80 localhost/fedora-httpd";
    # Verify the container is running
    validate_script_output "podman container ls", sub { m/fedora-httpd/ };
    # Test apache is working
    assert_script_run "curl http://localhost";
    # Open the firewall, except on CoreOS where it's not installed
    unless (get_var("SUBVARIANT") eq "CoreOS") {
        assert_script_run "firewall-cmd --permanent --zone=internal --add-interface=cni-podman0";
        assert_script_run "firewall-cmd --permanent --zone=internal --add-port=80/tcp";
    }
    # tell client we're ready and wait for it to send the message
    mutex_create("podman_server_ready");
    my $children = get_children();
    my $child_id = (keys %$children)[0];
    mutex_lock("podman_connect_done", $child_id);
    mutex_unlock("podman_connect_done");

Suggest potential solution

It would be lovely to get a list of commands or functionality we should test in Podman on Fedora OpenQA that we are not currently testing.

Have you considered any alternatives?

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

[lsm5]

I feel this can be run as a packit + TMT task on every PR.

@cevich @edsantiago PTAL, are we running some version of these tests already in our CI?

[lsm5]

I feel this can be run as a packit + TMT task on every PR.

Sorry, I mean that is just so that we find any issues before it goes to OpenQA.

[edsantiago]

Sorry.... I'm feeling a little lost her, more so because I have no familiarity with OpenQA. But first, am I understanding correctly that there are two questions at play here?

1. from @mairin, "what other tests should be run in OpenQA"; and
2. from @lsm5, "should podman CI incorporate OpenQA tests"

If my interpretation is wrong, then nothing I say below is of any worth, and I would appreciate clarification.

If my interpretation is correct: what problem are we trying to solve here? What problem is OpenQA trying to address? When/where does OpenQA run? Who is writing those tests? Who is checking that they pass? Who is identifying test failures, reporting podman bugs if appropriate, fixing test bugs, or ignoring flakes? Is it important for OpenQA to have its own set of tests, instead of using existing ones?

And, one thing I immediately noticed is, the excerpt above uses registry.fedoraproject.org. All registries are flaky, but that one is the flakiest of the flaky (the redhat.com one is a close second, but still second). I expect those tests to flake often.

[lsm5]

Sorry.... I'm feeling a little lost her, more so because I have no familiarity with OpenQA. But first, am I understanding correctly that there are two questions at play here?

I'm kinda sorta wrapping my head around it too, chatting with fedora infra people on the side. Here's what I gathered. @mairin correct me if I'm wrong.

1. from @mairin, "what other tests should be run in OpenQA"; and
2. from @lsm5, "should podman CI incorporate OpenQA tests"

I think my question can be ignored for now. Let's focus on @mairin's.

If my interpretation is correct: what problem are we trying to solve here? What problem is OpenQA trying to address? When/where does OpenQA run?

Testing podman at the bodhi update level. Whether a podman bodhi update works well with other bodhi updates, builds meant to be released in fedora.

Who is writing those tests?

@mairin and/or Fedora QA (I think).

Who is checking that they pass? Who is identifying test failures, reporting podman bugs if appropriate, fixing test bugs, or ignoring flakes?

That would be both Fedora QA and the bodhi owners (us).

Is it important for OpenQA to have its own set of tests, instead of using existing ones?

They are perl tests for now. So, it'll need to be its own set.

[edsantiago]

Testing podman at the bodhi update level.

Ack, thanks. So at the bodhi level we then have gating tests, manual (human) thumbs-ups, and OpenQA. What do the OpenQA tests offer that is not adequately addressed by the other two?

[mairin]

@edsantiago for just Podman tests in OpenQA.... every time a core Fedora component is updated (for example, systemd, or selinux policy), the Podman tests will be run again in Fedora OpenQA. This means if a change to systemd or selinux breaks Podman, Fedora OpenQA will let us know.

The dream I am particularly pursuing on this, is getting Podman Desktop into Fedora OpenQA too. This could also benefit Podman in that if a change in Podman breaks Podman Desktop, the Podman Desktop test would fail and we'd know which Podman update broke it.

[edsantiago]

AHA! Thank you, that's what I was missing: we're talking reverse dependency testing, our long-sought-after holy grail. This is excellent. The question now becomes: how much does podman get, resource-wise? (Seconds? Minutes?) Do tests run as root or rootless (or, best, both)? The testing candidates that spring immediately to mind are:

• user namespaces
• SELinux
• checkpoint/restore
• events and logs
• signals
• volumes

[mairin]

@edsantiago from adamw:

• commands can be run as root or user. user is a bit awkward but doable and a non-issue.
• default max execution time on tests is two hours
• resource limits aren't really enforced by the system, but adamw does worry about how many test runners there are and how long tests take
• adding an extra test or test step that is an extra 30s is a non-issue. Adding one that takes an extra half hour is more concerning.
• generally: "if a test takes less than ten minutes i'll be more relaxed about adding it, if it takes more than half an hour it better be really valuable"

The current podman test linked above takes about 5 minutes and most of that is installing podman, the rest is creating a simple httpd container.

[edsantiago]

All right! Thanks for your patience. Would it be acceptable for OpenQA to invoke external tests? To be specific, see #19302.

TL;DR we already package podman-tests. Those are overkill. However, by tagging the most integration-risky tests, it's possible to get maximum coverage with least effort and duplication. The selection I've chosen for #19302 , for example, takes 3 minutes on my laptop (probably longer on underpowered CI systems). Most of that time is in one or two big-image tests.

@mairin, @adamw, is this a reasonable starting point?

[AdamWill]

IIUC, yes, that's totally fine (and awesome as it saves me work!) If literally all openQA needs to do, once it's set up a generally-working environment to test and has podman installed, is install podman-tests and run one command (and fail if the command fails), yes, that is super easy to do, I can implement that in thirty seconds flat. No problem at all.

[edsantiago]

Excellent! Thank you!

install podman-tests and run one command (and fail if the command fails)

Yes and yes, but (I keep having to learn not to make assumptions) would it be possible to report the output as part of the failure? Assuming so, I will proceed with #19302.

[AdamWill]

yes, sure. you'd get a screenshot of it for free, but we can easily redirect the output to a file and upload it on failure.

[Luap99]

I fully support better and especially reverse dependency testing.
Just to be clear what do these results mean, if a kernel, systemd, etc.. breaks a podman test will this actually stop a package (say kernel update) from going into stable? Or this is only a heads up for us?

[AdamWill]

Yes, it would be prevented from going stable unless the failure was somehow resolved or the maintainer waived it.

[cevich]

Ha! Thanks for the PR link @edsantiago, it answers exactly the question I was going to ask:

How do we ensure new OpenQA relevant tests get picked up, and ensure old/renamed/moved/retired tests are handled appropriately?

[AdamWill]

This is now landed in openQA downstream.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@mairin @AdamWill @lsm5 @edsantiago @Luap99 any update on this?

[AdamWill]

Well, as far as I'm concerned, it's implemented - we landed the testing downstream, as I mentioned above. The tests are now running on every Fedora update that contains a package in the critical-path-base or core comps groups, see https://openqa.fedoraproject.org/tests/2149856#next_previous for how frequently they're running (that's just on Fedora 39). There are a couple of warnings shown at the console when the tests complete, but they don't seem to be a problem:

If there's any further testing we can do, I'm happy to add it, but I'm not the best person to suggest tests, I don't think!

[Luap99]

I think we can close this, the test tags are in our upstream repo so if anyone wants to add more tests we can add them at any point, see 33891e8