cirrus: enable cgroups v2 tests with crun

[giuseppe]

Signed-off-by: Giuseppe Scrivano gscrivan@redhat.com

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: giuseppe

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [giuseppe]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rhatdan]

LGTM
Don't see why this is a WIP. If it passes it should be merged.

[giuseppe]

LGTM
Don't see why this is a WIP. If it passes it should be merged.

I've opened the PR just to see what would fail and keep working on it. I had expected a lot of failures, as the cgroup v2 backend is not really tested. From 70+ failures it went down now to a few ones. There were a couple of issues in crun, but most of the failures were related to the tests not ready for cgroups v2.

[giuseppe]

🎉 🎉 🎉 cgroups 2 tests are passing now 🎉 🎉 🎉

[giuseppe]

it is using the master branch for crun. I'll cut a new release in the next days

[giuseppe]

tests are all green

[giuseppe]

@cevich @mheon PTAL

[mheon]

This is just test changes, right?

I don't like the duplication around determining if we're in V2 mode, but I don't know if we can remove it easily.

Otherwise LGTM

[giuseppe]

yes, it is just tests code changes beside: 04d333f

[giuseppe]

@cevich I think we will need to enable the same tests on the crun repository, so that we don't risk to break Podman. How difficult would it be to run Podman tests there?

[cevich]

I'm with Matt, that's a lot of duplication. IMHO, this PR should depend on #3754 (includes crun package) rather than inline package + source install (see my comments above). It's one of my top priorities this week to get that merged ASAP - CI in other repos depend on it as well.

[cevich]

The VM images from PR #3754 already include the crun package. Unless there's a really good reason for it, I'm against having both the package installed AND installing from source at the same time. It creates a nightmare for both test/automation maintenance and seriously complicates debugging.

If using the source is required at this early stage, this sub-shell build/install should be moved into a dedicated lib.sh function to support readability and ease of maintenance. Also let me know so I can drop the dnf install -y crun from my PR.

[giuseppe]

we need to install from source for now, as master contains some fixes necessary to get the tests passing

[cevich]

Are you intending to support running of the system-tests? If so, please add to .cirrus.yml:

special_testing_cgroupv2_task
    ...
    system_test_script: '$SCRIPT_BASE/system_test.sh |& ${TIMESTAMP}'

[giuseppe]

I've not yet looked into them, I'd expect them to fail at this point. We can add them later, once we are sure they work with a different OCI runtime

[cevich]

How difficult would it be to run Podman tests there?

I'm not a big fan of running "child" tests in "parent" projects, it tends to result in very messy environments (in the parent) which don't accurately test compatibility and require an extraordinary amount of maintenance and duct-tape.

OTOH, I see that crun looks to already have some dedicated testing setup in travis?.

If you need help converting or extend that to something more modern than old/crusty Ubuntu Xenial host environment, that's something I could help with. It's not trivial, but having existing/functional automation helps a lot.

If it's a short/medium term automation need, I can get something going in a day or two worth of work (need a jira card). If there's a chance it will be long-term (beyond 6-months), then I'd prefer to setup a dedicated cloud-environment (like we do for c/storage, buildah, etc.). That adds about 4-hours of work for me but a few days waiting for e-mail requests/approvals for funding. I'm fine either way, but would rather get started on dedicated long-term stuff early, if that's what is needed (also needs Jira card)

[giuseppe]

if it is too much effort, I can open WIP PR to containers/libpod once in a while and before cutting a new release to verify it still works

[giuseppe]

I'm with Matt, that's a lot of duplication. IMHO, this PR should depend on #3754 (includes crun package) rather than inline package + source install (see my comments above). It's one of my top priorities this week to get that merged ASAP - CI in other repos depend on it as well.

for now we need to use crun from git as there are fixes required for the tests. I'll cut a new release and then we can use the package

[cevich]

if it is too much effort, I can open WIP PR to containers/libpod once in a while and before cutting a new release to verify it still works

That would probably be okay a few times, then get to be a major pain-point, don't you think?

Since crun is required for this cgroupsv2 test, it'll get a fairly big safety net that way (it's going to run for everybody's changes and post-merge also). It sounds like you're mainly worried about new, crun-specific changes affecting downstream compatibility, eh?

Traditionally, this would be where crun would have it's own integration and system-tests, as a kind of "contract" with the "outside world" (libpod). So for example, if downstream changes in libpod require different crun behaviors, the tests in crun need to be modified accordingly.

Honestly there are simply not a lot of quick/cheap/easy ways to solve this, which also play nicely with (potentially many) other parallel callers/users/developers of different crun versions/releases (including libpod). Testing and automation is just hard work sometimes.

That said, I'm willing to do most/all the automation implementation work. All I need is you/other community members to support me if I have crun-specific issues/questions along the way. It's way better if we plan and get it right/good early on. Doing/Re-doing it later is almost always much more difficult.

Which jira cards should I write up for this (I'll CC you), short/medium term or long-term or other?

[cevich]

for now we need to use crun from git as there are fixes required for the tests. I'll cut a new release and then we can use the package

Okay, that's fine if it's just for now. Just please make sure there's a "post-it note" (or jira card or in-line comment) somewhere so it's actually "just for now". I can't say how many times these have slipped and become permanent debugging-nightmares for $FUTURE_DEVELOPER. Best to avoid that.

[giuseppe]

Which jira cards should I write up for this (I'll CC you), short/medium term or long-term or other?

Fedora 31 is getting closer, so I think we should try to get at least these tests running for Podman as soon as we can, and have some sort of confidence with cgroups v2.
We can discuss the details next week on how to coordinate and proceed with the crun integration.

[rhatdan]

/lgtm