test_libdnf test fails on RISC-V

[rwmjones]

The test_libdnf test fails (reliably) on RISC-V, at least when we build in Koji. I was unable to reproduce the test failure locally or debug it further.

[rwmjones]

Build logs:
http://oirase.annexia.org/tmp/dnf5-issue-503/

[jan-kolarik]

The RpmTransactionTest::test_transaction test returned TransactionRunResult::ERROR_RPM_RUN at the end instead of the TransactionRunResult::SUCCESS, which basically means there was an error installing the testing package.

Could you please provide also the /var/log/dnf5.log?

[jan-kolarik]

But this could be an issue I noticed in the log:

2023-03-08T21:17:41+0000 [4030443] TRACE [rpm] entering chroot /tmp/libdnf5_unittest.xH7dWH/installroot/
2023-03-08T21:17:41+0000 [4030443] ERROR [rpm] Unable to change root directory: Operation not permitted

So maybe your build container doesn't have the right capabilities like CAP_SYS_CHROOT ...

[rwmjones]

The build happens on a build server so we cannot retrieve other files. If /var/log/dnf5.log is important then the test should probably be changed to print it on error.

So maybe your build container doesn't have the right capabilities like CAP_SYS_CHROOT ...

It's using mock with the chroot method running on a baremetal machine so I don't think there is any container involved.

[kontura]

Our nightly builds have also been failing for other arches than x84_64:

Traceback (most recent call last):
  File "/builddir/build/BUILD/dnf5-5.0.10-20230515005615.6.g6875710a/test/python3/libdnf5/base/test_transaction.py", line 43, in test_check_gpg_signatures_fail
    self.assertFalse(transaction.check_gpg_signatures())
AssertionError: True is not false

It might be related.

[jrohel]

I had the same problem some time ago. The solution was to not run tests using the librpm library in parallel.
PR #335

[rwmjones]

That's interesting because the RISC-V builders we're using at the moment are quite slow. They might expose race conditions which wouldn't be evident on x86-64.

[jan-kolarik]

Our nightly builds have also been failing for other arches than x84_64:

Traceback (most recent call last):
  File "/builddir/build/BUILD/dnf5-5.0.10-20230515005615.6.g6875710a/test/python3/libdnf5/base/test_transaction.py", line 43, in test_check_gpg_signatures_fail
    self.assertFalse(transaction.check_gpg_signatures())
AssertionError: True is not false

It might be related.

I've submitted a fix for this here, though it is not connected with this issue as it only affects Python bindings.

[j-mracek]

Because the related PR is merged I suggest that the issue is resolved. @rwmjones May I ask you for verification?

[jan-kolarik]

Because the related PR is merged I suggest that the issue is resolved. @rwmjones May I ask you for verification?

Actually this nightly builds issue was originally mentioned as "might be related", but as I responded in the previous comment, it is not. Therefore this is not fixed yet AFAIK.

[jan-kolarik]

@rwmjones Please could you try running the libdnf tests after adding the following line at the end of the test/libdnf/CMakeLists.txt?

set_tests_properties(test_libdnf PROPERTIES RUN_SERIAL TRUE)

As I cannot reproduce it on our available nodes.

[rwmjones]

Fedora Rawhide has been updated to dnf5 5.0.14 which includes PR 335, so I'm testing a build now:
http://fedora.riscv.rocks/koji/taskinfo?taskID=1456001

[rwmjones]

Situation unclear. Is this the same error as before? http://fedora.riscv.rocks/kojifiles/work/tasks/6036/1456036/build.log

[jan-kolarik]

Yep, still the same error, so the serial running of the tests didn't fix the issue ☹️

[jan-kolarik]

I cannot say what is actually wrong, but it seems like the build configuration is quite different than what we are using in CI, f.e. here for COPR build. Also in your logs there is Mock Version: 2.15, while the currently used is 4.1. But maybe the more important thing is how the rpmbuild is done as reading through this comment from possibly related BZ ticket, the chroot method there might not be compatible with what rpm does inside then ...

[j-mracek]

We have an issue with builds related to libsolv default flags in your distribution - see #547

[ppisar]

@rwmjones, is there a way for Fedora packagers to do scratch builds on any of the RISC-V Koji instances? We really cannot help you if we cannot instrument the test run.
Also the fact that the chroot failure only happens in that Koji instance and not locally hints that something is strange with that Koji. Modern Mock uses containers (systemd-nspawn).
Looking at dnf5 build history http://fedora.riscv.rocks/koji/packageinfo?packageID=31152, I can see there was a success in dnf5-5.0.9-3.fc38. Was that a random success, or is that reproducible?

[rwmjones]

We don't have any public instances, but if you give me instructions I can try them out.

Best current hardware option if you'd like to buy something is the Vision Five 2: https://rwmj.wordpress.com/2023/07/25/heads-up-lichee-pi-4a-vs-visionfive-2-vs-hifive-unmatched-vs-raspberry-pi-4b/

[Conan-Kudo]

It is trivially reproducible on openSUSE RISC-V in the openSUSE Build Service too. https://build.opensuse.org/package/show/home:Pharaoh_Atem:DNF5_SUSE/dnf5

If you create an openSUSE account and log into the openSUSE Build Service, you can fork my package there and do what you like.

[davidlt]

Fedora/RISCV does not have integration with FAS. If you have experience with Fedora packaging and Koji you can get an account. There are at least several folks that already have it.

Years a go we used to run systemd-nspawn, but we switched back to old-style chroot as that was default in Fedora until somewhat recently. We haven't switched back to systemd-nspawn yet.

[jan-kolarik]

I revisited this issue and reviewed the openSUSE builders' results. I believe we can unblock this by temporarily removing the two unit tests that are failing on RISC-V. This will enable further testing, including running the CI tests against the built package on those systems, providing insights into the issues with dnf5 behavior. I plan to prepare the pull request next week.

[Conan-Kudo]

Thank you.

[Conan-Kudo]

I've been able successfully build for openSUSE RISC-V with DNF5 5.1.9: https://build.opensuse.org/package/show/system:packagemanager:dnf/dnf5

[davidlt]

I haven't used DNF5 yet, but otherwise it now builds in Fedora/RISCV: http://fedora.riscv.rocks/koji/buildinfo?buildID=278057