WIP: Test Fedora integration

[pvalena]

Needs packit "Github Application" (service) enabled.

Currently Testing Farm requires Packit, which requires the RPMS to be build, to be able to run the tests. Packit team is working on simple config which would be able to use spec file from Fedora dist-git. Anyway I'm not sure what to do about the Source1 in that case, so having a spec file copy (or some dummy spec file) in this repo would be benefitial.

Testing Farm has the resulting RPMS installed, and can run any integration tests (testing actual runtime in Fedora), including kernel-dependent tests or those that require network connection.

Do you have any idea for integration test that's missing?

Note: the test logs are currently displayed using a bit inconvinient UI, but the files themselves are accessible as well.

---

You can test this locally by running f.e.:

# To create the SRPM:
$ packit srpm

# To run in VM, uses Fedora stable image by default
$ tmt run -vv

# To run in container, use Fedora rawhide
$ tmt run -avv provision -h container -i fedora:rawhide

Note: -vv shows actual test output, which is otherwise omitted, but you can display it even after the run, without re-executing like so:
tmt run -l report -fvvv.

---

In this PR does not include Tests metadata (test suite is executed using simple execute: script in plan metadata instead).
Ideally we would add test metadata (can be an test itself as well). That would mean one (or more, depends on us) additional file with test descriptions etc.. There are several metadata "forms" to choose from.

Alternatively, tests can be run from plan ilke so (but those can't be pulled from Fedora f.e.), or using execute (currenlty only single test).

Afterwards, the same test suite (living actually in github) can be easily run from Fedora (when CI gets is enabled). Anyway, AFAIK this can be done even without any metadata present in upstream repo as well (not recommended).

---

Examples:

• This test suite, passing

• This test suite, failing

• Example Fedora configuration

• Example Test metadata (simple, multiple tests)

• Example Test metadata (advanced, single, uses inheritance)

[pvalena]

@voxik WDYT?

[voxik]

Needs packit "Github Application" (service) enabled.

Currently Testing Farm requires Packit, which requires the RPMS to be build, to be able to run the tests. Packit team is working on simple config which would be able to use spec file from Fedora dist-git. Anyway I'm not sure what to do about the Source1 in that case, so having a spec file copy (or some dummy spec file) in this repo would be benefitial.

Sorry, but having .spec file, especially .spec file which might resemble any production .spec file is no-go. Spec files have to be in Fedora dist-git for reasons such as mass rebuilds/fixes. Alternatively, if it was somewhere buried alongside packit configuration, that might be acceptable.

Do you have any idea for integration test that's missing?

Actually I don't really know. After last updates to fix the Ruby 2.7 compatibility, the test has much better coverage.

[TomasTomecek]

Needs packit "Github Application" (service) enabled.
Currently Testing Farm requires Packit, which requires the RPMS to be build, to be able to run the tests. Packit team is working on simple config which would be able to use spec file from Fedora dist-git. Anyway I'm not sure what to do about the Source1 in that case, so having a spec file copy (or some dummy spec file) in this repo would be benefitial.

Sorry, but having .spec file, especially .spec file which might resemble any production .spec file is no-go. Spec files have to be in Fedora dist-git for reasons such as mass rebuilds/fixes. Alternatively, if it was somewhere buried alongside packit configuration, that might be acceptable.

You don't need to have the spec file here and can download it from dist-git on-the-fly:

actions:
  post-upstream-clone: "wget https://src.fedoraproject.org/rpms/abrt-ruby/raw/master/f/abrt-ruby.spec"

With the line above, whenever packit service clones this repo and checks out a specific PR, it runs the specified command - downloads the spec and uses it further.

[pvalena]

With the line above, whenever packit service clones this repo and checks out a specific PR, it runs the specified command - downloads the spec and uses it further.

@TomasTomecek WRT that .spec - I'm wondering whether it'd be possible to remove %check section and corresponding sources in some generic way? (I'm trying to avoid maintaining sed scriplets.) Alternative would be to opensource my gems tooling and use that...

[pvalena]

Sorry, but having .spec file, especially .spec file which might resemble any production .spec file is no-go. Spec files have to be in Fedora dist-git for reasons such as mass rebuilds/fixes. Alternatively, if it was somewhere buried alongside packit configuration, that might be acceptable.

Right. Let's say I move the plan/example.fmf and .spec into ci/ folder. And let's add disclaimer about how I got the .spec file, and where's the authoritative source. Or would you be inclined to have it automated (WRT my previous comment) instead?

Do you have any idea for integration test that's missing?

Actually I don't really know. After last updates to fix the Ruby 2.7 compatibility, the test has much better coverage.

Great! Let's stick with the rspec test suite for now. That would also mean that usefulness of running this test suite in Fedora CI (as a gating f.e.) would be of no benefit, right?

[voxik]

With the line above, whenever packit service clones this repo and checks out a specific PR, it runs the specified command - downloads the spec and uses it further.

@TomasTomecek WRT that .spec - I'm wondering whether it'd be possible to remove %check section

--nocheck ? That is supported by rpmbuild as well as mock

and corresponding sources in some generic way? (I'm trying to avoid maintaining sed scriplets.) Alternative would be to opensource my gems tooling and use that...

This is harder ...

[pvalena]

With the line above, whenever packit service clones this repo and checks out a specific PR, it runs the specified command - downloads the spec and uses it further.

@TomasTomecek WRT that .spec - I'm wondering whether it'd be possible to remove %check section

--nocheck ? That is supported by rpmbuild as well as mock

Yes, well, not sure packit has support for it.

and corresponding sources in some generic way? (I'm trying to avoid maintaining sed scriplets.) Alternative would be to opensource my gems tooling and use that...

This is harder ...

We need also remove -b 1 from setup macro. In case packit does not resolve this, would you rather:

• (1) sync up spec from Fedora from time to time
• (2) use some custom tooling to generate the additional sources

[pvalena]

@voxik I've fixed the paths and added disclaimer into spec file.

Test: pvalena#8
(direct link to execute.log)

This should work until end of month with TMT (instead of cruncher; mentioned in comments) here on github as well as in Fedora CI.

You can also run a test with:

$ podman run --rm -it -v"$PWD:/home/test/run:Z" pvalena/fedora-tmt-all:rawhide sudo tmt run -avv provision -h local

Using rawhide tag, but you can also use f33 or latest f.e..

[voxik]

Ugh, I had old .gem files lying around and now they are moved to ci/ folder, that is not nice

[voxik]

If only RubyGems were not so broken 🤯

rubygems/rubygems#3953

[pvalena]

Oh, I didn't think of that. But I'm glad I didn't have anything like rm *.gem there in the beginning.

Unfortunately, I couldn't think of any other (generic) way to do it. Ideally, there wouldn't be any of moving, or changing directories, but the .gem file needs to be in ci/ folder for packit to find it (not sure whether that's a bug; I noted that in some issue, but maybe it's in it works, don't touch it state).

Creating the gem using -C would definitely help. Thanks for starting that thread, and creating PR!

[TomasTomecek]

I missed those questions somewhat. We are using copr to run builds and I'm not sure if we can pass args to mock/rpmbuild - I haven't seen any option for that.

[pvalena]

I missed those questions somewhat. We are using copr to run builds and I'm not sure if we can pass args to mock/rpmbuild - I haven't seen any option for that.

It's not an argument for mock(but for gem instead). And it's a result of this issue I've encountered (I didn't check whether it got fixed subsequently).