Running upstream tests in Fedora dist-git needs upstream tarball discovery or flexible ref: -- still needs #978

[martinpitt]

cockpit-podman now has an FMF test plan which runs in packit, which works very nice. I am currently investigating how to run the exact same tests downstream in Fedora. (Thanks to @psss for your great help!).

plans/upstream.fmf currently looks like this:

discover:
  how: fmf
  repository: https://github.com/cockpit-project/cockpit-podman
  ref: "29"
execute:
  how: tmt

But this is awkward: I don't want to manually bump the ref: "29" each time there is an upstream release. This is error prone and breaks automation. The default value of ref: is master, which is not suitable at all -- upstream's mainline tests are pretty much guaranteed to fail on the last or even earlier release. discover does not currently have any other method.

So one way to fix this would be to support a shell command, or even provide that in tmt itself -- in our case, we'd parse Version: from the spec and requests that as a tag.

But even that is a bit of a crutch -- having to check out a remote git tree is a point of brittleness, and this shell hackery is too much indirection. What I'd really like to do is what STI does: It unpacks the upstream tarball into $WORKDIR/source, and tests can just run there. See cockpit-podman's STI playbook role for example.

It would be really nice to retain that semantics with FMF as well. The Fedora dist-git should be a small skeleton only, all the interesting bits should be done, gated, and released upstream. To keep TMT agnostic to the dist-git packaging mechanics, I could imagine a prepare-discover: or pre-discover: shell command which could then do

prepare-discover: |
    fedpkg sources
    tar xf cockpit-podman*.tar.?z
discover: 
    how: fmf
    path: cockpit-podman/

This would actually also help @jscotka with his efforts of dynamically generating FMF metadata from Python unittests -- that step could then run the generator, so that the subsequent discover step can pick them up.

This is actually very similar to prepare:, but that runs after discover.

[psss]

Yes, this seems to be a demanded feature. Let's try to find a solution for this soon. Proposing for 1.5.

[psss]

Possible temporary workaround I'm using for referencing tmt integration tests from fmf is to set up a branch which contains tests relevant to the latest released bits:

• https://github.com/psss/fmf/blob/master/plans/integration.fmf

So the only thing I need to do when releasing a new tmt is to move forward the fedora branch.

[jkonecny12]

Hi @psss, your solution is great. However, it has a problem that the tests will fail during time between upstream and downstream release is done. In that period the upstream test branch will point to the tests which are not yet in gating.

[psss]

Yes, yes, just a workaround, not very perfect... ;-)

[jscotka]

@psss as Martin referenced my issue #588 it could be also more generic and very useful. Not just about getting some version of test before discovery.

[martinpitt]

My hack until then is to have our release scripts bump the ref automatically.

[jkonecny12]

@psss as Martin referenced my issue #588 it could be also more generic and very useful. Not just about getting some version of test before discovery.

I like that solution. As you said it's more general.

[pvalena]

I always thought this is where https://src.fedoraproject.org/tests/***/ comes into play. Yes, it's similar to maintaining an upstream branch, but I think that's the point- having the ACLs in the OS repo (for tests run on the OS CI), not upstream ACLs.

I can see your point, when you're in charge of upstream, that you dont want your tests duplicated. But this basically generates tests dynamically from arbitrary upstream source... which I think is an "unsafe" operation, when I simply want to "run" the tests, not modify a host OS. I'd be fine with this running in a VM, f.e., but that's a lot of overhead for one archive...

(I wrote more on this in #588.)

[martinpitt]

@pvalena: For me this is exactly not the point -- I want to maintain and run the tests during upstream CI, so having this separate tests/ namespace is useless, too expensive [1], and error prone. The nice thing about STI is that you can effortlessly run tests from the upstream release tarball, which I consider the right thing to do. I"m not much fussed about how exactly to get this back (unpacking the source tarball in tmt, or mapping the spec version to a tag, or something equivalent).

[1] in the sense of keeping gating tests actually working -- if they only run downstream, they always fail at the wrong time, and are nothing but a nuisance.

[martinpitt]

But this basically generates tests dynamically from arbitrary upstream source... which I think is an "unsafe" operation, when I simply want to "run" the tests

No, not really - I want to take the tests as is from the upstream source which is defined in the spec. This is "safer" than tmt checking out some arbitrary git repo, as it does now. It's also awkward that test discovery has to call out to the internet in the first place, while it already has everything that it needs locally and validated.

[jkonecny12]

In general you won't get much adaption if there will be more work to be done manually like syncing upstream tests to another downstream repo. Also why would people use TMT if it would gave them more work than STI which they have now.

Honestly, I think the whole downstream tests repository is a wrong idea from my PoV. I know some teams would benefit that but in most cases you want to use upstream tests and work on them not having copy downstream.

[martinpitt]

Reading #588 and the replies here, I do understand your concern about running too much code in the prepare step on the host machine. For this specific issue, I'd really be content with discovering the tests not from a git repo, but a local tarball or dir:

discover:
  how: fmf
  archive: dist-git

... or something like that. That could invoke {fed,cent,rh}pkg sources to download the tarball, unpack it, and discover tests from there? Admittedly this is rather specific to RPM packaging, but one way or the other we need to support that.

The standard-test-roles standard-test-source playbook should be a good inspiration, as that pretty much did the right thing in the last years.

[lukaszachy]

I need *pkg source too. However I wouldn't bind it to how:fmf and keep it a separate thing. Reason - test might not be in fmf but e.g shell, or later pytest (or any future how:plugin).

[lukaszachy]

In addition to how:fmf I'd like to have possibility to use also other how plugins (currently just shell)

discover:
   how: shell
   archive: dist-git
    tests:
    - name: run check
      test: make check

Imho we can make archive (or better name) common option for discover plugins and for dist-git value it would:

1. Run fedpkg sources in cwd
2. Run rpm-build -bp on fetched sources
3. Finally do 'how' on outcome of previous step.

Would that work for you @martinpitt too?

[martinpitt]

@lukaszachy Sure! That's pretty much what I asked above as well.

[kkaarreell]

Just a thought on the rpmbuild -bp approach mentioned above. This step involves a check that all BuildRequires are installed (this could be skipped with --nodeps). While some of those packages may be really necessary for running actual tests, it may not always be the case. In fact, installing unnecessary BuildRequires would probably result in requiring CRB and Buildroot repos and due to having too many non-RHEL-supported packages installed possibly partially invalidate test results (like a test could be passing just because a non-distribution package is present, although it should not be required).
I am thinking whether an option to go the --nodeps way and manually specifying test requires in FMF won't be safer.

[lukaszachy]

Hopefully Autumn will bring more time to work on this. Moving to 1.9

[martinpitt]

Ooh, I somehow missed that this got fixed, nice! I tried this in https://src.fedoraproject.org/rpms/cockpit-podman/pull-request/37 , but it fails with

Found 1 plan.

/plans/upstream
    discover
        how: fmf
        directory: /var/ARTIFACTS/workdir-repository-None-9LLabI
        hash: 78bf01e
        summary: 0 tests selected
        warning: No tests found, finishing plan.

https://tmt.readthedocs.io/en/stable/spec/plans.html doesn't say a lot about this new option, but the example looks like that's all required?

[lukaszachy]

I'm still working on fixing this - Just guessing, but isn't this affected by #978 ?

[martinpitt]

@lukaszachy oh right, thanks for the pointer -- that was the reason why I ignored the closing of this issue last time. Cheers!