Discover from dist-git source should explore metadata under the source directory

[psss]

Currently the source tarball is unpacked under the discover/default/tests directory and test discover is done from there. This brings two problems:

• When the source tarball includes the .fmf directory it is ignored as a nested fmf.Tree.
• Discovered tests names receive an extra prefix from the source directory

We should discover tests directly from the unpacked directory and make sure that it has metadata tree root defined, e.g. call tmt init if needed.

@martinpitt ran into this problem when working on this cockpit pull request.

[lukaszachy]

Thank you for the report, I'll try to create a fix this week.
Proposal: Expectation is that extracted source contains .fmf root. When it is not there tmt will do tmt init in extracted sources.

New option --dist-git-init will change the behaviour:

• False -> disable feature (don't run tmt init at all),
• str -> relative path in extracted sources where tmt init should be called,
• True -> call tmt init in the root of extracted sources [default]

[psss]

The --dist-git-init feature drafted above looks like a very elegant solution to me. Thanks!

[bcl]

I have a possibly separate problem with dist-git-source, I expected it to act like it did in the STI tests and extract the source and then run my test script, but it seems to be extracting the source, not finding any tests, and then not running the execute section. The test has already been found, that's what it is running, so why is execute skipped?

[lukaszachy]

I expected it to act like it did in the STI tests and extract the source and then run my test script,

That is a bit different I'm afraid. In that case you are not expected to search for any tests in the extracted sources, right?
So dist-git-source just adds code but metadata are already present.
I think it could be quite easy to add.

[bcl]

I think I figured it out. The differences between plans and tests aren't obvious and I got it working by putting the actual test execution into tests/unit.fmf and the setup with dist-git-source into a corresponding plan under plans/

In that case you are not expected to search for any tests in the extracted sources, right?

Right, the tmt tests are in dist-git.

[psss]

Just realized one aspect which will probably make the implementation less straightforward: If I remember correctly, there can be multiple source tarballs included in the sources list, right? What shall we do in such case? We cannot just randomly choose where to perform the tmt init. Plus combining tests from both sources would need the metadata root to be in discover/default/tests directory.

[lukaszachy]

Needs more time to evaluate possible corner cases

[martinpitt]

Would it be possible to provide a solution for a single/first tarball, i.e. look at Source0 only? The vast majority of packages have only one tarball anyway, and even more only have tests in their "primary" source tarball. We used to have a hack in our cockpituous scripts, but with the world moving to packit (which does not have "nice" ways to arbitrarily hack files in dist-git) this becomes more awkward to work around.

[martinpitt]

For those following along, I created this hack and successfully tested it: https://github.com/cockpit-project/cockpit/pull/17378/files (but.. eww!)

[lukaszachy]

Could you try with #1128 please? I believe having more sources doesn't matter for that implementation.

[psss]

Should be covered by #1128.

[martinpitt]

@psss, @lukaszachy : Sorry for the late reply, sprint week and long weekend. I'm super happy to see this, thanks! Does this still need a release, or does the Testing Farm use git? I. e. time to retry https://src.fedoraproject.org/rpms/cockpit-machines/pull-request/4# now or later? Thanks!

[psss]

I've just pushed out bodhi updates. It should get to the Testing Farm around tomorrow.

[martinpitt]

Thanks @psss! We collided, I rebased https://src.fedoraproject.org/rpms/cockpit-machines/pull-request/4 an hour ago, and it still failed. Nice! Then I'll retry with 1.14.0 once that lands.