tests are missing from pypi downloads

[gdt]

The 6.15.0 release notes say there are pytest tests. In preparing an update for pkgsrc to 6.151, I added TEST_DEPENDS based on the output from trying to run pytest (which is default), pytest, pytest-cov and anyio. But, it found no tests.

Looking at the unpacked distirbution file from pypi, I can find no tests. The test directory is missing.

(I'm taking it as obvious that users should be able to build from pypi downloads and run tests in their environment, vs tests only being appropriate for git checkouts. pkgsrc encapsulates testing as well as building.)

[philthompson10]

I disagree that it is obvious, see this discussion.

My view is that the purpose of the tests is not for users to test their installation, rather they are for developers to make sure they haven't introduced any regressions.

[gdt]

I agree to disagree about obvious :-)

The purpose of tests is to verify/check that software behaves correctly. Software behavior is a combination of the code in the repository and the operating environment, including dependencies and operating system. It is not possible for the developers of a project to test in all environments. I suspect for example that you haven't tested on NetBSD 10 aarch64 with dependencies provided from pkgsrc.

The division into "developers" and "users" isn't really right, though and I suspect this is the root of how we see things differently. There is a third category "packaging systems" and/or "packagers". Upstrream projects, as packaging systems call them, produce distfiles (sdist in python). Packaging systems have control files that explain where to get that distfile, what dependencies are needed, any patches to fix problems (not yet fixed in a release, but which should be reported), etc. Within a packaging system, one can build a binary package that can then be installed by a package manager, and one can run tests.

pkgsrc in particular is a source-focused packaging system, so users of pkgsrc building their own packages is normal. Especially when they build packages on platforms likely not tested by upstream, it's useful to run upstream's regression tests. My experience in packaging many things is that fairly often these tests turn up some issue, and sometimes it is a platform bug, sometimes it is an upstream bug, and sometimes it is a test bug. An upstream bug exanple is using bare abs() in C++ when std::abs() should have been used. C++ allows various resolutions of that, and some systems resolve to C99 abs(), which truncates. The fix is easy, to use std::abs(), but my point was that this bug was surfaced by running tests in packaging context.

I don't see why it's harmful to include them. Surely they aren't that big, and the benefit in human time of finding a single bug is huge.

As it is I test new sip before putting it into pkgsrc by building a package and then rebuilding qgis with it, and seeing if qgis works. Even with tests I'll still do that, but the more packages run their tests with "make test" in the package dir, the better.

[philthompson10]

I don't see a difference between pkgsrc and the many Linux distros I don't test. What the distros do is test using the repository so that regressions get identified before a release rather than afterwards.

However I accept that the more testing the better so I'll have a look at what's involved in including them.

[gdt]

The only difference with pkgsrc is that it runs on a greater variety of systems, and in particular it is run by pkgsrc users on systems that the pkgsrc maintainer couldn't test on (due to human cycles), both different OS and different CPU arch.

I do test from upstream git several things (qgis, proj, gdal), but I package far more projects than I have the capacity to follow so closely.

I hope it's just something that feels like "extra_sdist = tests".

Thanks for being willing to look into this.