0.2.2: missing files in pypi sdist tarball: unable to run tests

[dvzrv]

Hi! I'm currently bringing python-dbus-next to the official Arch Linux repositories.
As best practice we usually run tests against the packages (if there are any). Unfortunately this fails due to missing files, when using the sdist tarball on pypi:

============================= test session starts ==============================
platform linux -- Python 3.9.6, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python
cachedir: .pytest_cache
rootdir: /build/python-dbus-next/src/python-dbus-next-0.2.2, configfile: pytest.ini
plugins: timeout-1.4.2, asyncio-0.15.1
timeout: 5.0s
timeout method: signal
timeout func_only: False
collecting ... collected 74 items / 2 errors / 72 selected

==================================== ERRORS ====================================
_________________ ERROR collecting test/test_introspection.py __________________
test/test_introspection.py:5: in <module>
    example_data = open(f'{os.path.dirname(__file__)}/data/introspection.xml', 'r').read()
E   FileNotFoundError: [Errno 2] No such file or directory: '/build/python-dbus-next/src/python-dbus-next-0.2.2/test/data/introspection.xml'
___________________ ERROR collecting test/test_marshaller.py ___________________
test/test_marshaller.py:20: in <module>
    table = json.load(open(os.path.dirname(__file__) + '/data/messages.json'))
E   FileNotFoundError: [Errno 2] No such file or directory: '/build/python-dbus-next/src/python-dbus-next-0.2.2/test/data/messages.json'
=========================== short test summary info ============================
ERROR test/test_introspection.py - FileNotFoundError: [Errno 2] No such file ...
ERROR test/test_marshaller.py - FileNotFoundError: [Errno 2] No such file or ...
!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!
============================== 2 errors in 0.30s ===============================

[acrisci]

Hi,

Thanks so much for packaging. Running tests are only supported in the provided docker container because they interact with a real dbus daemon.

python-dbus-next/Makefile

Lines 22 to 24 in 7d651b6

	docker-test:
	docker build -t dbus-next-test .
	docker run -it dbus-next-test

[dvzrv]

I can run the tests using dbus-run-session -- pytest -v.

Unfortunately in 0.2.3 the sdist tarball on pypi is still missing some files:

==================================== ERRORS ====================================
_________________ ERROR collecting test/test_introspection.py __________________
test/test_introspection.py:5: in <module>
    example_data = open(f'{os.path.dirname(__file__)}/data/introspection.xml', 'r').read()
E   FileNotFoundError: [Errno 2] No such file or directory: '/build/python-dbus-next/src/python-dbus-next-0.2.3/test/data/introspection.xml'
___________________ ERROR collecting test/test_marshaller.py ___________________
test/test_marshaller.py:20: in <module>
    table = json.load(open(os.path.dirname(__file__) + '/data/messages.json'))
E   FileNotFoundError: [Errno 2] No such file or directory: '/build/python-dbus-next/src/python-dbus-next-0.2.3/test/data/messages.json'
=========================== short test summary info ============================
ERROR test/test_introspection.py - FileNotFoundError: [Errno 2] No such file ...
ERROR test/test_marshaller.py - FileNotFoundError: [Errno 2] No such file or ...
!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!
============================== 2 errors in 0.31s ===============================

[leycec]

Just hit this. In my case, I'm packaging dbus_next for Gentoo Linux at my third-party overlay. We prefer to locally run tests on installation for obvious reasons, because tests passing == package working. ^{usually. crossing fingers here.}

Sadly, local test execution doesn't appear to be supported. We're reluctantly disabling tests for dbus_next – but not particularly happy about that.

Perhaps the requisite files could simply be bundled with release tarballs as @dvzrv suggests? It's usually best to just let the users sort it out. 🤷‍♂️

[acrisci]

If you would like to make that change, you can go ahead. However, keep in mind that these are integration tests and may make assumptions about dbus configuration that may not be true on your system. You may use dbus-run-session to run the tests but this may not be compatible with future changes to the test suite that may be made such as system bus interactions, alternative transports, or testing of a daemon with an alternative configuration. You should decide which tests you need to skip based on your application. I will merge changes to some tests that make it possible to run with dbus-run-session when it's not intrusive, but I won't merge changes into the tests for compatibility between systems or bus implementations.

[leycec]

Affirmative. Would it also be feasible to gracefully @pytest.mark.skipif(..., reason='Non-portable dbus test.') the relevant integration tests away, in that case? Tests that can only reliably run within a unique non-portable environment (e.g., a particular Docker container running a particular dbus configuration) should ideally just be skipped with an informative message rather than fatally halting the entire test suite.

Just my 🪙 🪙 as a friendly package maintainer, of course.

You should decide which tests you need to skip based on your application.

B-b-but... I'm packaging for Gentoo Linux. There is no specific "application" as such.

The intended userbase is literally all users running Gentoo Linux. The intended use case is dbus_next installed as a system-wide Python package for use by other downstream reverse dependencies also installed as system-wide Python packages (e.g., bleak).

Again, the test suite itself should gracefully scale by selectively ignoring unit and integration tests that are inapplicable to the current execution environment. Thankfully, @pytest.mark.skip makes that trivial – assuming it's trivial for your test suite to detect whether it's running in your Docker container, of course. That should be trivial. That's usually just a lookup of a unique environment variable exported by your Docker container.

Annnnnyway. I've marked our Gentoo ebuild for dbus_next as RESTRICT=test for now, meaning that tests are known to fail and should be ignored. I'm fine with that. It is what it is. 🤷‍♂️

[dvzrv]

On Arch Linux we have now started to discourage the use of PyPI hosted sdist tarballs (https://rfc.archlinux.page/0020-sources-for-python-packaging/).