Please include docs and tests directories upon releases on PyPI

[P-EB]

Hi,

I'm packaging sphinx-tabs in Debian, and I really appreciate being able to test the package using your tests and providing the compiled documentation and the potential examples along with the package I upload, so that the users without an internet connection can use them.

Could you please consider adding the docs and tests directories on your releases on PyPI?

Thanks in advance!

[welcome]

Thanks for opening your first issue here! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out EBP's Code of Conduct. Also, please try to follow the issue template as it helps other community members to contribute more effectively.

If your issue is a feature request, others may react to it, to raise its prominence (see Feature Voting).

Welcome to the EBP community! 🎉

[foster999]

The HTML docs are surprisingly large relative the package source, so I'm not sure we could justify including them. I appreciate that it would be preferable to see a working example in the package, but including the raw (.rst) docs and tests would be a simple change if that would still be an improvement?

[P-EB]

Hi @foster999

I realize my message was less clear than what I expected.

I'd like to be able to ship the HTML documentation along with the package, but I was asking you if you could include the rst files in the manifest, not the HMTL ones, which I'll produce at build-time.

So if you could incldue the raw RST docs and the test files it'd be perfect for me!

Thanks a lot. :)

[chrisjsewell]

I would be against this; if every package included all this in their distribution it would significantly increase the size of python environments. Distributions should be limited to only the essential code.

If you want all this, why not just clone the repository and install in development mode? pip -e .

[P-EB]

Many distributions (and Debian is part of it) tend to be keen on providing the actual documentation going with the package instead of forcing users to read one which can be not accurate on the internet (because it is for a more up-to-date version of the code), or to rely on non-distro package applications like pip. It's a policy choice which also reflects the idea that a software without its documentation is in many cases less worthy. (needless to say the documentation is part of the actual software)

This can also be beneficial when one doesn't have an internet access.

When suitable, the compiled documentation along with the RST files are shipped in specific -doc package that the user can install or not and which is not necessary for the main package to be used.

The rst files are actually not that big anyway.

[chrisjsewell]

Again I would ask; if you want to achieve this, why not just directly download the repository source code and install from there?

[chrisjsewell]

@pradyunsg, you're the packaging man lol, what's your opinion?

[P-EB]

Again I would ask; if you want to achieve this, why not just directly download the repository source code and install from there?

I guess the answer is the same as the one to "why do Linux Distributions exist?"

[chrisjsewell]

I guess the answer is the same as the one to "why do Linux Distributions exist?"

and the answer is ... 😬

[P-EB]

Because a lot of people is quite happy not having to test that each and every component of their system works together, and to compile these on their own.

I know some people are happy otherwise, if you are one of them, feel free to compile and use your window manager on your own!

But that gets us far from the original question which is "please provide a couple python files and some rst files in your release".

[Daltz333]

I don't see the benefit for the average consumer. Additionally, it's really not much more work to just clone the repository yourself, as previously stated.

I'm of the group answering "no" here.

[Daltz333]

If there are no other comments from other maintainers, I'd like to close this issue and mark it as "wontfix"

[pradyunsg]

I don't think including tests/docs in PyPI sdists is necessary, and would go as far to say that doing so is generally harmful IMO. This is obviously an opinion, and it might make sense for certain types of projects; but not for Python projects that get distributed on PyPI.

I'm aware of lots of packages on these distros where the repackagers treat the Github repository's tags as the source tree (the easiest one to point to being pip). As noted above, bundling these increases the size of the distributions for basically no gain for the vast majority of the users and unnecessarily increases bandwidth usage/costs for everyone involved.

I'd strongly recommend using the git tags in this repository as the source of truth for packaging this into a Linux distro. That's also what Fedora did for sphinx-inline-tabs for example. (yes, that's a different package)

[pradyunsg]

Actually, before we close this out, may I ask which part of the Debian policy led you to request the upstream project to bundle more data in their PyPI sdist?

My understanding was that this isn't necessary and using a GitHub based source distribution isn't inherently any different from using a PyPI based one for repackaging from Debian's PoV.

Heck, based on my past discussions with Debian package maintainers, using the Github tarball is preferred?

[chrisjsewell]

FYI, to help a little I've added automation to RTD, to activate version tags:

so now on https://sphinx-tabs.readthedocs.io new version tags should appear in the selection tab:

[P-EB]

Actually, before we close this out, may I ask which part of the Debian policy led you to request the upstream project to bundle more data in their PyPI sdist?

My understanding was that this isn't necessary and using a GitHub based source distribution isn't inherently any different from using a PyPI based one for repackaging from Debian's PoV.

Heck, based on my past discussions with Debian package maintainers, using the Github tarball is preferred?

The main issue with Github releases and git tags is that they tend to be less signed by upstream.

In your case it doesn't change much as your pypi upload is not signed, but I have the habit of relying on PyPi because there is the benefit of the upstream GPG signatures for plenty packages.

GPG signatures are useful for me to make sure the thing I rely on for packaging is actually what upstream intended us to get.

[kloczek]

I don't think including tests/docs in PyPI sdists is necessary, and would go as far to say that doing so is generally harmful IMO. This is obviously an opinion, and it might make sense for certain types of projects; but not for Python projects that get distributed on PyPI.

What you mean "necessary"?

I'm aware of lots of packages on these distros where the repackagers treat the Github repository's tags as the source tree (the easiest one to point to being pip). As noted above, bundling these increases the size of the distributions for basically no gain for the vast majority of the users and unnecessarily increases bandwidth usage/costs for everyone involved.

If sdist tar ball does not includes test suite it is ONLY way to perform tests during packaging processes.

I'd strongly recommend using the git tags in this repository as the source of truth for packaging this into a Linux distro. That's also what Fedora did for sphinx-inline-tabs for example. (yes, that's a different package)

Please provide justification of that recommendation.

I'm working on Linux/Solaris distro which on building distro Linux rpm and IPS Solaris distribution uses rpm spec files. In that distribution including %check section is obligatory. That is because testing just build package is not only about testing just build package but as well all packages installed in build env.
Ergo: more test units -> higher probability that some errors in just packages stuff or used during build packages will be cached on early packaging stage.
By tripping dist tar balls package maintainer is tripping down such opportunity to early detect some issues.

Using above methodology I was able catch on spot hundreds of issues only in last few months.

Using VCS tagged tar balls has another very useful propose. It quarantines that on integrating on top of the release any VCS commit will quarantines that it will be applied without rejections. sdist tar balls does not provide that feature. Other thing is that using fact that new release tag is possible design and use automation which automatically will try to build new package without human intervention and flag only to packagers that it is some new stuff to review manually without spend single human/second.
This why in case of the distribution on which I'm working:

[tkloczko@barrel SPECS]$ grep 'Source.*VCS' python-*.spec | wc -l; ls -l python-*.spec |wc -l
392
551

and still many more packages can be transformed to save a lot of time people working on that stuff and remove a lot of manual work on which humans can make many mistakes.