GitHub actions seems to still be using sdist - should this be adjusted to bdist_wheel?

[sgbaird]

Description of your problem

Please provide a minimal, self-contained, and reproducible example.

The following command would probably work.

putup --github-actions

See this GitHub actions runner.

I got a "File too large" error when trying to publish to PyPI because files other than what's in src are added to the distribution. I came across #404 which mentioned the use of sdist vs. bdist_wheel, where the latter is recommended. Checking the GitHub actions log, I'm seeing that sdist is being used during:

pipx run --spec tox==3.27.1 tox -e clean,build

which calls:

python -m build

where build is a Python package and produces the message:

...
Getting build dependencies for sdist...
...

Please provide the full traceback using the --very-verbose flag.

# [The error output here]

Please provide any additional information below.
I'd like for only src files to be added to the distribution by default.

Versions and main components

• PyScaffold Version: 4.2.3.post1.dev10+g7a0f254
• Python Version:
• Operating system: Windows
• How did you install PyScaffold: (conda/pip) I don't remember, but can check if it's pertinent

[abravalheri]

Hi @sgbaird , when using build without any flags both sdist and wheel are generated by default (the wheel file is generated from the sdist, to improve testability, this is a common practice in the ecosystem).

The sdist works more or less as a snapshot of the repository, so it will naturally include more files than the wheel (e.g. docs, tests, auxiliary scripts etc). In general people preparing packages for Linux distributions really like this behaviour...

If your sdist is too big for being uploaded, maybe you can consider not building/submitting the sdist for that particular project.

Alternatively you can also review the .gitignore file to not store big files in the repository, or add a MANIFEST.in to remove some of the directories from the sdist (PyScaffold already sets up setuptools-scm, so I believe that you don't need any include or graft instructions, just prune and exclude).

But in general, I believe that this is not a general problem for all types of packages...
@FlorianWilhelm , this issue seems to be related to the existence of folders with big files in the dsproject extension. Is there anything specific we should be doing for that extension?

[sgbaird]

Thanks for clarifying! I'll close the PR

[FlorianWilhelm]

Sorry for coming so late to the party. Actually, I hope that sdist packages are hopefully soon a thing of the past and going back to a MANIFEST.in would surely complicate things for newcomers.

@abravalheri : What if we drop it all together by default as @sgbaird suggested?

[abravalheri]

Hi @FlorianWilhelm, this is certainly possible, but I would like to share this thread https://discuss.python.org/t/should-sdists-include-docs-and-tests/14578 as a reading material for this discussion.

Some distribution repackagers (the ones that create OS-specific packages for ubuntu, arch, homebrew, etc...) really like to work from the source instead of the built wheel. And also there is some level of consensus that submitting sdists to PyPI whenever possible is a good thing...

Usually we can place these things in a spectrum:

reposity -->
sdist (may contain *some* level of compilation/optimization, e.g. Cython => c, but it is still platform independent and preserves development-time files like tests, docs) -->
wheel (platform-specific, may contain compilated/optimized files) -->
installed files (platform-specific, contain compiled/optimized files, including .pyc)

[FlorianWilhelm]

Thanks @abravalheri, I didn't know that the intention of sdist is to actually include everything from the repository itself.

What if we add a comment that says something like:

# use the following line instead if you want to publish only wheel files
# run: pipx run tox -e clean,build -- --wheel

or the other way around with wheel and sdist. Just a thought. I am really no expert regarding the practical usage of sdists.

[abravalheri]

@FlorianWilhelm I created something along these lines in #694.

[abravalheri]

@sgbaird, I think that #694 addresses the main concerns that are captured in this issue.

Now, there is an entry in the FAQ explaining the approach for sdists, which also indicates a possible workaround for publishing the package when the sdist is too big.

As discussed in this thread, sdists are expected to contain docs, tests, and everything that is used during development time.

If you have files in your repo that are not used (or that are just optional) during development time, you can still use the approach I mentioned here, i.e. MANIFEST.in just as a complement to setuptools-scm. I did not include this part in the FAQ because the process may confuse newcomers as argued by Florian.

If you are using data files, you can also consider using a solution like DVC to avoid storing big files in the git repo.

For now I will close this issue because it looks completed from my point of view, but please let me know if you have a different opinion. We can re-open this issue.