Cannot exclude file in bdist

[ghost]

Originally reported by: tomi77 (Bitbucket: tomi77, GitHub: tomi77)

---

• setuptools==20.2.2
• https://github.com/tomi77/tastypie-sorl-thumbnail

MANIFEST.in

#!

exclude tastypie_sorl_thumbnail/tests.py
exclude tastypie_sorl_thumbnail/models.py

python setup.py sdist excludes these files. python setup.py bdist does not.

More info: https://bitbucket.org/pypa/wheel/issues/99/cannot-exclude-directory#comment-25155235

---

• Bitbucket: https://bitbucket.org/pypa/setuptools/issue/511

[viraptor]

This is still the case for bdist and bdist_wheel.
Running with:
setuptools (38.4.0)
wheel (0.30.0)

The files can be excluded from sdist by using MANIFEST.in, but they're packaged in the wheel anyway.

[busimus]

I'm not sure it's the same functionality, but replacing exclude with prune made it exclude files for me with bdist_wheel.

[viraptor]

I'm using prune, but it doesn't seem to work. Package in question is phply (https://github.com/viraptor/phply)

[busimus]

Have you tried deleting the build directory? For me the changes to MANIFEST.in only seem to apply after a clean build.
This has no effect on exclude, but prune starts working.

[viraptor]

Even with build and dist deleted, running python setup.py clean sdist bdist_wheel results in tests in the wheel, but not in the sdist.

[maerteijn]

I'm encountering the same problem, so it probably still exists. Maybe my explanation will ring a bell to somebody, or maybe we are doing something completely wrong which is not clearly documented.

It seems like the problem is related with building the project, and limited to .py files only.

I created an example repo to demonstrate the problem here: https://github.com/maerteijn/manifestexample

It has a excluded.py and a excluded.txt file which are marked as excluded in the MANIFEST.in file. However, excluded.py still is included in bdist and bdist_wheel distributions.

I've tested this with python 2.7 and 3.6 with:
pip (9.0.2)
setuptools (39.0.1)
wheel (0.30.0)

Running setup.py sdist shows that both files are not included:

$ python setup.py sdist
running sdist
...
reading manifest template 'MANIFEST.in'
writing manifest file 'manifestexample.egg-info/SOURCES.txt'
running check
...
creating manifestexample-0.0.1/manifestexample.egg-info
copying files to manifestexample-0.0.1...
copying MANIFEST.in -> manifestexample-0.0.1
copying README.md -> manifestexample-0.0.1
copying setup.cfg -> manifestexample-0.0.1
copying setup.py -> manifestexample-0.0.1
copying manifestexample/__init__.py -> manifestexample-0.0.1/manifestexample
copying manifestexample/cmd.py -> manifestexample-0.0.1/manifestexample
copying manifestexample/manifest.py -> manifestexample-0.0.1/manifestexample
copying manifestexample.egg-info/PKG-INFO -> manifestexample-0.0.1/manifestexample.egg-info
...
Creating tar archive

When running setup.py build, you see that the first step is to copy the files to the build directory,
which includes excluded.py and skips excluded.txt:

$ python setup.py build
running build
...
creating build/lib/manifestexample
copying manifestexample/__init__.py -> build/lib/manifestexample
copying manifestexample/cmd.py -> build/lib/manifestexample
...
copying manifestexample/excluded.py -> build/lib/manifestexample
...
copying manifestexample/manifest.py -> build/lib/manifestexample
running egg_info
writing requirements to manifestexample.egg-info/requires.txt
writing manifestexample.egg-info/PKG-INFO
writing top-level names to manifestexample.egg-info/top_level.txt
writing dependency_links to manifestexample.egg-info/dependency_links.txt
writing entry points to manifestexample.egg-info/entry_points.txt
reading manifest file 'manifestexample.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'manifestexample.egg-info/SOURCES.txt'

$ ls -l build/lib/manifestexample/
total 16
-rw-r--r--  1 martijn  staff    0 Mar 21 13:29 __init__.py
-rw-r--r--  1 martijn  staff  231 Mar 21 13:35 cmd.py
-rw-r--r--  1 martijn  staff   45 Mar 21 13:34 excluded.py
-rw-r--r--  1 martijn  staff    0 Mar 21 13:29 manifest.py

The SOURCES.txt file however still respects MANIFEST.in:

$ cat manifestexample.egg-info/SOURCES.txt 
MANIFEST.in
README.md
setup.cfg
setup.py
manifestexample/__init__.py
manifestexample/cmd.py
manifestexample/manifest.py
manifestexample.egg-info/PKG-INFO
...
manifestexample.egg-info/requires.txt

Running setup.py bdist_wheel clearly shows that excluded.pyis included (and excluded.txt is not)

$ python setup.py bdist_wheel
running bdist_wheel
...
writing requirements to manifestexample.egg-info/requires.txt
...
reading manifest file 'manifestexample.egg-info/SOURCES.txt'
eading manifest template 'MANIFEST.in'
writing manifest file 'manifestexample.egg-info/SOURCES.txt'
installing to build/bdist.macosx-10.12-x86_64/wheel
running install
...
creating build/bdist.macosx-10.12-x86_64/wheel/manifestexample
copying build/lib/manifestexample/__init__.py -> build/bdist.macosx-10.12-x86_64/wheel/manifestexample
copying build/lib/manifestexample/cmd.py -> build/bdist.macosx-10.12-x86_64/wheel/manifestexample
...
copying build/lib/manifestexample/excluded.py -> build/bdist.macosx-10.12-x86_64/wheel/manifestexample
...
copying build/lib/manifestexample/manifest.py -> build/bdist.macosx-10.12-x86_64/wheel/manifestexample
running install_egg_info
Copying manifestexample.egg-info to build/bdist.macosx-10.12-x86_64/wheel/manifestexample-0.0.1-py2.7.egg-info
running install_scripts
creating build/bdist.macosx-10.12-x86_64/wheel/manifestexample-0.0.1.dist-info/WHEEL
creating '/Users/martijn/Dev/oss/manifestexample/dist/manifestexample-0.0.1-py2.py3-none-any.whl' and adding '.' to it
adding 'manifestexample/__init__.py'
adding 'manifestexample/cmd.py'
...
adding 'manifestexample/excluded.py'
...
adding 'manifestexample/manifest.py'
adding 'manifestexample-0.0.1.dist-info/DESCRIPTION.rst'
adding 'manifestexample-0.0.1.dist-info/entry_points.txt'
adding 'manifestexample-0.0.1.dist-info/metadata.json'
adding 'manifestexample-0.0.1.dist-info/top_level.txt'
adding 'manifestexample-0.0.1.dist-info/WHEEL'
adding 'manifestexample-0.0.1.dist-info/METADATA'
adding 'manifestexample-0.0.1.dist-info/RECORD'

[willcharlton]

Being able to exclude *.py would be great for supporting *.py[co] wheel distributions.

[pabloariasal]

@maerteijn I think it excludes the .txt because it not added per default anyways, see this. Pyhton modules, on the other hand, are always added per default.

[jaraco]

I've reclassed this issue as an enhancement, as it appears to be working as designed, although the design could be improved. I welcome continued work on the associated pull request or another.

[maerteijn]

According to pypa/packaging.python.org#306 the MANIFEST.in is not used at all with bdist_wheel

[jaraco]

To clarify, the MANIFEST.in (and resulting MANIFEST) represent a mechanism for source distributions. Their purpose is to help create a reliable description of the sources of a project (from which to build/install). The build steps, however, have different protocols and semantics; they, given a source checkout or source distribution, build/install the project from those sources following their rules.

It may be unwise to expect MANIFEST (and its template) to affect the build/install commands; perhaps these commands should instead have their own mechanism to determine specific file exclusion or inclusion.

Or, thinking about it from a different perspective, perhaps a build operation should only ever operate on a source distribution, such that if a file was excluded from the sources, it would also be excluded from the build.

It's not obvious to me what the best solution is here.

[maerteijn]

Or, thinking about it from a different perspective, perhaps a build operation should only ever operate on a source distribution, such that if a file was excluded from the sources, it would also be excluded from the build.

In a first thought that would feel pretty logic, but it isn't as sometimes you probably want other files to be included when you distribute a source or binary distribution. And it would drastically change the current behavior which is not really desirable I suppose.

I think the core of the problem is that there is not one, clear way of specifying what should be included or excluded in source / binary distributions.

For source distributions we have a MANIFEST.in file, a pretty fine grained, more logical approach to include / exclude files.

For binary distributions we have a (hacky) way in the form of https://setuptools.readthedocs.io/en/latest/setuptools.html#using-find-packages And it does does not support excluding single files or non-python files at all. Which is something desired sometimes as why this issue exists. And to make the confusion complete, the build command is using the manifest file somehow, but it's hardcoded to ignore .py files: https://github.com/pypa/setuptools/blob/master/setuptools/command/build_py.py#L149

We can override the distutils method of selecting the package files (see for an example https://stackoverflow.com/questions/5974982/setup-py-exclude-some-python-files-from-bdist), but writing specific python code to package your python modules feels strange for something that should be supported by the build tools we already have.

Related to this: I've seen this PR which at least gives us a way of excluding files as well, but then we are using the exclude patterns for packages also for single files, which would be a "band aid solution" and not really well thought of.

Adding an option to specify a manifest file to the build command (and bdist / bidst_wheels commands) would be a simple solution for this I guess, so we all can use one mechanism for the inclusion and exclusion of files.