[BUG] [options.package_data] in setup.cfg does nothing

[greyltc]

setuptools version

57.0.0

Python version

3.9.5

OS

Arch

Additional environment information

No response

Description

I have this project: https://gitlab.com/greyltc/runpanel/-/tree/a93fac114f5443dc5ae55b9a8eb395b3244e282b

It has a setup.cfg file with

[options]
include_package_data = True

and

[options.package_data]
runpanel = *.yaml

As you can see from the repo there, there's a system_config.yaml file present that I expect to be included as package data. But after I install the package, and run

import importlib.resources
list(importlib.resources.contents('runpanel'))

I get only ['__pycache__', '__init__.py', '__main__.py', 'app.py', 'sysdbus.py']. There's no yaml file.

Expected behavior

I expect that when I follow the packaging docs for including package_data files here: https://setuptools.readthedocs.io/en/latest/userguide/declarative_config.html?highlight=package_data
that data should make it into my package.

How to Reproduce

1. Install my project here https://gitlab.com/greyltc/runpanel/-/tree/a93fac114f5443dc5ae55b9a8eb395b3244e282b
2. Notice that there are no .yaml package data files included

Output

$ ipython
In [1]: import importlib.resources
   ...: list(importlib.resources.contents('runpanel'))
Out[1]: ['__pycache__', '__init__.py', '__main__.py', 'app.py', 'sysdbus.py']

Code of Conduct

• I agree to follow the PSF Code of Conduct

[greyltc]

I've solved this today by REMOVING include_package_data = True from my setup.cfg:
https://gitlab.com/greyltc/runpanel/-/commit/ccbd1ea4258e7db168c7822e2dd7463317c95664

include_package_data = True has been okay for months, I have no idea why this began to break my stuff recently.

[kpfleming]

This is related to/caused by #1461.

[psarka]

Just a note: currently it does not work without include_package_data = True, and I can't get it to work in any way, except by using the MANIFEST.in file. My repro:

TESTFILE.md:

setup.cfg:

[metadata]
name = mypackage
version = 0.0.1

[options.package_data]
* = *.md

pyproject.toml

[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"

Commands:

python3.8 -m venv venv
source venv/bin/activate
pip install build
python -m build

dist/mypackage-0.0.1.tar.gz does not contain TESTFILE.md :(

[abravalheri]

Hi @psarka, could you show us how your file tree looks like?

Please note that include_package_data = True will only affect files inside the actual package directory, not the project root.

If you want to avoid using MANIFEST.in you can also have a look on setuptools-scm or another plugin for you specific VCS.

[psarka]

Ah, I see, I missed that the files have to be in the package folder. I'll stick to MANIFEST.in then.

Just in case, my tree is just these three files:

├── pyproject.toml
├── setup.cfg
└── TESTFILE.md

[abravalheri]

Ah, I see, I missed that the files have to be in the package folder. I'll stick to MANIFEST.in then.

Sorry if that is confusing in the docs: https://setuptools.pypa.io/en/latest/userguide/datafiles.html

Both include_package_data and package_data will only work with files inside the package directory (not the root of the project).

[abravalheri]

Hi @greyltc, I think this issue has been solved in #2844, and should already be fixed in the latest version of setuptools.

I tried to reproduce the problems with the commands bellow, and everything seems to be fine:

$ git clone https://gitlab.com/greyltc/runpanel.git
$ cd runpanel
$ virtualenv .venv
$ .venv/bin/python -m pip install buil
$ .venv/bin/python -m build

$ tar tf dist/runpanel-0.0.1.tar.gz
runpanel-0.0.1/
runpanel-0.0.1/LICENSE
runpanel-0.0.1/PKG-INFO
runpanel-0.0.1/README.md
runpanel-0.0.1/data/
runpanel-0.0.1/data/config_stubs/
runpanel-0.0.1/data/config_stubs/badger.yaml
runpanel-0.0.1/data/config_stubs/do_virt.yaml
runpanel-0.0.1/data/config_stubs/otter.yaml
runpanel-0.0.1/data/runpanel.desktop
runpanel-0.0.1/data/runpanel.svg
runpanel-0.0.1/pyproject.toml
runpanel-0.0.1/setup.cfg
runpanel-0.0.1/setup.py
runpanel-0.0.1/src/
runpanel-0.0.1/src/runpanel/
runpanel-0.0.1/src/runpanel/__init__.py
runpanel-0.0.1/src/runpanel/__main__.py
runpanel-0.0.1/src/runpanel/app.py
runpanel-0.0.1/src/runpanel/sysdbus.py
runpanel-0.0.1/src/runpanel/system_config.yaml
runpanel-0.0.1/src/runpanel/ui.glade
runpanel-0.0.1/src/runpanel.egg-info/
runpanel-0.0.1/src/runpanel.egg-info/PKG-INFO
runpanel-0.0.1/src/runpanel.egg-info/SOURCES.txt
runpanel-0.0.1/src/runpanel.egg-info/dependency_links.txt
runpanel-0.0.1/src/runpanel.egg-info/entry_points.txt
runpanel-0.0.1/src/runpanel.egg-info/requires.txt
runpanel-0.0.1/src/runpanel.egg-info/top_level.txt

$ unzip -l dist/runpanel-0.0.1-py3-none-any.whl
Archive:  dist/runpanel-0.0.1-py3-none-any.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2022-01-17 13:41   runpanel/__init__.py
      164  2022-01-17 13:41   runpanel/__main__.py
   131069  2022-01-17 13:41   runpanel/app.py
     3472  2022-01-17 13:41   runpanel/sysdbus.py
    19470  2022-01-17 13:41   runpanel/system_config.yaml
   255696  2022-01-17 13:41   runpanel/ui.glade
      684  2022-01-17 13:41   runpanel-0.0.1.data/data/share/applications/runpanel.desktop
     9706  2022-01-17 13:41   runpanel-0.0.1.data/data/share/icons/runpanel.svg
     2757  2022-01-17 13:41   runpanel-0.0.1.data/data/share/runpanel/configs/badger.yaml
      201  2022-01-17 13:41   runpanel-0.0.1.data/data/share/runpanel/configs/do_virt.yaml
     4095  2022-01-17 13:41   runpanel-0.0.1.data/data/share/runpanel/configs/otter.yaml
    19470  2022-01-17 13:41   runpanel-0.0.1.data/data/share/runpanel/configs/system_config.yaml
    35149  2022-01-17 13:55   runpanel-0.0.1.dist-info/LICENSE
      460  2022-01-17 13:55   runpanel-0.0.1.dist-info/METADATA
       92  2022-01-17 13:55   runpanel-0.0.1.dist-info/WHEEL
       59  2022-01-17 13:55   runpanel-0.0.1.dist-info/entry_points.txt
        9  2022-01-17 13:55   runpanel-0.0.1.dist-info/top_level.txt
     1640  2022-01-17 13:55   runpanel-0.0.1.dist-info/RECORD
---------                     -------
   484193                     18 files

$ .venv/bin/python -m pip install --no-deps dist/runpanel-0.0.1-py3-none-any.whl
$ cd tests
Python 3.8.10 (default, Sep 28 2021, 16:10:42)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib.resources
>>> list(importlib.resources.contents('runpanel'))
['app.py', 'sysdbus.py', 'system_config.yaml', 'ui.glade', '__init__.py', '__main__.py', '__pycache__']

Please let me know if there is any problem left, otherwise we can close this issue.

[kwinkunks]

I can't quite tell what was fixed, or if the docs were updated or not, but by way of another data point:

• I added a directory called include containing the files to include to the package directory, not the project root.
• I added this to setup.cfg:

[options.package_data]
* = include/*

And it worked: my files are included in site-packages/mypackage/include.

What did not seem to make any difference, inasmuch as if I only used these things it didn't help, and if I left them out it didn't matter:

• Adding include_package_data = True to setup.cfg.
• Adding a MANIFEST.in or MANIFEST file containing graft include or any variants I could think of like include include/* etc.

So I don't have either of those two things. This situation does not seem to match the docs you linked to but those don't really talk about setup.cfg as far as I can tell.

I'm on Python 3.9.7 and setuptools version 58.0.4.

[abravalheri]

Hi @kwinkunks thank you very much for the feedback.

Adding include_package_data = True to setup.cfg.
Adding a MANIFEST.in or MANIFEST file containing graft include or any variants I could think of like include include/* etc.

If you are specifying all the non-Python files with [options.package_data], then include_package_data should not make a lot of difference (I also seem to remember that files specified by package_data automatically get included in the sdist, so MANIFEST.in can also be neglected - but I might be wrong here).

include_package_data really shines when you use something like setuptools-scm. This way you don't need to specify package_data (which can get quite big if you have lots of folders).

this situation does not seem to match the docs you linked to

Are there specific parts of that page that need updating or are incorrect?

[kwinkunks]

Thank you very much for this explanation, and thanks for everything you're doing to make packaging easier.

Re the docs, at the top of the page it says this:

First, you can simply use the include_package_data keyword [...] This tells setuptools to install any data files it finds in your packages. The data files must be specified via the distutils MANIFEST.in file.

But that did not work for me. The only thing that worked, and it worked on its own (without either include_package_data or MANIFEST.in), was options.package_data.

I am using only setup.cfg, with only the package name being set in setup.py, so I'm not 100% certain I'm mapping the advice on this page (which all seems to use setup()) to my situation.

I need to look at setuptools_scm... I have not been paying attention to it.

[abravalheri]

Thank you for the kind words and for the feedback @kwinkunks. I will try to propose a change in the docs soon (I have to admit I managed to accumulate quite a few TODOs for my open source projects this week, so I might take some time, please feel free to submit a doc PR meanwhile!).

Since I fixed the previous issue, I keep a benchmark for this class of problems in https://github.com/abravalheri/experiment-setuptools-package-data:

I will re-run it with the most updated version of setuptools to make sure everything is still fine and that we are not missing any bug introduced in the latest commits.

Any chance you are running setuptools<=58.3?

---

UPDATE: I re-run the experiment for setuptools==60.6.0 and it seems that the results summarised in the table above are the same.

[kwinkunks]

All good -- that table is gold. Thank you very much! (I was on v58.0.4, now updated.)

[abravalheri]

I think I will close this issue for now...
There were some updates in the docs to highlight all the package_data settings only refer to files inside the package directory, and also a new section in https://setuptools.pypa.io/en/latest/userguide/miscellaneous.html#controlling-files-in-the-distribution, so hopefully it is more clear now.

Any PR for improve the docs would also be appreciated!