package_data not doing recursive glob calls

[mvaal]

Even though glob supports it, glob support for recursive ** searching is not enabled when specifying package_data. This is not clear in the documentation.

setuptools/setuptools/command/build_py.py

Line 106 in a94ccbf

globs_expanded = map(glob, patterns)

This means that only single * searches in a specific directory are supported. By changing the glob function to support recursive, this would allow us to specify ** patterns to a directory instead of having to specify every sub-directory in the structure.

Reference:
https://github.com/python/cpython/blob/master/Lib/glob.py#L18

[jaraco]

Yes, I can confirm that without calling glob(pattern, recursive=True), the ** pattern isn't expanded.

What this issue needs is more background. What is a use-case where having recursive globbing support would be helpful?

More importantly, this routine is for find_data_files in a package. Since a Python package is already defined as a directory containing files, it may not even be viable for a package to contain directories of data. Even if you're able to cause subdirectories and their contents to be added as package data, I'm not sure that would be a supported scenario. Before adding facilities to allow a packager to more easily add this data to a package, we should be sure that's something that the packaging systems endorses/supports.

[mvaal]

Thanks for getting back. I have a nested directory of configuration files (.yml) and a nested directory of validation files (.json). Unfortunately, I have to go through each sub directory in order to include them in the package. This works and is working for my current use case, but every time I add addition sub-directories, I have to remember to add another path.

[options.package_data]
module = config/A/1/*.yml, config/B/1/*.yml, validation/A/*.json, validation/B/*.json

I would prefer to just have to do this (in fact I originally tried this, but didn't work which is why I reported the issue).

[options.package_data]
module = config/**/*.yml, validation/**/*.json

I also tried *.yml, *.json that the docs alluded to working, but it does not (same issue non recursive so only works on root directory files).

[options.package_data]
# If any package contains *.txt or *.rst files, include them:
* = *.txt, *.rst
hello = *.msg

[JaapJoris]

Agreed, it would be massively helpful if I could use the ** glob in setup.py's package_data. I deploy Django apps which include many templates in potentially endless subdirectories. There is no glob pattern except **/*.html to describe their paths.

[thenewguy]

I just found this issue working with Django templates as well. Recursive glob support would be tremendously helpful when packaging Django apps that include templates

[psam44]

I got the same problem with stub files for typing. These data files are *.pyi and may be spread at multiple folder levels.

I have two workarounds to this difficulty.

One is with a custom implementation:

import os
from typing import List

def find_stubs(top: str) -> List[str]:
    stubs = []
    for root, _, files in os.walk(top):
        dir = root.split(os.path.sep, 1)[-1] # rid of 'top'
        for f in files:
            if f.endswith('.pyi'):
                if os.path.sep in root:
                    f = os.path.join(dir, f)
                stubs.append(f)
    return stubs

setup(
    …
    package_data={'foo': find_stubs('foo')},
    …
)

Another one is more generic, with a manifest template:

setup(
    …
    include_package_data=True,
    …
)

MANIFEST.in

recursive-include foo *.pyi

Working with Django, I prefer the second way because I also need to include many other data files:

recursive-include foo/locale *.po *.mo
recursive-include foo/static *
recursive-include foo/templates *.html *.txt
graft docs
prune docs/_build

[cvanelteren]

I would like to add the issue that for my cython code setuptools does not copy the data files to site-packages.

Sources list my pxd files but they are not transferred to the install directory. I have both tried using MANIFEST.in as well as package_data = {"": "**pxd **pyx".split()}. A workaround form was to use data_files which is not preferred ...