Recursive include of data_files

[FlorianWilhelm]

Using MANIFEST.in has the advantage of the recursive-include command. Using wildcards in data_files does not work recursively. Can we somehow fix this?

[xhochy]

We could pass the data_files through the glob module to resolve the wildcards.

[FlorianWilhelm]

The module glob cannot really be used in this case since it does not work recursively. For instance tests/*.csv would only include tests/data.csv but not tests/special_tests/large_data.csv. For this to work we would need a combination of os.walk and fnmatch like described here.

[FlorianWilhelm]

MANIFEST.in allows recursive inclusion with:
recursive-include DIRECTORY PATTERN1 PATTERN2
Maybe we mimic this syntax like this in setup.cfg:

[data_files]
share/pyscaffold = .rst, *.txt, tests:.py, tests:*.csv
This would mean that all rst and txt files from the root of the project are included as well as all py and csv files in the tests folder and its subfolders.

I am not 100% sure about this syntax though since it seems a bit confusing. I don't want to use a white space to separate the directory from the pattern since the directory itself could have white spaces.

[FlorianWilhelm]

Another approach would be that we differentiate between greedy and non-greedy wildcards like * vs *?. By default we assume greedy and even match recursively since this seems to be what most users expect and want actually. For non-greedy behavior *? could be used like in the Python regular expressions.

Anyone wants to share his/her thoughts on that issue?

[xhochy]

As an alternative, we could use the more shell-like approach (the one that also git uses) and use ** for recursive includes.

[FlorianWilhelm]

@xhochy Seems like we had about the same idea :-) Do you think recursive includes or non-recursive includes should be default?

[xhochy]

@FlorianWilhelm for conistency, I would handle it exactly like git/bash/zsh do.

• *.py: all files ending in .py in the current directory
• **/*.py: all files ending in .py in all subfolders (recursively) of the current directory

[FlorianWilhelm]

@xhochy I like your suggestion. How would we do the implementation? It is somewhat similiar to regular expressions in the sense that * is *? in re and ** is * in re. Could we maybe use the functionality of the re module somehow? I'd like to avoid having even more code in setup.py.

[FlorianWilhelm]

@xhochy I'll try doing a replacement of * into *? followed by a replacement of *?*? into *. After that I should be able to just use a regular expression. os.walk can be used to get a recursive list of each file in a repository which can be then matched with the regular expressions. At that point one should be aware of links causing cycles.