Fix bug that prevents packages.find.exclude to take effect when include_package_data=True

[abravalheri]

Summary of changes

When discovering files in build_py, avoid including files and directories that correspond to a sub-package.

This way excluded sub-packages are automatically ignored.

Closes #3260.

Pull Request Checklist

• Changes have tests
• News fragment added in changelog.d/.
  (See documentation for details)

[abravalheri]

Side note

This f == oldf condition is a bit odd.
According to the git history (f01c17e), the original intention of this code was to:

Allow .py files found by the include_package_data option to be automatically included.

To me that sounds a little bit as a misuse. My opinion is that modules should not be affected by include_package_data...

Unfortunately, I cannot find the original repository (I guess it was bitbucket) to check if it was associated with a PR or feature request to understand the context.

[jaraco]

All of the issues and PRs were migrated from Bitbucket, but that commit comes from when Setuptools was hosted on Sourceforge and Subversion, where issues and merges weren't tracked as easily, so the commit history is the best we have.

[abravalheri]

Here I thought about checking against importlib.machinery.all_suffixes, but I decided to go only with .py because I don't know what to do with .so files...

(Maybe the .so is not generated by setuptools, but instead compiled out-of-band and included by the user on purpose?)

[abravalheri]

Hi @jaraco, would it be possible for you to have a look in this PR?

There is some very old implementation involved in the faulty behaviour so I am not exactly sure if the provided patch is the best way to go (but is the only way I can imagine 😅)

[jaraco]

I'll plan to look today.

[jaraco]

Today is not going as planned. Maybe tomorrow.

[abravalheri]

No problems Jason, whenever you have the time. It is not something urgent, I was just not confident enough with the change to merge it directly. Thank you very much.

[jaraco]

Thanks for this proposal. Your patch made it easy to see the intention here.

In the referenced issue, I wrote a response about the problem generally, where I expressed my reservations.

In spite of those reservations, I'd be open to merging this change and rolling it out to see what impact it has. Even if not optimal, perhaps this proposed behavior is preferable to the prior behavior.

Either way, I'd like us at some point to explore a redesign of the mechanisms for determining the 'build' manifest.

[akx]

Tiny typo :)

[abravalheri]

Thank you very much @jaraco.

There is an alternative implementation: we can consider all nested folders as nested packages, and avoid traversing them in the first place. In terms of implementation it would mean removing this loop:

setuptools/setuptools/command/build_py.py

Lines 137 to 140 in 372652d

	while d and d != prev and d not in src_dirs:
	prev = d
	d, df = os.path.split(d)
	f = os.path.join(df, f)

I think this behaviour would is more intuitive and reflects better what I expect from include_package_data. But I think it is also "more disruptive".

Either way, I'd like us at some point to explore a redesign of the mechanisms for determining the 'build' manifest.

I am more than open to work on that. Do you have something in mind?

[jaraco]

I don’t have anything specific in mind at this time. I’ll defer to your judgment. I’m okay with disruptive as long as we manage the disruption.

[abravalheri]

Thank you very much @jaraco.

I changed the PEP 660 implementation proposal to not depend on this PR.

I think the right thing going forward is to treat any folder that is importable to Python as a package and therefore subject to the expected effects. packages.find.exclude (at least until we come up with a different mechanism as discussed in #3260). However in order to manage the disruption we might want to issue a deprecation warning for users relying on this behavior.

I plan to implement the following strategy in the following days:

1. Create another PR that will issue a warning when a project relies on include_package_data to include any folder or Python script that is importable by Python
2. Change this PR accordingly
3. Postpone this PR for a while so people can react to the deprecation warning, or until we create a better mechanism (currently being discussed in [BUG] options.packages.find.exclude not taking effect when include_package_data = True #3260).

[abravalheri]

1. Create another PR that will issue a warning when a project relies on include_package_data to include any folder or Python script that is importable by Python

I have created #3308 to start with a deprecation warning. After some time (during which the users have the chance to do the necessary changes), we can revisit the topic and fix the bug.

@jaraco what is the release policy for setuptools in terms of deprecations? Is there a target in terms of recommended time interval (or number of major versions) between deprecating a behaviour/mechanism and its removal from setuptools?