setuptools_scm breaks the behavior of setuptools.find_packages

[vorj]

related: tianzhi0549/FCOS#368

It seems that setuptools_scm breaks the behavior of setuptools.find_packages .

$ python --version
Python 3.8.10
$ pip3 install torch==1.6.0 numpy==1.21.4
$ git clone https://github.com/cocodataset/cocoapi
$ pushd cocoapi/PythonAPI
$ python3 setup.py build_ext install  # cocoapi depends on matplotlib, and matplotlib >= 3.5 depends on setuptools-scm
$ popd
$ pip3 freeze | grep setuptools-scm
setuptools-scm==6.3.2
$ git clone https://github.com/tianzhi0549/FCOS
$ sed 's@cpu/vision.h@vision.h@g' -i ./FCOS/fcos_core/csrc/cpu/{nms,ROIAlign}_cpu.cpp
$ pip3 install ./FCOS  # This fails like below:
...
error: can't copy 'fcos/configs': doesn't exist or not a regular file
$ # NOTE: FCOS/fcos/configs is a symbolic link for directory
$ pip3 uninstall -y setuptools-scm
$ pip3 install ./FCOS  # This succeeds

Then, is this behavior expected? If not, please investigate the cause and fix it.

Thanks

[RonnyPfannschmidt]

this is a expected and broken interaction due to how setuptools handles file finders
unfortunately unless setuptools is fixed in that regard, setuptools_scm cannot provide the file finding without breaking extra packages

the suggested workaround currently is to uninstall setuptools_scm for the effected packages
i hope we can eventually provide editable installs without needing to install setuptools_scm in the target (the new peps)

[vorj]

OK, I have understood that this is not a kind of bug on setuptools_scm , and aside from that I have also done that some packages shouldn't be used with setuptools_scm . Thank you.

[RonnyPfannschmidt]

@vorj the key issue here is, that its not directly possible to properly opt in or out of the features in setuptools

@jaraco i wonder if there is any path towards having setuptools help with more correct controlled plugin opt-in/out

my proposal would be that once setuptools gains toml support, it could limit the active plugins to the ones listed in the build requirements (however only if a [tool.setuptools] section exists in the toml file

[jaraco]

this is a expected and broken interaction due to how setuptools handles file finders

I don't understand the issue. I'd appreciate if someone could distill what the issue is with Setuptools and report it to the project (or link to a known issue). Try to put together a minimal reproducer or at least characterize the nature of the issue (why is fcos/configs implicated in file finding but only when setuptools_scm is involved?).

@jaraco i wonder if there is any path towards having setuptools help with more correct controlled plugin opt-in/out

my proposal would be that once setuptools gains toml support, it could limit the active plugins to the ones listed in the build requirements (however only if a [tool.setuptools] section exists in the toml file

The PEP 517 standard is meant to address this issue. Each project can declare its build dependencies by opting in to the build dependencies as specified. If a project is not using a builder with build isolation or if they're resolving the build dependencies on their own, then the issue is that they're supplying functionality that the project did not request. And due to the nature of inversion-of-control, the presence of the plugin implies the activation of the plugin.

I'm disinclined to rely on the build requirements to determine which plugins to activate, because it removes the ability for an integrator to customize the build environment, and puts the onus on Setuptools to enforce isolation. I believe Setuptools wants to stay out of the business of managing the build environment and instead wants to rely on whatever build environment was created, relying on the build tools and integrators to do the right thing.

I do think maybe there's a case for the ability to opt in or out of features. I did at one point suggest that the plugin itself could enable opt-out of the file finder through something like:

[tool.setuptools_scm]
find_files = false

However, I don't believe that addresses the reported use-case.

I could imagine a situation where Setuptools could solicit a configuration parameter that says something like "disable-plugins=file-finders", but such a parameter would probably only be supported through PEP 517 build mechanisms.

---

Oh, I see now the failure is happening with pip3 install. Out of curiousity, have you tried pip3 install --use-pep517? That will enable the isolated build logic instead of the fallback compatibility logic. The FCOS project can opt in to the same behavior by adding a pyproject.toml with a [build-system] section with something like:

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

There's a good chance that either of those techniques will address the issue.

[RonnyPfannschmidt]

@jaraco bascialyl the pep wont help if a global plugin is loaded in a global env

if anyone installs setuptools_scm in their env, then in that env setuptools will be "miss-configured" for all packags that do not want file finders

in essence file finders break the env if they are unwanted for the package
if a integrator adds extra deps, they should be expected to actually list them somewhere to prevent fragile plugin behaviour changes

[salotz-sitx]

Just want to point out that if any of your build dependencies depend on setuptools_scm then you are along for the ride as well!!

Beyond setuptools being broken I think it is totally broken to have a transitive dependency set configuration for your project that you have to opt out of pre-emptively.

The library in question I'm using is cmake-build-extension which is only using setuptools_scm as a library.

Regardless of whether setuptools is broken or not setuptools_scm should be split into two different libraries. One that is a -- library -- and the other that is a plugin. Otherwise you've unleashed a packaging virus.

[RonnyPfannschmidt]

There is a plan to split the versioning parts into a new library

Based on my bandwidth sometime within the next 12 months

Ps I'll stop to engage if the hyperbole keeps coming up