using setuptools_scm forces all SCM files to be packaged into sdist #190
Comments
|
right now not, its suggested to use a MANIFEST.in to add excludes, documentation is needed |
|
@RonnyPfannschmidt I've tried but fails, in what form should that be added? |
|
this also makes that we ignore |
|
@gaborbernat not sure what you mean by that |
|
once setuptools_scm is enabled we package everything, by adding all files to the default (via https://github.com/pypa/setuptools/blob/5ecd7575c9c09d4ec2d8f993c5fb405388c3f3c1/setuptools/command/egg_info.py#L563); so this basically has the effect that we ignore whatever was specified in find_packages, package_data, not? |
|
running even though running https://github.com/borgbackup/borg/blob/master/setup.py#L817 states please no pyx, h, c files: see how we just packaged pyx files there. |
|
gah, does it also add them to a wheel? or does it just pollute the sdist? |
|
just the sdist 👍 |
gaborbernat
changed the title
|
Is there any update on this? This functionality is really not stated anywhere and is very undesirable for me. My manifest.in already lists everything that should be in the package and I would really love to use |
|
For what it's worth, I hacked around this by doing: try:
import setuptools_scm.integration
setuptools_scm.integration.find_files = lambda _: []
except ImportError:
passin my |
|
If it weren't for the bootstrapping challenges, I'd suggest that setuptools_scm should be split into two packages such that a project could selectively include the behavior they desire (and only that behavior). Unfortunately, the ecosystem (setuptools) assumes that if you've checked files into your source code repo that you intend to copy them into your sdist. I think the issue here shouldn't be addressed by setuptools_scm, but should be addressed by setuptools (as the issue would apply to any plug-in supplying file finder capability). |
|
Would it not be possible to create a dummy package, say try:
import setuptools_scm_no_finder
def find_files(_):
return []
except ImportError:
def find_files(...):
# current implementation
...Then you could list that module as an extra_require to this module, and one would do, say Edit: after further thought, scratch that, I just realised doing this would inadvertedly affect setup for all packages; oh well... |
|
Another idea: choose a special filename which, if encountered, would cause find_files to return an empty list? something like Or try to read configuration settings from some standard files like |
|
I looked into this in detail, being annoyed with it and I think it's fine the way it works now.
Tldr: files added by this find_files only will not be installed on the users machine. It's only the build start out material. Wheels distribution packages e.g. do not contain these as proof of this. |
|
im happy to accept a pr that adds a when setup is called, the file finding isnt diretly triggered, but we can at that point apply the evil monkeypatch in a reasonable manner @jaraco oppinions? |
|
Gaborbernat has already indicated that the existing behavior is acceptable... so the remaining opinion is that of axnsan12:
My recommendation is to eliminate the manifest.in and rely on the file finder. I've yet to see a compelling use-case where SCM-based file finders isn't a suitable approach if not exactly what a project needs. What is the reason that doesn't work for you, @axnsan12? My opinion is that if it's necessary to support disabling file finders, that should be either implemented in Setuptools or it should be a hack added by the individual projects. Adding it at setuptools_scm feels like the wrong place as it leaves other file finder plugins without that support. |
|
@jaraco thanks for elaborating, i will close this one as wontfix then |
|
@jaraco I have a use case where we have blobs in the repo (historically, don't ask) and I'd like to rip them off. It seems to be impossible to do via So I'm forced to resort to P.S. Another similar integration claims that they support |
Using ``MANIFEST.in`` because of setuptools limitation. Ref: pypa/setuptools-scm#190 (comment)
Using ``MANIFEST.in`` because of setuptools limitation. Ref: pypa/setuptools-scm#190 (comment)
Using ``MANIFEST.in`` because of setuptools limitation. Ref: pypa/setuptools-scm#190 (comment)
|
@jaraco, @gaborbernat Could you elaborate on your thinking why this is desirable behaviour? In the current project (where by the way it took me close to an hour to figure out that Also that this leads to disregard of the developer intention expressed with /edit: Also the argument that "it doesn't matter since only things listed in MANIFEST.in will get installed" doesn't hold water IMO since size of the sdist package size is also a concern. For the project I'm talking about above the size of the archive increases by almost 5x (~500 kB vs. ~2.5 MB) between including only required package files vs. all vcs files. |
Setuptools_scm forcibly includes all files under version control into the sdist package, ignoring `MANIFEST.in` and `find_packages`. This fixes this by replacing the `find_files` function with a dummy one (`see setup.py::EggInfo.__init__()`). See pypa/setuptools-scm#190
|
Regardless of all the buck passing about how its "not our fault" you have to realize how disruptive of a behavior this is for this package to implement. Its essentially a packaging virus that unwitting consumers have to deal with with preemptive hacks in our build scripts! Its great that you have a workflow that you like (including everything in the source control) but A) not everyone can do this even if they wanted to and B) I never actually opted in to use this project. I don't care that a project I am consuming uses it, but that shouldn't force me to account for it. Please deal with this mess because until setuptools gets fixed it is this project's problem. |
|
Im happy to merge a reasonable contribution, i don't have the bandwidth to do this myself |
Source distributions such as sdist must include test directories. Note that all files matching the pattern `test/test*.py` (or perhaps `tests/test*.py` with an `s`?) are included implicitly: https://packaging.python.org/guides/using-manifest-in/#how-files-are-included-in-an-sdist Binary distributions such as wheel must exclude test directories. This seems to be done implicitly for top-level `tests/` directories only. I do not know how to explicitly exclude files from binary distributions. This setuptools issue might be of interest: pypa/setuptools-scm#190 I change `codespell_lib/tests/` to a top-level `tests/` directory as a workaround. Not that I like it, but I currently lack an alternative. Fixes (partially) codespell-project#2592.
Using `setuptools_scm` allows us to reliably ensure that all files in git are in the sdist, which avoids problems caused by occasionally forgetting to update `MANIFEST.in` when adding a new directory. `MANIFEST.in` still needs to be kept (in a reduced form) because there are a few files needed by webpack and by kolibri-installer-android which are only created at dist time, and exist outside setuptools. References: - https://github.com/pypa/setuptools_scm#readme - https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/ - pypa/setuptools-scm#190 - https://packaging.python.org/en/latest/guides/using-manifest-in/ Signed-off-by: Philip Withnall <pwithnall@endlessos.org> #647
Using `setuptools_scm` allows us to reliably ensure that all files in git are in the sdist, which avoids problems caused by occasionally forgetting to update `MANIFEST.in` when adding a new directory. `MANIFEST.in` still needs to be kept (in a reduced form) because there are a few files needed by webpack and by kolibri-installer-android which are only created at dist time, and exist outside setuptools. References: - https://github.com/pypa/setuptools_scm#readme - https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/ - pypa/setuptools-scm#190 - https://packaging.python.org/en/latest/guides/using-manifest-in/ Signed-off-by: Philip Withnall <pwithnall@endlessos.org> #647
Using `setuptools_scm` allows us to reliably ensure that all files in git are in the sdist, which avoids problems caused by occasionally forgetting to update `MANIFEST.in` when adding a new directory. `MANIFEST.in` still needs to be kept (in a reduced form) because there are a few files needed by webpack and by kolibri-installer-android which are only created at dist time, and exist outside setuptools. References: - https://github.com/pypa/setuptools_scm#readme - https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/ - pypa/setuptools-scm#190 - https://packaging.python.org/en/latest/guides/using-manifest-in/ Signed-off-by: Philip Withnall <pwithnall@endlessos.org> #647
Using `setuptools_scm` allows us to reliably ensure that all files in git are in the sdist, which avoids problems caused by occasionally forgetting to update `MANIFEST.in` when adding a new directory. `MANIFEST.in` still needs to be kept (in a reduced form) because there are a few files needed by webpack and by kolibri-installer-android which are only created at dist time, and exist outside setuptools. References: - https://github.com/pypa/setuptools_scm#readme - https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/ - pypa/setuptools-scm#190 - https://packaging.python.org/en/latest/guides/using-manifest-in/ Signed-off-by: Philip Withnall <pwithnall@endlessos.org> #647
Using `setuptools_scm` allows us to reliably ensure that all files in git are in the sdist, which avoids problems caused by occasionally forgetting to update `MANIFEST.in` when adding a new directory. `MANIFEST.in` still needs to be kept (in a reduced form) because there are a few files needed by webpack and by kolibri-installer-android which are only created at dist time, and exist outside setuptools. References: - https://github.com/pypa/setuptools_scm#readme - https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/ - pypa/setuptools-scm#190 - https://packaging.python.org/en/latest/guides/using-manifest-in/ Signed-off-by: Philip Withnall <pwithnall@endlessos.org> #647
Using `setuptools_scm` allows us to reliably ensure that all files in git are in the sdist, which avoids problems caused by occasionally forgetting to update `MANIFEST.in` when adding a new directory. `MANIFEST.in` still needs to be kept (in a reduced form) because there are a few files needed by webpack and by kolibri-installer-android which are only created at dist time, and exist outside setuptools. References: - https://github.com/pypa/setuptools_scm#readme - https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/ - pypa/setuptools-scm#190 - https://packaging.python.org/en/latest/guides/using-manifest-in/ Signed-off-by: Philip Withnall <pwithnall@endlessos.org> #647
|
Any updates on how to fix this? |
In the Setuptools project, investigate and understand how the file finders functionality works. Develop an understanding of the nuances and differences between files discovered for source distributions and files used for installation. Then come up with a design for Setuptools that allows users that are using a file finder plugin to exclude found files from their sdist. |
- Stop including the built JS file in the package, instead it is built during sdist time (and hence included in the wheel). Stolen / inspired from JupyterHub itself. - setuptool_scm includes *all* checked-in files by default in the sdist (pypa/setuptools-scm#190). This cleans it out to trim down the size of our sdist - I think our previous wheel files didn't actually have the python package correctly. It is set up correctly now. - Generated js files are put under a dist/, so we can have non-generated static files under static/ in the future. dist/ is added to .gitignore
The pyproject.toml file is now the single location for the version to be declared. It will be read by __init__.py automatically. The old system used setuptools-scm but apparently that has the unintended side-effect of adding all files tracked by version control to the sdist! yikes Refer to pypa/setuptools-scm#190 for more details
any way we can disable this?
The text was updated successfully, but these errors were encountered: