[BUG] Setup function copies the entire directory into the dist

[freiburgermsu]

setuptools version

setuptools == 61.3.1

Python version

Python 3.8.10

OS

Windows 10

Additional environment information

No response

Description

This setup.py file

# -*- coding: utf-8 -*-
from setuptools import setup, find_packages

with open('README.rst', 'r', encoding='utf-8') as file:
    readme = file.read()

setup(
  name = 'PDIpy',      
  package_dir = {'pdi':'pdipy'},
  packages = find_packages(),
  package_data = {
          'pdipy': ['parameters/*'],
          'test': ['*']
          }, 
  version = '0.0.5',
  license = 'MIT',
  description = "Simulate Photodynamic Inactivation (PDI) from a kinetics model of membrane oxidation.", 
  long_description = readme,
  author = 'Andrew Freiburger',               
  author_email = 'andrewfreiburger@gmail.com',
  url = 'https://github.com/freiburgermsu/PDIpy',   
  keywords = [
          'antibacterial',
          'photodynamic', 
          'biophysics',
          'computational',
          'biology',
          'medicine', 
          'PDI', 
          'antibiotics'
          ],
  install_requires = [
          'matplotlib',
          'tellurium', 
          'scipy', 
          'pandas',
          'sigfig',
          'hillfit',
          'chemw',
          'numpy'
          ]
)

copies this entire directory into the dist folder, perplexingly including the examples folder that is neither mentioned in the setup.py file nor contains any script that could be perceived to be a package by find_packages(). This error is strangely also occurring with old setup.py files that previously worked perfectly. Changing or removing most of the arguments and values has been ineffectual.

What in the setup file is erroneous, and how can it be resolved?

Expected behavior

A dist folder should be created that exclusively contains content from the test and pdipy folders, which are specified in the setup.py file.

How to Reproduce

python setup.py sdist in Powershell.

Output

PS C:\Users\Andrew Freiburger\Dropbox\My PC (DESKTOP-M302P50)\Documents\UVic Civil Engineering\PDIpy\PDIpy> python .\setup.py sdist
running sdist
running egg_info
writing PDIpy.egg-info\PKG-INFO
writing dependency_links to PDIpy.egg-info\dependency_links.txt
writing requirements to PDIpy.egg-info\requires.txt
writing top-level names to PDIpy.egg-info\top_level.txt
writing manifest file 'PDIpy.egg-info\SOURCES.txt'
running check
creating PDIpy-0.0.5
creating PDIpy-0.0.5\.github
creating PDIpy-0.0.5\.github\workflows
creating PDIpy-0.0.5\PDIpy.egg-info
creating PDIpy-0.0.5\docs
creating PDIpy-0.0.5\docs\source
creating PDIpy-0.0.5\examples
creating PDIpy-0.0.5\examples\Beirao_et_al
creating PDIpy-0.0.5\examples\Beirao_et_al\Beirao_et_al, 10 uM solution
creating PDIpy-0.0.5\examples\Beirao_et_al\Beirao_et_al, 20 uM solution
creating PDIpy-0.0.5\examples\Beirao_et_al\Beirao_et_al, 5 uM solution
creating PDIpy-0.0.5\examples\Beirao_et_al\Beirao_et_al, biofilm, 10 uM solution
creating PDIpy-0.0.5\examples\Beirao_et_al\Beirao_et_al, biofilm, 20 uM solution
creating PDIpy-0.0.5\examples\Beirao_et_al\Beirao_et_al, biofilm, 5 uM solution
creating PDIpy-0.0.5\examples\Bozja_et_al
creating PDIpy-0.0.5\examples\GSWL
creating PDIpy-0.0.5\examples\GSWL\.ipynb_checkpoints

... The console lines were too great for GitHub to post in this issue ...

copying supplementary\.ipynb_checkpoints\photosensitizer_volume_proportion-checkpoint.ipynb -> PDIpy-0.0.5\supplementary\.ipynb_checkpoints
copying test\__init__.py -> PDIpy-0.0.5\test
copying test\test_pdi.py -> PDIpy-0.0.5\test
Writing PDIpy-0.0.5\setup.cfg
Creating tar archive
removing 'PDIpy-0.0.5' (and everything under it)

[abravalheri]

Hi @freiburgermsu, thank you for reporting the problem.

old setup.py files that previously worked perfectly.

Could you specify which version of setuptools you used to have installed in your machine when the setup.py was working perfectly?

copies this entire directory into the dist folder,

I cannot understand what you are reporting here. When I try the following steps, I cannot observe anything other than the tar.gz inside the dist folder. Moreover, inside the tar.gz file, the examples folder is not present.

cd $env:TEMP
git clone https://github.com/freiburgermsu/PDIpy.git
cd PDIpy
python -m virtualenv .venv
.\.venv\Scripts\python.exe -m pip install -U pip setuptools
.\.venv\Scripts\python.exe -m pip list
# Package    Version
# ---------- -------
# pip        22.0.4
# setuptools 61.3.1
# wheel      0.37.1
.\.venv\Scripts\python.exe setup.py sdist
dir dist
# PDIpy-0.0.5.tar.gz
tar tf .\dist\PDIpy-0.0.5.tar.gz
# PDIpy-0.0.5/
# PDIpy-0.0.5/PDIpy.egg-info/
# PDIpy-0.0.5/PDIpy.egg-info/PKG-INFO
# PDIpy-0.0.5/PDIpy.egg-info/SOURCES.txt
# PDIpy-0.0.5/PDIpy.egg-info/dependency_links.txt
# PDIpy-0.0.5/PDIpy.egg-info/requires.txt
# PDIpy-0.0.5/PDIpy.egg-info/top_level.txt
# PDIpy-0.0.5/PKG-INFO
# PDIpy-0.0.5/README.rst
# PDIpy-0.0.5/pdipy/
# PDIpy-0.0.5/pdipy/__init__.py
# PDIpy-0.0.5/pdipy/parameters/
# PDIpy-0.0.5/pdipy/parameters/light.json
# PDIpy-0.0.5/pdipy/parameters/photosensitizers.json
# PDIpy-0.0.5/pdipy/parameters/wells.json
# PDIpy-0.0.5/pdipy/pdi.py
# PDIpy-0.0.5/setup.cfg
# PDIpy-0.0.5/setup.py
# PDIpy-0.0.5/test/
# PDIpy-0.0.5/test/__init__.py
# PDIpy-0.0.5/test/test_pdi.py

Do you have a reproduction for the problem you are facing?

[freiburgermsu]

Hello @abravalheri ,

1. I believe that the previous version of setuptools that worked a month ago was version 56.0.0. I upgraded the version to 61.3.1 when I experienced the error, and then tried 60.0.0, yet none of these versions resolved the issue.
2. I reproduce the problem each time I run python setup.py sdist in my local version of the repository. The generated tar.gz file (PDIpy-0.0.5.tar.gz), which contains the entire examples directory. The console outputs are contained within this TXT file (setuptools_error.txt). I meet via zoom to articulate and demonstrate these problems.

[abravalheri]

I have a theory... Any chance you have setuptools-scm installed in your machine?

When setuptools-scm is installed, everything that is committed to git is added to the sdist. That is how this setuptools plugin works.

Could you try using a virtual environment as I suggested in #3246 (comment) to see if the problem persists?

Alternatively you can also try the following:

python -m pip install build
# Instead of `python setup.py sdist` use:
python -m build --sdist

[freiburgermsu]

Hello @abravalheri ,

I followed the sdist commands with build, however, the examples directory was still copied into the zip file.

I indeed seem to have setuptools-scm:

PS C:\Users\Andrew Freiburger\Dropbox\My PC (DESKTOP-M302P50)\Documents\UVic Civil Engineering\PDIpy\PDIpy> pip show setuptools-scm
Name: setuptools-scm
Version: 6.3.2
Summary: the blessed package to manage your versions by scm tags
Home-page: https://github.com/pypa/setuptools_scm/
Author: Ronny Pfannschmidt
Author-email: opensource@ronnypfannschmidt.de
License: MIT
Location: c:\users\andrew freiburger\appdata\roaming\python\python38\site-packages
Requires: packaging, setuptools, tomli
Required-by: matplotlib
PS C:\Users\Andrew Freiburger\Dropbox\My PC (DESKTOP-M302P50)\Documents\UVic Civil Engineering\PDIpy\PDIpy>

Uninstalling setuptools-scm via pip did not prevent the examples from being included in the zip. Must the program be uninstalled elsewhere as well?

[abravalheri]

I followed the sdist commands with build, however, the examples directory was still copied into the zip file.

Thank you very much for checking that. There is a chance some files were cached, if you are running python -m build --sdist in the same folder that you have previously run python setup.py sdist with setuptools-scm. You can have some luck if you remove the cache (build, dist and PDIpy.egg-info folders).

@freiburgermsu is there any chance you can provide a minimal isolated reproducer for this issue so I can investigate the problem? (You can find information about minimal reproducers in: ref1 and ref2).

For example, based on #3246 (comment) and #3246 (comment), I would expect one of the following snippets to be such a reproducer:

cd $env:TEMP
rm -R -force .\PDIpy\
git clone https://github.com/freiburgermsu/PDIpy.git
cd PDIpy
python -m virtualenv .venv
.\.venv\Scripts\python.exe -m pip install -U pip setuptools
.\.venv\Scripts\python.exe setup.py sdist
tar tf .\dist\PDIpy-0.0.5.tar.gz

OR

cd $env:TEMP
rm -R -force .\PDIpy\
git clone https://github.com/freiburgermsu/PDIpy.git
cd PDIpy
python -m virtualenv .venv
.\.venv\Scripts\python.exe -m pip install -U build
.\.venv\Scripts\python.exe -m build --sdist
tar tf .\dist\PDIpy-0.0.5.tar.gz

However, I cannot observe the behaviour you described.

Please note that if we cannot find a minimal isolated reproducer, chances are the problem is being caused by circumstances unique to your machine, and that would be outside of the scope of this bug tracker.

[drewcassidy]

Im having the same issue on macOS 12 with python 3.9.12. Every directory in the project is being copied into lib/ when I run pip install ., including the build directory and the virtual environment itself. This happens regardless of if setuptools-scm is enabled or not

Here's a log of installing yaclog:
log.txt

and the wheel from pip wheel .
yaclog-1.0.3-py3-none-any.zip

its worth noting that it also includes build and env which are explicitly ignored by git, so setuptools-scm shouldn't care about them anyways

Update: using build --sdist does not include the build or env directories, so this may be a problem with wheel or pip? im still unclear on how all of the python packaging tools fit together

[abravalheri]

Hi @drewcassidy, maybe that is being caused by cached files?

Currently the recommended way of building the package is via python -m build (if you don't use --sdist, it will create both sdist and wheel), so I am glad to see that it works.

Just for the sake of completeness, for the procedures you are using (other than python -m build), if you repeat them with a fresh project directory (e.g. a new clone) and a brand new virtual environment (as exemplified in #3246 (comment)), do you still face this issue?

[drewcassidy]

Clearing the build directory fixed the issue. that makes sense, I was messing with package discovery and I guess it cached it with a bunch of files I didn't want

I think there is still a bug here, since the cache should be cleared/updated instead of keeping old files around that shouldn't be there. Is the build directory handled by setuptools or something else?

[abravalheri]

Thank you for checking out @drewcassidy .

I understand that automatic invalidation of cache is a "nice to have" feature. If you would like to give a try and propose a PR, I would be more than happy to review it.

Meanwhile note that this feature is not very relevant when considering the workflow recomended in the Quickstart docs: python -m build will ensure proper isolation, and therefore caches will have no effect.

[drewcassidy]

I was using pip install . when I got this result, not just building wheels. I am a bit confused since I thought modern pip always used build isolation

[abravalheri]

pip's isolation less powerful than build's. For example, in this case it seems that the existence of build caches might influence the end result.

I understand that automatic invalidation of cache is a "nice to have" feature. If you would like to give a try and propose a PR, I would be more than happy to review it.

☝️ meanwhile users can try rm -rf build dist *.egg-info (or src/*.egg-info) during the troubleshooting.

[freiburgermsu]

Hello @abravalheri ,

Your suggestion of removing the dist and PDIpy.egg-info folders resolved the bug. It must have been caching other local content.

Thank you for your patient troubleshooting :)

[Xu-Justin]

point_up meanwhile users can try rm -rf build dist *.egg-info (or src/*.egg-info) during the troubleshooting.

Hey, thank you very much. Removing dist and .egg-info folder resolved the problems.