Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setup and Test Requirements considered harmful #71

HolgerPeters opened this issue Sep 15, 2015 · 6 comments

Setup and Test Requirements considered harmful #71

HolgerPeters opened this issue Sep 15, 2015 · 6 comments
enhancement New feature or request


Copy link

Currently the that pyscaffold generates contains these lines:

    setup(setup_requires=['six', 'pyscaffold>=2.4rc1,<2.5a0'] + sphinx,
          tests_require=['pytest_cov', 'pytest'],

usage of setup_requires and tests_require is unfortunately a little problematic. Setuptools/distutils will install the packages listed there into the .eggs directory upon calling python develop and python test. Since the introduction of virtualenvs in the python workflows, one is probably more interested to use one virtualenv for developing and building the tool, and another one for running the tool that does not include the dev requirements.

Copy link

+1 the whole approach seems to be screwed at the moment and we need to find a way to fix it...

Copy link
Contributor Author

To be fair, setup_requires and tests_require are the way that setuptools/distutils are layed out.

Copy link

Hm... but setup_requires is actually needed for the whole concept of tools like PyScaffold, setuptools_scm and pbr. Can you elaborate some more where the problem is? Is it directly connected to #70 ?

Copy link
Contributor Author

Let's separate two things, for one, what we would like to have, and then what setuptools does and how this differs from what we would like to have in an environment where we use virtualenvs:

What we want: we would like to have a way to specify dependencies that we only need for executing a and not during runtime.

What setuptools offers: Setup requirements can be specified and are installed in a .eggs directory in the project. They appear as if they were installed and interfere with the state of the virtualenv (or the system python packages if no virtual env is used). In my experience, having packages in .eggs directory is a typical cause for trouble.

Nowadays, we have virtualenvs, so what we want is, to install all packages we need during development into a development virtualenv. We build a wheel for packaging our package up and install it in another, fresh virtualenv that does not pull any setup requirements for running the software. Virtualenvs enable us to not use the .eggs directory.

My workflow involves having a dev_requirements.txt file in the repo that I just install into the environment I am developing in. This means that I have to execute one pip install -r dev_requirements.txt before I can execute the, but ensures that .eggs will not trouble me.


What kind of trouble is it when packages are installed in a .eggs directory?

% putup asdf
% cd asdf
% pip install -e .
% python test
running test
Searching for pytest
Best match: pytest 2.7.3
Installed /............/asdf/.eggs/pytest-2.7.3-py2.7.egg
% py.test
pyenv: py.test: command not found

So I will go on and install pytest in my virtualenv. No I have two pytest installations.

Copy link

@HolgerPeters, good explanation. So when everything is already in your virtual environment, it should be no problem. It only causes trouble if setuptools installs software into the egg folder that was not available before. The question is if setuptools allows changing from easy_install to pip. Another option would be to raise an error for at least all packages in setup_requires instead of installing them with the help of setuptools.
I am looking forward to your thoughts on that @HolgerPeters and @sebastianneubauer.

@FlorianWilhelm FlorianWilhelm added wontfix This will not be worked on enhancement New feature or request and removed wontfix This will not be worked on labels Sep 16, 2015
Copy link

After our discussion, several things will be done to avoid this issue. First of all #72 will limit those possible conflicts to PyScaffold itself. Then avoiding test_requires and using an test-requirements.txt file like in #70 should at least limit the number of eggs you'll have if you set up your venv beforehand.

@FlorianWilhelm FlorianWilhelm added this to the 2.5 milestone Sep 16, 2015
@FlorianWilhelm FlorianWilhelm self-assigned this Sep 30, 2015
FlorianWilhelm added a commit that referenced this issue Jan 1, 2016

a035b38 make readme pypi compatible again
7b72db5 prepare 1.10.0 release
cb38b67 whitespace fixes
4c5789a introduce parse_scm_fallback concept and fix parse hooks
d80fde9 coding and documentation style cleanups
c6f4ee6 introduce scm parser overriding fixes #70 and #71
af0095b support enforcing version numbers via the env
b4baef9 Merge pull request #68 from pypa/issue-67
f0d86ef add missing space
ff897ce first attempt at fixing issue #67
2d3b653 minor flake8 cleanup
04d8a87 address #63 - change the tag matching pattern
25b0529 fix issue #63 by passing a match parameter to git describe
9672e57 finish restructuring tests
32f8bca simplify callable_or_entrypoint(group, callable_or_name)
cab8949 remove forgotten tests
b977fe0 extract git tests
937a308 finish extracting mercurial tests into own file
05e4600 simplify data from mime test
5f9dd40 move hg archival test
1227628 Now that /tmp is no longer hard-coded, supply a more interesting root.
5294e37 Extend the extraction of the root assertion so that it's re-used by all test_root functions
20c57ae It seems the cache parameter is unused and needs not be supplied
4283d06 Transfer documentation from docstring to README. Fixes #64.
0ac8221 Added tag v1.9.0 for changeset 1092123ef785
a2fbff3 Add alias for simple distutils-based release.
d3923f6 Add relative_to parameter to get_version. Fixes #44 and fixes #45.
b6320b0 Extract assert_root_tmp fixture
52ecdba Merge pull request #58 from mcmtroffaes/feature/fix-main
6792706 Wrap test_main into a function so travis will pick it up.
644edd2 Fix to import from the correct places.
27f0b64 Test 'python -m setuptools_scm' invocation (
81a0e74 Merge pull request #57 from felixonmars/master
9542ba9 Fix small typos in README

git-subtree-dir: pyscaffold/contrib/setuptools_scm
git-subtree-split: a035b38bad6a3505fd2c24306080dc9a6e1dffe3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
enhancement New feature or request
None yet

No branches or pull requests

3 participants