New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Egg-info location incorrect for modified package_dir in setup.py #464

Closed
selimb opened this Issue Feb 27, 2017 · 9 comments

Comments

4 participants
@selimb
Contributor

selimb commented Feb 27, 2017

My Setup

My package lies inside an src directory. This decision was inspired by Ionel's blog post.
The parts of my config relevant to this issue are:
In setup.py

setup(
  packages=find_packages('src'),
  package_dir={'': 'src'},

tox.ini:

[testenv]
usedevelop=True

Problem

I noticed that develop-inst-nodeps was being run on every call to tox, even though the docs mention:

There is an optimization coded in to not bother re-running the command if $projectname.egg-info is newer than setup.py or setup.cfg.

This is because it only looks for the egg-info in the same directory as setup.py. However, it seems to me setup.py develop outputs egg-info to the package_dir, in my case src. Here is the relevant part of the tox code (in venv.py):

setup_py = setupdir.join('setup.py')
egg_info = setupdir.join('.'.join((name, 'egg-info')))

Suggestions

Of course, it could be I'm just missing something and that this is not the recommended approach. Assuming this is indeed a bug, here are my suggestions.

The best way would be to find out what package_dir is and look for egg-info in that directory. After a bit of digging it doesn't seem it can simply be extracted from the python setup.py command (like --name for instance). Then, one could either parse the setup.py file or the output of python setup.py develop, both of which I think are quite messy.

The other, much much simpler way, is to allow a package_dir key in tox.ini. This results in duplicated information...but how often is that setting likely to be changed?

I'd be up for submitting a PR for the latter approach.

@obestwalter

This comment has been minimized.

Show comment
Hide comment
@obestwalter

obestwalter Feb 27, 2017

Member

edit: Sorry. Misread your post the first time around ... it's the other way around and you expect the same things as I would have, so there is definitely something strange going on. Will mark this as a bug for now.

Just to give you quick feedback ...

I did not read that blog post yet, but let me tell you that your posted setup settings look weird to me and I don't quite see the point. I will definitely look into that later, because I want to understand the reasoning behind that.

Of course, it could be I'm just missing something [...]

AFAIK the .egg-info is always located in the top level of the project - so next to where setup.py lies. If you expect it anywhere else you might get disappointed. This as well with the caveat of my current world view that has not been changed by that blog post yet.

Member

obestwalter commented Feb 27, 2017

edit: Sorry. Misread your post the first time around ... it's the other way around and you expect the same things as I would have, so there is definitely something strange going on. Will mark this as a bug for now.

Just to give you quick feedback ...

I did not read that blog post yet, but let me tell you that your posted setup settings look weird to me and I don't quite see the point. I will definitely look into that later, because I want to understand the reasoning behind that.

Of course, it could be I'm just missing something [...]

AFAIK the .egg-info is always located in the top level of the project - so next to where setup.py lies. If you expect it anywhere else you might get disappointed. This as well with the caveat of my current world view that has not been changed by that blog post yet.

@obestwalter obestwalter added the bug label Feb 27, 2017

@selimb

This comment has been minimized.

Show comment
Hide comment
@selimb

selimb Feb 27, 2017

Contributor

Edit: just read your edit lol. Will leave this here for future reference anyway.

Thanks for the quick reply!

It's not the most conventional setup settings perhaps, but setuptools' documentation itself has an example of using src as the root of the source tree (it's also mentioned in several other places on that page, just Ctrl+F src).

Regarding the location of .egg-info, I refer to you this link, again from setuptools' documentation. Specifically:

This should normally be the root of your project’s source tree (which is not necessarily the same as your project directory; some projects use a src or lib subdirectory as the source root).

After reading that, I gather it is possible to modify it if calling setup.py egg_info explicitly with the egg-base option. For some reason you can't do this with the setup.py develop --egg-path -- it's not what it's for.

Anyhow, I'd personally refrain from calling setup.py with the more advanced and obscure options XD.

Contributor

selimb commented Feb 27, 2017

Edit: just read your edit lol. Will leave this here for future reference anyway.

Thanks for the quick reply!

It's not the most conventional setup settings perhaps, but setuptools' documentation itself has an example of using src as the root of the source tree (it's also mentioned in several other places on that page, just Ctrl+F src).

Regarding the location of .egg-info, I refer to you this link, again from setuptools' documentation. Specifically:

This should normally be the root of your project’s source tree (which is not necessarily the same as your project directory; some projects use a src or lib subdirectory as the source root).

After reading that, I gather it is possible to modify it if calling setup.py egg_info explicitly with the egg-base option. For some reason you can't do this with the setup.py develop --egg-path -- it's not what it's for.

Anyhow, I'd personally refrain from calling setup.py with the more advanced and obscure options XD.

@RonnyPfannschmidt

This comment has been minimized.

Show comment
Hide comment
@RonnyPfannschmidt

RonnyPfannschmidt Feb 28, 2017

Contributor

@obestwalter egg-info for develop is put into sys.path, if sys.path is a package dir like src, it goes there

Contributor

RonnyPfannschmidt commented Feb 28, 2017

@obestwalter egg-info for develop is put into sys.path, if sys.path is a package dir like src, it goes there

@obestwalter

This comment has been minimized.

Show comment
Hide comment
@obestwalter

obestwalter Feb 28, 2017

Member

I don't understand. sys.path is a list of paths to search for modules. Do you mean it is put into the first entry in sys.path? I guess I have to read up about this a bit. My experience is that this always ends up next to setup.py.

Member

obestwalter commented Feb 28, 2017

I don't understand. sys.path is a list of paths to search for modules. Do you mean it is put into the first entry in sys.path? I guess I have to read up about this a bit. My experience is that this always ends up next to setup.py.

@RonnyPfannschmidt

This comment has been minimized.

Show comment
Hide comment
@RonnyPfannschmidt

RonnyPfannschmidt Feb 28, 2017

Contributor

@obestwalter when you use package_dirs it does not unless you do a sdist as far as i understood

Contributor

RonnyPfannschmidt commented Feb 28, 2017

@obestwalter when you use package_dirs it does not unless you do a sdist as far as i understood

@obestwalter

This comment has been minimized.

Show comment
Hide comment
@obestwalter

obestwalter Feb 28, 2017

Member

o.k. thanks for the clarification @RonnyPfannschmidt.

Member

obestwalter commented Feb 28, 2017

o.k. thanks for the clarification @RonnyPfannschmidt.

@selimb

This comment has been minimized.

Show comment
Hide comment
@selimb

selimb Feb 28, 2017

Contributor

To expand on the sys.path comment. Here is the output of pip install --editable (note I'm currently on Cygwin but I don't think that matter).

$ python2.7 -m virtualenv ~/.venvs/srctest && source ~/.venvs/srctest/bin/activate

(srctests) $ ls mypackage
src  setup.py

(srctests) $ cat mypackage/setup.py
from setuptools import find_packages, setup

setup(
    name='in_src',
    packages=find_packages('src'),
    package_dir={'': 'src'},
    install_requires=[
        'requests',
        'click==5.0'
    ]
)


(srctest) $ python -c "import sys; print '\n'.join(sys.path)"

/home/selimb/.venvs/srctest/lib/python27.zip
/home/selimb/.venvs/srctest/lib/python2.7
/home/selimb/.venvs/srctest/lib/python2.7/plat-cygwin
/home/selimb/.venvs/srctest/lib/python2.7/lib-tk
/home/selimb/.venvs/srctest/lib/python2.7/lib-old
/home/selimb/.venvs/srctest/lib/python2.7/lib-dynload
/usr/lib/python2.7
/usr/lib/python2.7/plat-cygwin
/home/selimb/.venvs/srctest/lib/python2.7/site-packages

(srctest) $ pip install --editable mypackage
Obtaining file:///cygdrive/d/scripts/tmp/mypackage
Collecting requests (from in-src==0.0.0)
Collecting click==5.0 (from in-src==0.0.0)
Installing collected packages: requests, click, in-src
  Running setup.py develop for in-src
Successfully installed click-5.0 in-src requests-2.13.0

(srctest) $ python -c "import sys; print '\n'.join(sys.path)"

/home/selimb/.venvs/srctest/lib/python27.zip
/home/selimb/.venvs/srctest/lib/python2.7
/home/selimb/.venvs/srctest/lib/python2.7/plat-cygwin
/home/selimb/.venvs/srctest/lib/python2.7/lib-tk
/home/selimb/.venvs/srctest/lib/python2.7/lib-old
/home/selimb/.venvs/srctest/lib/python2.7/lib-dynload
/usr/lib/python2.7
/usr/lib/python2.7/plat-cygwin
/home/selimb/.venvs/srctest/lib/python2.7/site-packages
/cygdrive/d/scripts/tmp/mypackage/src


(srctest) $ tree mypackage
mypackage
├── setup.py
└── src
    ├── in_src
    └── in_src.egg-info

Notice how a line was appended to sys.path (which AFAIK it does by creating an egg-link file in site-packages, amongst other things).

I purposefully didn't do pip install --editable . to show that the current working directory doesn't matter.

Contributor

selimb commented Feb 28, 2017

To expand on the sys.path comment. Here is the output of pip install --editable (note I'm currently on Cygwin but I don't think that matter).

$ python2.7 -m virtualenv ~/.venvs/srctest && source ~/.venvs/srctest/bin/activate

(srctests) $ ls mypackage
src  setup.py

(srctests) $ cat mypackage/setup.py
from setuptools import find_packages, setup

setup(
    name='in_src',
    packages=find_packages('src'),
    package_dir={'': 'src'},
    install_requires=[
        'requests',
        'click==5.0'
    ]
)


(srctest) $ python -c "import sys; print '\n'.join(sys.path)"

/home/selimb/.venvs/srctest/lib/python27.zip
/home/selimb/.venvs/srctest/lib/python2.7
/home/selimb/.venvs/srctest/lib/python2.7/plat-cygwin
/home/selimb/.venvs/srctest/lib/python2.7/lib-tk
/home/selimb/.venvs/srctest/lib/python2.7/lib-old
/home/selimb/.venvs/srctest/lib/python2.7/lib-dynload
/usr/lib/python2.7
/usr/lib/python2.7/plat-cygwin
/home/selimb/.venvs/srctest/lib/python2.7/site-packages

(srctest) $ pip install --editable mypackage
Obtaining file:///cygdrive/d/scripts/tmp/mypackage
Collecting requests (from in-src==0.0.0)
Collecting click==5.0 (from in-src==0.0.0)
Installing collected packages: requests, click, in-src
  Running setup.py develop for in-src
Successfully installed click-5.0 in-src requests-2.13.0

(srctest) $ python -c "import sys; print '\n'.join(sys.path)"

/home/selimb/.venvs/srctest/lib/python27.zip
/home/selimb/.venvs/srctest/lib/python2.7
/home/selimb/.venvs/srctest/lib/python2.7/plat-cygwin
/home/selimb/.venvs/srctest/lib/python2.7/lib-tk
/home/selimb/.venvs/srctest/lib/python2.7/lib-old
/home/selimb/.venvs/srctest/lib/python2.7/lib-dynload
/usr/lib/python2.7
/usr/lib/python2.7/plat-cygwin
/home/selimb/.venvs/srctest/lib/python2.7/site-packages
/cygdrive/d/scripts/tmp/mypackage/src


(srctest) $ tree mypackage
mypackage
├── setup.py
└── src
    ├── in_src
    └── in_src.egg-info

Notice how a line was appended to sys.path (which AFAIK it does by creating an egg-link file in site-packages, amongst other things).

I purposefully didn't do pip install --editable . to show that the current working directory doesn't matter.

@asottile

This comment has been minimized.

Show comment
Hide comment
@asottile

asottile Feb 28, 2017

Member

Note that I also ran into something similar to this while doing some (admittedly awful) setup.py introspection. My solution (while slightly janky) worked pretty well for me and may benefit the tox project:

            out, srcdir = _call_setup_py(srcdir, _find_setup_py(), 'egg_info')
            for line in out.decode('UTF-8').splitlines():
                if line.startswith('writing ') and line.endswith('/PKG-INFO'):
                    _, pkg_info = line.split(' ')
                    egg_info_dir, _, _ = pkg_info.rpartition('/')
            requires_txt = os.path.join(srcdir, egg_info_dir, 'requires.txt')

The implementation of _call_setup_py is essentially subprocess.check_output((setup_py, *args)) (I also had to do some stuff with trying different executables and environments, but you probably won't need that bit).

rpartition can probably be replaced with os.path.split (lol a bit sloppy on my part).

I was of course searching for requires.txt -- I only skimmed the issue but I imagine the egg_info_dir is probably more of interest to tox

Member

asottile commented Feb 28, 2017

Note that I also ran into something similar to this while doing some (admittedly awful) setup.py introspection. My solution (while slightly janky) worked pretty well for me and may benefit the tox project:

            out, srcdir = _call_setup_py(srcdir, _find_setup_py(), 'egg_info')
            for line in out.decode('UTF-8').splitlines():
                if line.startswith('writing ') and line.endswith('/PKG-INFO'):
                    _, pkg_info = line.split(' ')
                    egg_info_dir, _, _ = pkg_info.rpartition('/')
            requires_txt = os.path.join(srcdir, egg_info_dir, 'requires.txt')

The implementation of _call_setup_py is essentially subprocess.check_output((setup_py, *args)) (I also had to do some stuff with trying different executables and environments, but you probably won't need that bit).

rpartition can probably be replaced with os.path.split (lol a bit sloppy on my part).

I was of course searching for requires.txt -- I only skimmed the issue but I imagine the egg_info_dir is probably more of interest to tox

selimb added a commit to selimb/tox that referenced this issue Mar 9, 2017

selimb added a commit to selimb/tox that referenced this issue Mar 9, 2017

@obestwalter obestwalter added this to backlog in Squash all the bugs Mar 10, 2017

@obestwalter

This comment has been minimized.

Show comment
Hide comment
@obestwalter

obestwalter Mar 10, 2017

Member

@selimb looks like a PR is on the horizon for this - great :)

Member

obestwalter commented Mar 10, 2017

@selimb looks like a PR is on the horizon for this - great :)

@obestwalter obestwalter moved this from backlog to in progress in Squash all the bugs Mar 10, 2017

selimb added a commit to selimb/tox that referenced this issue Mar 10, 2017

selimb added a commit to selimb/tox that referenced this issue Mar 16, 2017

selimb added a commit to selimb/tox that referenced this issue Mar 16, 2017

@obestwalter obestwalter moved this from doing to done in Squash all the bugs Mar 26, 2017

gaborbernat added a commit to gaborbernat/tox that referenced this issue Mar 27, 2017

gaborbernat added a commit to gaborbernat/tox that referenced this issue Mar 27, 2017

@obestwalter obestwalter moved this from done to released/wontfix in Squash all the bugs Apr 18, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment