bad interaction: entrypoint script vs 'setup.py develop'

[warner]

I noticed a bad interaction today. In my "magic-wormhole" project (which uses versioneer), I created a new virtualenv and ran "pip install -e .". The project uses an entrypoint, and it wrote the entrypoint script into venv/bin/wormhole. The generated script looks like:

#!/Users/warner/stuff/tahoe/magic-wormhole/ve/bin/python2.7
# EASY-INSTALL-ENTRY-SCRIPT: 'magic-wormhole==0.2.0+22.g894da44.dirty','console_scripts','wormhole'
__requires__ = 'magic-wormhole==0.2.0+22.g894da44.dirty'
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.exit(
        load_entry_point('magic-wormhole==0.2.0+22.g894da44.dirty', 'console_scripts', 'wormhole')()
    )

This works fine until I make a commit, which changes the version that setup.py version reports into something different than the one frozen into the entrypoint script. After making a commit, running the script gives me an error:

pkg_resources.DistributionNotFound: The 'magic-wormhole==0.2.0+22.g894da44.dirty' distribution was not found and is required by the application

That makes me sad: if I want to use the virtualenv for testing the executable, I have to re-run pip install -e . after basically every commit.

I think one workaround might be to avoid entrypoints and just use a regular scripts= argument (in setup.py), with a small executable that does a plain import-and-run. I haven't been able to come up with a better idea.

[warner]

One additional datapoint: the console scripts generated when wheels are involved is much simpler, and just imports and runs. The new version of pip (7) tries to use wheels all the time, which is why I'm seeing many generate scripts on my system that would work with versioneer. Unfortunately pip install -e / setup.py develop can't use wheels, so they'll always use the version-pinning pkg_resources.load_entry_point() approach that interacts so badly with Versioneer's continually-updated version strings.

A new workaround option is to make sure your main package can be used as an executable (by using PACKAGE/__main__.py as the entry point, perhaps using moshez's new mainland tool), and then get into the habit of running python -m PACKAGE in your develop-based virtualenvs instead of running the installed executable script. You have to remember to not run the normal installed script (even though you've activated the virtualenv), since that will fail.

Another (painful) workaround would be to:

• install the package normally (just pip install into the virtualenv, without -e or setup.py develop)
• stash a copy of the generated script
• pip uninstall
• pip install -e
• then overwrite the (version-pinning) script with your (non-pinning) copy

I have to admit I'm still strongly tempted to have setup.py use scripts= on unix-like systems, and only add the entrypoint/console_script on windows, which would let real-install and devel-install work the same way on the unix platforms that I use.

[warner]

I think I understand this a bit better now.

• pip install -e . calls setup.py egg_info, then builds the entrypoint scripts (among other installation tasks)
  • setup.py egg_info uses Versioneer to compute the version (once), then writes the result into SRCDIR/PKG.egg-info/PKG-INFO
• that computed version is written into the entrypoint scripts, in two significant places (__requires__ and the first argument to pkg_resources.load_entry_point()).
  • when pkg_resources is imported, it scans for __requires__ in the top-level __main__ namespace, and bails if anything it wants is unavailable
  • load_entry_point() uses the packagename/version to find the right egg-info/entry_points.txt, to import the right module

If the egg_info (SRCDIR/PKG.egg-info.PKG-INFO) version doesn't match the generated entrypoint script's version, you get the error.

The trick is that a local development install (setup.py develop or pip install -e .) will update the egg_info in the source tree, and regenerate the entrypoint scripts in the install directory. So multiple install directories might cause you problems:

• virtualenv ve2; ve2/bin/pip install -e .; do some tests
• make a commit, or dirty the tree
• realize you should test against python3 too
• pyvenv-3.5 ve2; ve3/bin/pip install -e .

Now the ve2/bin/SCRIPT has the old version baked in, but second pip install -e rewrote the source tree's egg_info to have the new version. ve3/bin/SCRIPT will work, but not ve2/bin/SCRIPT.

Another common pattern that will break the local install is to run pip install .../path/to/tree from an unrelated directory: say a separate virtualenv for testing a downstream application against your development version of the Versioneer-controlled library. The install will, as a side-effect, update the egg_info in the library's source tree, leaving the other entrypoint scripts behind.

Even if you only have a single virtualenv, there are other commands that will run setup.py egg_info as a side-effect (register, sdist, and test being the most surprising ones). So you might take a break from your testing to make a new release, and the act of creating the sdist tarball changes the egg_info but not the virtualenv's scripts.

So my new workaround is:

• when using multiple (py2/py3) virtualenvs for a single project, do a pip install -e . in all of them at the same time
• if I ever need to run other setup.py or pip commands using that source tree, re-install-e in all virtualenvs afterwards

This seems to be ok. (incidentally, I think the scripts= entrypoints got a version baked in too, so that workaround didn't actually work)

If anyone can think of a better fix, I'm all ears, but I suspect this is going to have to be resolved with a docs change, rather than code.

[warner]

https://bitbucket.org/pypa/setuptools/issues/439 , fixed in setuptools-18.6, might fix this: they've stopped pinning the precise version number when installing with -e or develop.

[pelson]

Thanks for the commentary @warner - interesting problem. Looks like the 18.6 change broke setuptools for a few users so be sure to use 18.6.1.

[effigies]

Cannot reproduce with recent setuptools.