Trigger virtualenv recreation on Python updates

[jhermann]

virtualenvs tend to break horribly after certain host Python updates – those that change internal, statically linked modules, or make symlinked .so modules incompatible to the old "python" binary contained in the virtualenv.

Just putting the idea out there, I might do it if and when I find the time.

[nailor]

I have a vague memory that the python update triggers a hook one could register to. That might be a way to go.

This feature would be really sweet :)

[jhermann]

I'm considering another path of resolution, by simply symlinking the critical binaries to their official place (after calling virtualenv). Since we know we're on Debian, have symlinks, paths are versioned, yada yada, this is no problem in that limited scope (unlike for virtualenv itself).

Do you have an "aesthetic" problem with such a pragmatic solution? ;) If not, I'd try if that works as expected.

[jhermann]

I looked into an actual package tree on Debian Wheezy, and pretty much everything except the bin/python binary already is a symlink (including "lib-dynload", which is the exact reason why things can go awry when the host Python is upgraded). The binary must be a physical copy (i.e. a hardlink or a copy), else some readlink magic interferes and breaks the venv.

Anyway, force-hardlinking (and if that fails, copying) the executable is thus enough to get existing virtualenvs in sync again, so that should be done in a trigger script – and also in a postinst. A full recreate of the virtualenv possibly breaks more than it helps.

[nailor]

Thanks for your investigations!

I think force-hardlinking/copy would probably be a good idea. Do you know if the Debian packaging system offers a possibility to detecting the change of the Python version?

[jhermann]

Working on this now… and looking good.

# dpkg -l nodeenv 
ii  nodeenv                     0.11.1-1           amd64              Node.js virtual environment builder

# dpkg-trigger --by-package python2.7-minimal /usr/bin/python2.7

# dpkg -l nodeenv 
it  nodeenv                     0.11.1-1           amd64              Node.js virtual environment builder

# dpkg --configure nodeenv
Processing triggers for nodeenv (0.11.1-1) ...
postinst triggered called with 2 args: triggered /usr/bin/python2.7

# apt-get install --reinstall python2.7-minimal
…
Processing triggers for nodeenv ...
postinst triggered called with 2 args: triggered /usr/bin/python2.7
Setting up python2.7-minimal (2.7.3-6+deb7u2) ...

[jhermann]

See ekalinin/nodeenv#104

This BTW also allows to safely install (pure Python) packages built on Ubunto onto a Debian system with the same main Python version (which was actually my test case).

Now the question is how to properly integrate this into dh-virtualenv… as debhelper auto-generated code? I have no clue about how that works.

The other option would be to provide a https://github.com/audreyr/cookiecutter or similar stencil for dh-virtualenv projects as part of dh-virtualenv.

[nailor]

Yeah, it's a bit tricky thing to handle in the build steps. Debhelper indeed autogenerates code in some cases, but don't know the mechanism. I don't think there's any other method to do it though. Cookiecutter stencil is something I have always wanted, so that could be a really good starting point, even if it is not perfect.

Nice work digging and testing that trigger stuff!

[nailor]

Hmm. Looking at some documentation and, while it is sparse, it kind of hints that just generating postinst.debhelper file might get included by the dh_installdeb into the postinst.

Could be worth experimenting with that

[jhermann]

The magic incantation is "autoscript": https://github.com/dankamongmen/sprezzos-world/blob/master/packaging/debhelper/Debian/Debhelper/Dh_Lib.pm#L500

Combined with the "-n, --noscripts" option – grepping for that in dh_* scripts brings up quite a few examples.

[jhermann]

Basic autoscript code is in jhermann@a84e51d, I just have to add and test the actual postinst code tomorrow.

Already creates a snippet like this:

# Automatically added by dh_python2:
# dh-virtualenv postinst autoscript
dh_venv_install_dir='/usr/share/python/devpi'
dh_venv_package='devpi'

# End automatically added section