check on pkg_resources.declare_namespace breaks backwards compatibility

[boegel]

installing older versions of vsc-base with the current vsc-install fails with:

Traceback (most recent call last):
  File "setup.py", line 65, in <module>
    shared_setup.action_target(PACKAGE)
  File "/usr/lib/python2.6/site-packages/vsc/install/shared_setup.py", line 1345, in action_target
    vsc_setup().action_target(package)
  File "/usr/lib/python2.6/site-packages/vsc/install/shared_setup.py", line 233, in __init__
    self.package_files = self.files_in_packages()
  File "/usr/lib/python2.6/site-packages/vsc/install/shared_setup.py", line 375, in files_in_packages
    'Fix with pkg_resources.declare_namespace') % root)
Exception: vsc namespace packages do not allow non-shared namespace in dir /tmp/vsc40023/easybuild_build/vscbase/1.6.9/dummy-dummy/vsc-base-vsc-base-1.6.9/lib/vsc.Fix with pkg_resources.declare_namespace

In one way, this makes sense, but this is also very annoying because a system-wide vsc-install installation can't be easily bypassed, rendering older vsc-base versions impossible to install...

Thoughts on this?

[stdweird]

the problem has nothing to do with vsc-install, it's true for anything that you try to install and that has an entry in a .pth in PYTHONPATH.
and the problem might not be easy to fix, but it can be fixed.
is this a system python? out of curiosity, if this is a system python and you would manage to actually build a vsc-base module, would loading the module actually pick the vsc-base from the module via the PYTHONPATH, or also the system one via the .pth entry for vsc-base?

[boegel]

using a vsc-base/2.4.2 that was already installed:

$ module load vsc-base/2.4.2
$ ls /apps/gent/SL6/sandybridge/software/vsc-base/2.4.2/lib/python2.6/site-packages/
easy-install.pth  site.py  site.pyc  vsc_base-2.4.2-py2.6.egg
$ cat /apps/gent/SL6/sandybridge/software/vsc-base/2.4.2/lib/python2.6/site-packages/easy-install.pth 
import sys; sys.__plen = len(sys.path)
./vsc_base-2.4.2-py2.6.egg
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)

$ python -c "import vsc.utils.generaloption; print vsc.utils.generaloption.__file__"
/usr/lib/python2.6/site-packages/vsc/utils/__init__.py:29: UserWarning: Module vsc was already imported from None, but /vscmnt/gent_vulpix_apps/_apps_gent/SL6/sandybridge/software/vsc-base/2.4.2/lib/python2.6/site-packages/vsc_base-2.4.2-py2.6.egg is being added to sys.path
  import pkg_resources
/usr/lib/python2.6/site-packages/vsc/utils/generaloption.pyc

As long as the system vsc-base is backwards compatible, that's not a big issue though (but it does suck).

python -S may be a way around this, but you can't use this when installing Python packages to force Python to pick up the vsc-install you want (it breaks setuptools).

[boegel]

@stdweird also, I disagree that this is not a vsc-install issue, since we're literally breaking backwards compatibility; we're not longer able to install old vsc-* packages

and that without a major version bump of vsc-install...

[stdweird]

wait, you think that if you load a module and you get something else, this is OK? if EB can't properly install the software and module, why do you install them in the first place?
and if this is OK, it's even easier to fix, just see if you can import the python module you are trying to install, and if you can, print a message like "EB not installing package X, because, well, it doesn't matter anyway and we hope it's backwards compatible".

you can install older vsc- pacakges using an older vsc-install, if it makes you feel better we can increase the major version number, but not sure what it will change.

[boegel]

I'm not saying it's OK, I'd like it to b different, but I haven't found a way around it. Suggestions welcome.

[boegel]

Actually, we did figure this out for EB, and implemented a fix in the easyblock for EasyBuild, i.e. trash setuptools.pth in the install prefix because it results in reeling in system Python packages, cfr. easybuilders/easybuild-easyblocks#813.

It's not clear to me why this happens now with vsc-install too though...

I guess I'll have to dive into the magic that happens in shared_setup.py.

[JensTimmerman]

this error message is exactly what prevents the warning about
UserWarning: Module vsc was already imported from None, but /vscmnt/gent_vulpix_apps/_apps_gent/SL6/sandybridge/software/vsc-base/2.4.2/lib/python2.6/site-packages/vsc_base-2.4.2-py2.6.egg is being added to sys.path

We spent all this time on trying to get rid of it, and now you want it back? :p
I do see that you sometimes would want an old vsc-base for reproducing certain things, but then use an old vsc-install to install it, should work.

I do agree that this check breaks backwards compatibility, but the old stuff was doing weird things, that are fixed and are checked for now. So, wont fix.

[boegel]

I'm not saying the check shouldn't be there, it's just very annoying that this breaks installing older versions... Also, I obviously don't want the warning back.

Anyway, it doesn't seem to matter too much, since the declare_namespace stuff is effectively making it impossible to make a particular non-system installation of a vsc-* package 'win' over the system-wide installed version... So even if we could still install older versions, there's no use, since they won't be used at runtime.

I see no way in which the declare_namespace in /usr/lib/python2.6/site-packages/vsc/install/__init__.py can be escaped when using the system Python, which results in that path getting preference; the only way I can find is python -S, but then other things are broken too...

For now, this is just annoying, it shouldn't break anything, but sooner or later it will, when the system vsc-base breaks backward compatibility. Other than installing old versions of vsc-* packages that is, because vsc-install broke backward compatibility already...

W.r.t. EasyBuild, it's sort of fine, since other sites should not be affected (unless they have vsc-install installed system-wide).

[JensTimmerman]

well, it only breaks packages that where doing things wrong, new vsc-base won't fail this check, so even if it does something you don't like in future releases, you would still be able to install the current version, right?

[JensTimmerman]

ok, the issue of having setuptools push system things right in front of the path, right,
python -S will make sure you don't pick up system libraries, you will probably need to install pkt_resources and setuptools in a local path and addt his to your PYTHONPATH if you want vsc-install to keep working, but once you do this you should be isolated from your system environment. (or for easybuild, load the setuptools module)
but always running python -S in easybuild is probably something you want to do anyway?