python: append paths from PYTHON<VERSION>PATH env var to sys.path

[bjornfor]

Every python interpreter that loads the recursive-pth-loader module will
now append paths to sys.path from PYTHONPATH environment
variable.

This means we can easily mix python2/3 environments and have modules
that are installed in a profile "activated" for the user by default. In
other words, python modules will be as easy to access from the python
shell as programs are to the (bash) shell.

For NixOS, the plan is to export PYTHON{26,27,32,33}PATH with
$PROFILE/lib/pythonVERSION/site-packages (for PROFILE in system_profile,
default_profile and user_profile, similarly as is done with PATH,
PERL5LIB and all other environment variables used to "activate" stuff
from profiles).

[domenkozar]

I don't have anything against providing hooks to supply additional modules. Not sure what's the best way to do that, but maybe I'd name it PYTHON3ADDTIONALPATH, because PYTHONPATH is already used by nix internally.

I do have very strong objection of exporting PYTHON3PATH in any nix profile by default. If I understand correctly that would pollute global python site-packages with installed software which is very unpure and causes major problems on all other distirbutions.

[bjornfor]

@iElectric: I don't think this is as bad as you think.

• It will only take effect if recursive-pth-loader is used. If you use pkgs.python you are not affected. (But if you use pkgs.pythonFull you'll get it.)
• Wrapped python programs control what environment variables they accept and their priorities (prefix vs postfix). So the possible extra modules you get from the nix profile can easily be ignored in wrappers, even if the wrappers use pkgs.pythonFull. So, IMO, adding this (and the PYTHONPATH export in NixOS) will not cause major problems. It just means we have to make wrappers that put their dependencies first in PYTHONPATH :-)

Btw, since wicd has been mentioned a couple of times, I took a look at it. It is actually a broken wrapper, because it puts PYTHONPATH from the environment first in the PYTHONPATH passed to python. So by putting a incompatible module in PYTHONPATH you can break it. Wrappers like this is fixed simply by ensuring that the package dependencies are listed first in PYTHONPATH.

Lastly, this PR will improve purity, at least for me, because I currently have

export PYTHONPATH=~/.nix-profile/lib/python2.7/site-packages/

in .bashrc, so that I can have have some extra stuff in ipython and spyder (Python IDE). But that is impure! It totally disregards python version differences...

[bjornfor]

I noticed that buildPythonPackage uses recursivePthLoader (I didn't know about that earlier). I don't want to introduce impurities in the builders, only interactive python "shells". To see if my PYTHONPATH exports from /etc/profile (in my local nixos tree) would introduce impurities, I did this:

Add preConfigure = "false" to demjson expression

nix-build -A pythonPackages.demjson -K

grep for PYTHON..PATH in /tmp/nix-build-python2.7-demjson-1.6.drv-0/env-vars

Turns out that there are no PYTHON..PATH environment variables there (only PYTHONPATH). This is good, because /etc/profile is for interactive shells and if it was sourced by the builders we'd have lots of impurities through the environment variables that are exported there.

I cannot think of a single example of where this PR (+ exporting PYTHONPATH in /etc/profile) would cause unwanted impurities. But if I've missed a possible bug scenario, please tell me.

[bjornfor]

Hm, seems like myEnvFun pulls in recursivePthLoader, even if only pkgs.python is used. Why is that needed? Wrapped programs seems to work without recursivePthLoader, see "wicd" wrapper for example. Need to investigate.

[bjornfor]

Oh, it is not myEnvFun that is pulling in recursivePthLoader, it is python modules that are built with buildPythonPackage.

@chaoflow: It seems you have added and removed recursivePthLoader from propagatedBuildInputs in <nixpkgs>/pkgs/development/python-modules/generic/default.nix a couple of times. Can you say something about this? Is it possible to let the python interpreter itself choose whether to use recursivePthLoader? So that for example pkgs.python can be maximally "pure", even with this PR + exporting PYTHONPATH in /etc/profile, and that pkgs.pythonFull or ipython (which is meant for interactive use) can automatically pick up modules installed in nix system/default/user profiles?

[chaoflow]

What do you think about

alias mypython="PYTHONPATH=$HOMEnix-profile/lib/python2.7/site-packages python"

We currently have plenty of ways to create python environments, but lack documentation. I'm not sure whether (currently) adding more ways is a good idea. Other opinions? @garbas, @cillianderoiste, anybody?

[chaoflow]

@bjornfor: nix-env -iA nixos.pkgs.python does not pull in recursivePthLoader for me. In what way do you see it to be impure?

I have ipython installed in multiple isolated profiles and don't want any of these to see my user profile or any other profiles.

What do you think about my shell alias proposal?

[chaoflow]

recursivePthLoader is used by buildPythonPackage to load a dependency's dependencies, i.e. it is needed.

[bjornfor]

@chaoflow: The mypython alias is not a bad idea, but I'd prefer if there was some interactive python interpreter that would pick this up by default, similarly to how shells pick up programs installed with nix-env -i thisprogram. It's not like it would pick up all python dependencies of stuff you install, only the python stuff you have actively installed to your profile. NixOS users can put that alias in environment.shellInit, but those just using nixpkgs must manage the alias "manually".

Regarding multiple ways of setting up python environments, what exactly are they? I know of myEnvFun and buildEnv. Are there more? myEnvFun/buildEnv are general concepts which is useable for much more than python. But only when using python is this the only way to create an environment (AFAIK). For interactive shells, just a simple "nix-env -i git" is enough to make "git" available to all shells, but it's not that simple for interactive python shells (yet).

I feel that for python we have "skipped as step" in the type of environments we can set up. I think the nix profile is the primary environment (it is simple to set up) and then when you need more advanced stuff, then you go for e.g. myEnvFun.

@chaoflow: pkgs.python doesn't pull in recursivePthLoader for me either, but when you combine it with a python module in a myEnvFun environment you get it. I was just surprised by that fact, because I thought pkgs.python would never use recursivePthLoader. But now I understand now that recursivePthLoader is used to load dependencies of python modules. (Btw, are there other ways to manage these dependencies?)

I think I should add that myEnvFun doesn't create "pure" environments. Currently it extends shell PATH, it doesn't replace it. I don't mind this behaviour. It puts the old PATH at the very end of the new PATH, so it can override stuff that might be installed in profiles. If we extend the same prinsiple to python, it is not necessarily a bad thing that python modules you have installed in your profile are visible in myEnvFun environments. If you have installed a module in your profile and then put a different module in myEnvFun, you get the one specified in myEnvFun. This is good.

To sum up, I feel we currently treat python different from shell/perl and I don't see any reasons why. This patch (and the to-be-patch for NixOS) just tries to even things out. And make simple things simple ;-)

[bjornfor]

@iElectric, @chaoflow: What are your opinions about this now?

Maybe I should explain again that this pull is all about giving python user the choice of what scope the modules he installs should have.

For ordinary, native executable programs, users can choose whether to have them available globally, in a user profile or in a custom myEnv. But for python, currently, there is only the option of using myEnv (AFAIK). With this pull users can choose to also make some modules available globally, or in a user profile.

This is just extending the flexible package scoping that Nix provides over to python. IMHO, it is not "impure".

[bjornfor]

@cillianderoiste: Yes, I know it's non-standard :/ But I think think our current way also confusing:

• Install python (nix-env -i python) and play around
• Hm, I seem to miss a module. Install it: nix-env -i python2.7-jedi
• Restart python only to find that the new module is not available.

I think the reason this is non-standard is that upstream Python developers don't expect users to have more than one Python version installed in a system. Or if you do then it's such an odd situation that the extra overhead of writing custom wrapper scripts or using virtualenv is negligeable. So they don't see the problem with having non-versioned PYTHONPATH variable. Unfortunately, the python2/3 mix is going to continue for quite some time. And I don't see why we have to force users to use the myEnvFun unless they have special needs.

Adding a PYTHON3PATH variable to Python has also been suggested upstream:
http://bugs.python.org/issue2375

And Perl has PERL5LIB (upstream).

Btw, we're using NixOS, that's as far away from "standard" as any linux distro gets. By being non-standard we can do awesome stuff. So I don't think being non-standard is necessarily a bad thing :-)

[bjornfor]

@cillianderoiste: What is the "plugins" pattern you mention? (I only know about wrapped programs and myEnvFun.)

[cillianderoiste]

(for reference, since we discussed this a bit more on IRC)
To my mind, making a new python module available to IPython would ideally be done by configuring it via nix in the same way that Firefox plugins are enabled, that's what I meant by the "plugins pattern" ... so indeed, wrapped programs.

Since we already have the facility to easily create a generically useful development environment with myEnvFun, that seems like the simpler and less surprising option to me i.e. you should be able to use myEnvFun for Node.js, Ruby, Erlang etc. rather than introducing further (in this case nix-specific) environment variables for each.

From my perspective myEnvFun is a great improvement over using virtualenv and zc.buildout, so I don't see it as an extra burden for new users to learn or an inconvenience at all. It's one of the first things I tell python developers about nix, so I understand that we're looking at it from very different angles.

You pointed out on IRC that we already set a number of environment variables in /etc/profile e.g. ALSA_PLUGIN_DIRS, GST_PLUGIN_PATH (I hadn't noticed these before) so what you're suggesting is very much along the same lines, maybe there's a good reason to support both approaches.

[bjornfor]

Yes, myEnvFun is a really cool tool for creating short-lived environments (you enter it, do work, and exit). But we also have the nix system profile, default profile and user profile for longer-lived environments. And I think that it is up to the user to choose in what scope they want to install things. Developers shouldn't force any particular usage pattern.

I think that, ideally, all programs and modules for any language should be useable from both the nix profiles and from myEnvFun setups. I realize that it is very difficult to achieve this and staying pure. That said, this is my attempt to do that for Python, and I think it has negligible impact on purity.

[cillianderoiste]

I wouldn't say myEnvFun is for short lived environments, although nix-shell (formerly nix-build --run-env) is, so it's worth mentioning too. myEnvFun gives you persistent environments which you can activate as you need, whereas nix-shell instantiates the environment and then drops you into it. I don't think you can activate the environment without evaluating the expression, which may result in a different environment compared to the last time you ran it. With myEnvFun you know that it will only change when you explicitly update it ... just in case that wasn't clear.

Of course, you can also install a myEnvFun environment into the system profile or user profile if you wish, or use a separate profile altogether.

[bjornfor]

Maybe "short-lived" was badly worded. What I mean is that myEnvFun is for setting up a environments with a limited scope, similar to how you can have local variables in a sub-routine. The nix profiles on the other hand, are for setting up more "global" environments.

Use of myEnvFun is a supplement, not a replacement, for the nix profiles.

[domenkozar]

What about we add PythonFullWithEnvPackages or something similar with would be essentially the same as PythonFull but with PYTHONPATH set to point to the profile? Then you also don't need to include python version in the variable name.

[bjornfor]

If it handles mixing versions without interference (as in Python27FullWithEnvPackages and Python33FullWithEnvPackages not loading each others modules), I'm all for it.

[bjornfor]

Closing this as I guess there are better ways to do it, e.g. the PythonFullWithEnvPackages idea from @iElectric.

The idea is to be able to provide a python that can be installed with nix-env and also be able to install (and "activate") modules for this python with nix-env. Just like it works in "standard" linux distros.

[domenkozar]

@bjornfor half a year later we have pythonFullBuildEnv where you'll have to modify .nixpkgs/config.nix:

  pythonPlone = pkgs.pythonFullBuildEnv.override {
     extraLibs = with pkgs.python27Packages; [ Plone ];
   };