how to customize modules available to ipython?

[chaoflow]

No description provided.

[bjornfor]

I can think of two alternatives.

(1) If ~/.nix-profile/... is added to PYTHONPATH (like we do with PATH, PERL5LIB etc), then ipython would automatically pick it up. And so would regular python (pkgs.pythonFull will, pkgs.python will not, because of some recursivePthLoader thing that it's missing).

I think the only issue with this approach is that we'd have to chose which python version to point PYTHONPATH to. Either the python 2.x directory, ~/.nix-profile/lib/python2.7/site-packages/ or the similar for the 3.x series. Python 2.7.x will be the last of the python 2.x series, so "python2.7" is a safe string to use for the 2.x series. I don't know how python names the corresponding 3.x directory, will there be python3.2, python3.3 etc?

This feels like something that should be fixed in the python interpreter by having it read a PYTHON2PATH or PYTHON3PATH variable. Maybe we should suggest it upstream. IMHO, it is the only sane solution to mixing python 2 and 3 in the same environment anyway.

(2) Another approach to managing PYTHONPATH is having each python package add a shell snippet in /etc/profile.d/* which augments PYTHONPATH. /etc/profile can source these shell snippets. This will however not enable python2 and 3 in the same environment, but it lets the user select whether to have a python2 or python3 environment, simply based on the packages that he installs to the profile. So it is more "automatic".

Downsides to the latter approach is that python it will not automatically pick up newly installed modules; you have to start a new shell for it to find them. Also, it differs from how we do it with perl and regular binaries. I think that having PERL5LIB and PATH "statically" pointed to the profile and not being augmented individually by packages is a good thing. Mostly because it means you can install something and start using it within the same shell.

[cillianderoiste]

I think the main risk with both these approaches is that any python program you use will also see the modules you've installed. I wonder if it's really a problem or not. I can imagine there must be some applications which will break if you don't pin a particular module to a specific version. An imaginary scenario: perhaps wicd requires a particular version of gtk but you install another version into your profile, and then wicd picks that up and fails.

[cillianderoiste]

How about using myEnvFun to create the environment you want: https://nixos.org/wiki/Howto_develop_software_on_nixos . You could use virtualenv from there to install python packages not yet packaged in nixpkgs.

[bjornfor]

Yes, I agree that is a risk. But it is the similar risk we take with e.g. PATH and PERL5LIB. For each program installed in your profile, every other program now has the ability to call that program through PATH.

For python wrapped programs the risk can be reduced/eliminated by simply forcing the value of PYTHONPATH, instead of appending to it. Or ensure that the program dependencies are listed in the beginning of PYTHONPATH, and let the PYTHONPATH from the environment be at the very end.

Using myEnvFun would mean that you have to "load-mystuff-env" to get into a shell where things are set up. I really want a solution that enables the nix profile itself to be of some general usefulness, and saving the custom shell environments for special occasions.

I mean, we could use myEnvFun things for tons of things, for example:

• python stuff: python, ipython, with custom modules
• browsing web: chromium, firefox
• photo stuff: gimp, gwenview
• scm stuff: git and mercurial
• ...

But that would be pretty cumbersome (IMO). I want my default profile to provide a kind of all-round environment where I can do most of my work, and not have to jump in and out of custom environments. I like having git in my default shell, without entering a special environment, and similarly, I like having python-construct availiable in ipython without doing anything special.

The only real issue, as far as I can see, is the mixing of python2 and python3; what should PYTHONPATH point to? Here is a python issue about adding PYTHON3PATH:
http://bugs.python.org/issue2375. It was opened in 2008 and last action was in 2010.

[domenkozar]

I think we have the opportunity to isolate packages from each other, something that no other distribution can afford. I submitted a pull request supporting python3 packages (which will be followed by pypyPackages). #786

Regarding "global" imports, we could add a nixos configuration value so that people can pollute global site-packages if they so desire.

[bjornfor]

@iElectric: What kind of config option did you have in mind? enablePollution = true? Or something more advanced?

I guess we could implement our own PYTHON3PATH handling, like suggested in the 2nd post at http://bugs.python.org/issue2375:

Add to site.py:
sys.path[1:1] = os.getenv("PYTHON3PATH", "").split(os.pathsep)

We could even extend this to include major.minor version numbers.

Also, I'd like to point out that installing a python module in your profile probably means that you also want it "activated". Modules don't end up in the profile by accident :-) That's why I'd like python modules in .nix-profile/ to work out of the box.

[domenkozar]

More like environment.python3SitePackages = [ pythonPackages.jinja2 ... ];.

But that's something very similar to using myEnvFun. In any case, this needs more usage and thinking.

[cillianderoiste]

@bjornfor

I mean, we could use myEnvFun things for tons of things, for example:

python stuff: python, ipython, with custom modules
browsing web: chromium, firefox
photo stuff: gimp, gwenview
scm stuff: git and mercurial
...

I'm not quite sure I understand, so I'm trying to think of a concrete example. If I remember correctly, Kdenlive looks for command line tools on the PATH e.g. for creating DVDs. I suspect that it also resolves paths to the nix store so you have to reconfigure it after upgrading. Maybe I'm wrong about the details, but I hope it serves as an example.

Ideally (IMO), these optional run time dependencies should be specified in a wrapper, and the fact that it can find them on the PATH is a bug. When you talk about using myEnvFun for the applications above is this the kind of thing you have in mind? I guess you are also talking about plugins, e.g. for chromium and firefox which aren't found on PATH but from looking up other environment variables.

In my mind, the worst that can happen in these cases is that a particular capability of the application will disappear or fail. That's a pain, but it should be possible to fix in the wrapper.

When it comes to python, my fear is that someone installs a random module and things totally unrelated break. Perhaps wicd or pavucontrol fail to start and they can no longer connect to the network or use audio.

On the other hand, it would be nice if the python interpreter (ipython, IDE's and other tools) could see python modules installed in the user profile.

Perhaps we should add an option to the python wrapper which allows some things to explicitly ignore PYTHONPATH e.g. wicd, and others to take it into account?

[bjornfor]

@cillianderoiste: Sorry for not making myself clear. I'll try again.

What I meant was that if the idea of "activating" things in the profile (by default) is considered dangerous, then the solution would be (to take it to the extreme) to say that the shell doesn't have access to any other commands either, other than load-env-foo commands. So, to use SCM tools, you'd have to make your own environment for e.g. git and mercurial and load-env-scm to use them. And if you want to look at photos or listen to music, you'd have to load-env-music or load-env-photo.

But I think we can both agree that this is pretty cumbersome!

NixOS currently "activates" lots of stuff, for example bin/, lib/mozilla/plugins/ and lib/perl5/site_perl/* (see /modules/programs/bash/profile.sh). But all (wrapped) programs have the ability to put these paths first, last or be completely ignored. We don't need to do anything special, it's already possible to do that. So, to some extent, I think we can say that if a program breaks because of something being installed in a user profile, it is a bug in that program/wrapper (because the wrapper can ignore those paths, if it wants to, or it can put those extra paths at the very end of its search path so that all non-optional dependencies are found in the right place and not overridden with stuff from the environment).

I think the main issue with these global environment variables is that they are not versioned. For example, GST_PLUGIN_PATH is probably read by both gstreamer-0.10.x and 1.0.x. And I'm pretty sure that a plugin written for 0.10.x will crash in 1.0.x.

But at least with Python I think we can do better. I'm currently experimenting with loading paths from PYTHONPATH, by extending sitecustomize.py in the recursive-pth-loader thing.

# Append paths from PYTHON<VERSION>PATH environment variable to sys.path
pythonpath = os.getenv("PYTHON%d%dPATH" % (sys.version_info.major, sys.version_info.minor))
if pythonpath:
    sys.path.extend(pythonpath.split(os.pathsep))

AFAIK, normal wrapped python programs use pkgs.python and that python doesn't use recursive-pth-loader. But pkgs.pythonFull and pkgs.ipython do use it, so they will automatically pick up the python modules from the profile. (Of course, a wrapped python program can use pkgs.pythonFull if it wants.)

For NixOS, I'm thinking of exporting PYTHON{26,27,32,33}PATH with value $PROFILE/lib/pythonVERSION/site-packages, for PROFILE in system_profile, default_profile and user_profile, similarly to what is done with PATH, PERL5LIB and all other environment variables used to "activate" stuff from profiles.

What do you think of PYTHONPATH?

[bjornfor]

Pull-request for PYTHONPATH: #792

[shlevy]

Was there a resolution to this issue? What further steps need to be taken?

[cillianderoiste]

I'm closing this. The implementation has since changed significantly, and I believe we have all cases covered now. Please re-open, or create a new ticket if there is something more to be done.