python: can not use imperatively installed packages as libraries

[zhou13]

1. nix-env -i python2.7 python2.7-numpy
2. open python2.7
3. import numpy

I got

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named numpy

Is this expected?

[FRidh]

Whether this is to be expected I am not sure, perhaps @domenkozar or @chaoflow can explain? I still find this confusing myself as well.

Anyhow, I can recommend using nix-shell. E.g.

nix-shell -p python27Packages.numpy

gives you an environment which has numpy and all its dependencies in it. You can then simply run python to get a Python interpreter which has access to all specified modules. Adding multiple packages is easy as well:

nix-shell -p python34Packages.scipy python34Packages.notebook

In this case scipy, and the Jupyter Notebook are installed along with all dependencies.

[vcunat]

Is this expected?

You use an unsupported workflow, I believe. Please, use nix-shell & friends instead. (Nix+python people should know more particular details.)

[zhou13]

Well, I think this should be supported. That is inconvenient. Whenever I want to use packages such as bash, I do not need to nix-shell -p bash.

Maybe hard code the path ~/.nix-profile/lib/python2.7/site-packages/ into python 2.7 would solve this problem? Is there any reason not doing this as the user explicitly says nix-env -i?

[bjornfor]

Well, I think this should be supported. That is inconvenient. Whenever I want to use packages such as bash, I do not need to nix-shell -p bash.

+1.

I tried to implement it once: #792

[bjornfor]

Btw, I think it should be controlled by environment variables, not hardcoding. If, for instance, python picked up $NIX_PROFILES and appended ./lib/pythonVERSION/site-packages to each element, it'd be easy to have a pure / clean python by simply clearing $NIX_PROFILES.

[domenkozar]

In general (as said before) I'm against this.

But I do understand there is a need/expectation it works this way, so I'll think about it. The main problem is there is just one $PYTHONPATH so setting it for a profile might lead to weird side-effects.

[zhou13]

Is there any reason against this? Append the local profile path with python version in pkgs/development/python-modules/recursive-pth-loader/sitecustomize.py seems to be a elegant solution for me. You do not even need to touch $PYTHONPATH.

[vcunat]

TL;DR: the general reason against this is that you add (impurely) things to scope, which isn't always a good thing.

Would all python stuff would now magically "see" what you have in your profile? What would happen if it would now have multiple different versions of the same things in scope (e.g. built with a differently configured python)? On a standard distro all versions on a system are "kept consistent" but with nix we impose no such restrictions, and some packages don't handle well if you attempt to plug-in "incompatible versions".

[domenkozar]

Yes, there is only one $PYTHONPATH and python will use first package that can be imported from it. So setting $PYTHONPATH in profile will pollute anything else depending on it. We could then disallow $PYTHONPATH propagating into Nix packages.

[dlukes]

Well, I think this should be supported. That is inconvenient. Whenever I want to use packages such as bash, I do not need to nix-shell -p bash.

+1 Another user here who found this behavior unexpected :) But then I'm very new to NixOS, so it's not surprising that my expectations are somewhat "traditional".

I (think I) understand the reasons not to modify $PYTHONPATH, but why does then perl use this exact solution? (Modulo $PERL5LIB instead of $PYTHONPATH, of course.)

On a vanilla instance of zsh under NixOS, $PERL5LIB is set to /home/dvl/.nix-profile/lib/perl5/site_perl:/nix/var/nix/profiles/default/lib/perl5/site_perl:/run/current-system/sw/lib/perl5/site_perl.

(I realize fully well it's problematic -- I'm redefining $PERL5LIB in my own zsh init files, so it took me a while to figure out I must be careful not to clobber the previous value set system-wide. Then I thought, yay, python probably works the same -- only to end up here :) )

As an aside, isn't there perhaps a configuration option when compiling python/perl that sets the path to search for libs? I'm pretty sure there is one for perl, python probably has something similar... Wouldn't that be a clean way to do this?

[dlukes]

To make matters worse, it looks like different python packages have more than one way of doing it (no pun on perl intended). See for instance nix-env -ri python python2.7-flask (I'm on NixOS version 15.09.343.1b83abb (Dingo), btw):

>>> import flask
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/dvl/.nix-profile/lib/python2.7/site-packages/flask/__init__.py", line 19, in <module>
    from jinja2 import Markup, escape
  File "/nix/store/q8hxg7pvsnrax4jk141cbqx68m0jx33w-python2.7-Jinja2-2.7.3/lib/python2.7/site-packages/jinja2/__init__.py", line 33, in <module>
    from jinja2.environment import Environment, Template
  File "/nix/store/q8hxg7pvsnrax4jk141cbqx68m0jx33w-python2.7-Jinja2-2.7.3/lib/python2.7/site-packages/jinja2/environment.py", line 13, in <module>
    from jinja2 import nodes
  File "/nix/store/q8hxg7pvsnrax4jk141cbqx68m0jx33w-python2.7-Jinja2-2.7.3/lib/python2.7/site-packages/jinja2/nodes.py", line 18, in <module>
    from jinja2.utils import Markup
  File "/nix/store/q8hxg7pvsnrax4jk141cbqx68m0jx33w-python2.7-Jinja2-2.7.3/lib/python2.7/site-packages/jinja2/utils.py", line 520, in <module>
    from markupsafe import Markup, escape, soft_unicode
ImportError: No module named markupsafe

Unlike numpy (see OP), flask is reachable after installation, and even some of the dependencies which were fetched automatically (jinja2). Others (like markupsafe), however, are not, and have to be specified manually. Which is why import flask above ultimately fails, but if you do nix-env -ri python python2.7-flask python2.7-markupsafe python2.7-Jinja2 python2.7-itsdangerous, it succeeds.

Why -r? Because sys.path changes depending on whether numpy (and perhaps other packages?) is installed. This is perhaps the most confusing thing about this... Without numpy, sys.path contains directories under ~/.nix-profile/lib/ (where flask et al. are symlinked, with some dependencies linked automatically and some having to be requested explicitly, as explained above); with the numpy package installed, these directories are gone from sys.path (and numpy is not even symlinked there).

This really should be made consistent somehow, if at all possible. I guess the problem is that this is entirely up to the author of the python package in question, and if I understand it correctly, there are several competing tools for generating nixpkgs from PyPI?

To round the confusion off, when python is installed as a system package and flask via nix-env, flask is unreachable once again because the directories under ~/.nix-profile/lib/ are gone from sys.path once more (even though flask is symlinked there).

Again, all of this works flawlessly with perl. System-installed packages are consistently symlinked under /run/current-system/sw/lib/perl5/site_perl or similar, user-installed ones under ~/.nix-profile/lib/perl5/site_perl. Granted, currently, it works at the expense of the dirty trick with setting $PERL5LIB to point to these directories, but as said before -- these paths could also be hardcoded when compiling the perl interpreter.

A similar solution for python would go a long way towards alleviating the concerns described in this thread, and would be consistent with the principle of least surprise.

[domenkozar]

Installing python packages with nix-env is just not supported, all the problems you're seeing is because you're using software in a way that is not meant to be used. What exactly are you trying to achieve?

[dlukes]

Installing python packages with nix-env is just not supported

I understand that (I've read the rest of the thread), I just agree with the OP and others above that current behavior is confusing from the perspective of an outside user (i.e. someone who is not a dev on NixOS) and addressing this (not necessarily be enabling this type of workflow, see below) would ease the transition for newcomers.

The semantics of "installing" something is that afterwards, you're able to use it in the way it's intended to be used. In the case of an executable, this means running it; in the case of a library, this means accessing it as a resource. It's counterintuitive that one works, but the other doesn't (I guess because with bin directories, it's only a matter of setting up some symlinks and having a correct $PATH, so that's always done, whereas there's a jillion ways libraries can be set up...?).

I guess what I'm saying is that if installing libraries with nix-env (mostly) works for at least one language (perl) and (mostly) doesn't work for another one (python), nix-env should issue a warning that only executables are expected to be made accessible in the user environment by the install, and behavior for libraries is undefined.

Please note that I wouldn't dream of arguing about the technical design decisions on a project I still know very little about (though I find it fascinating, extremely useful, and am trying to learn more). This is all about user experience design -- communicating the technical decisions to (new) users, who may have a specific set of (wrong) expectations, in the most frictionless way possible.

What exactly are you trying to achieve?

Well, originally, I was trying to pull in some python packages via environment.systemPackages. They're imported by some of my utility scripts, so I thought it would be nice not to have to pull them in manually after each system install. But (same story) my scripts couldn't find them. This discussion around nix-env overlaps with my use-case in many respects, so I didn't want to fragment it by starting another issue :)

But the same logic applies here, perhaps even more -- what's the purpose of installing a python library as a system package, if not to define an extended standard library of sorts, i.e. modules that one happens to use a lot? I've now effectively achieved this anyway by emending my $PYTHONPATH, but trying to find out whether it wasn't working out of the box because I was doing something wrong, or because that's how it's intended to behave, was a bit frustrating...

Unfortunately, unlike with nix-env, where the user can easily be warned not to expect installed libraries to be importable whenever they run the command (perhaps with an option to turn the warning off in their configuration.nix?), I can't think of a good place to put a similar warning for environment.systemPackages. Perhaps after each nixos-rebuild? What about the first install?

[domenkozar]

I understand that (I've read the rest of the thread), I just agree with the OP and others above that current behavior is confusing from the perspective of an outside user (i.e. someone who is not a dev on NixOS) and addressing this (not necessarily be enabling this type of workflow, see below) would ease the transition for newcomers.

Yes, it takes effort and time to get used to Nix tooling. But you have to understand that supporting imperative installation of python packages brings a lot of problems. It's really hard to make it work reliably and I'd rather not support something than support it with lots of caveats and exceptions.

The semantics of "installing" something is that afterwards, you're able to use it in the way it's intended to be used. In the case of an executable, this means running it; in the case of a library, this means accessing it as a resource. It's counterintuitive that one works, but the other doesn't (I guess because with bin directories, it's only a matter of setting up some symlinks and having a correct $PATH, so that's always done, whereas there's a jillion ways libraries can be set up...?).

Coming from a traditional unix environment I understand the concern. But we don't install headers for C packages, so you're forced to use Nix tooling to build software. Same goes for Python, except the line between usage and development is very blurred.

Well, originally, I was trying to pull in some python packages via environment.systemPackages. They're imported by some of my utility scripts, so I thought it would be nice not to have to pull them in manually after each system install. But (same story) my scripts couldn't find them. This discussion around nix-env overlaps with my use-case in many respects, so I didn't want to fragment it by starting another issue :)

We do offer alternatives, like using python.buildEnv which is declarative alternative to nix-env imperative installation. They both install packages into an user environment.

But the same logic applies here, perhaps even more -- what's the purpose of installing a python library as a system package, if not to define an extended standard library of sorts, i.e. modules that one happens to use a lot? I've now effectively achieved this anyway by emending my $PYTHONPATH, but trying to find out whether it wasn't working out of the box because I was doing something wrong, or because that's how it's intended to behave, was a bit frustrating...

I really recommend using nix-shell -p pythonPackages.django and you'll get a shell with $PYTHONPATH set.

Unfortunately, unlike with nix-env, where the user can easily be warned not to expect installed libraries to be importable whenever they run the command (perhaps with an option to turn the warning off in their configuration.nix?), I can't think of a good place to put a similar warning for environment.systemPackages. Perhaps after each nixos-rebuild? What about the first install?

This is a documentation issue and I agree we should address it.

[zhou13]

At least nix-shell -p does not solves my problem. As a PhD student, I need to use a lot of python science libraries to handle data. I just hope I can use python as a calculator conveniently!
I found it unfair that $PATH is supported by NixOS but python library is not. I read all the discussion and I still do not understand why I cannot have a working numpy just like bash.

TL;DR: the general reason against this is that you add (impurely) things to scope, which isn't always a good thing.

The same argument applies to bash. Why you support impurely bash in 'nix-env' but not python packages?

Would all python stuff would now magically "see" what you have in your profile?

That is exactly what we want. (but not all python stuff, just the common packages and their dependency that we asked)

What would happen if it would now have multiple different versions of the same things in scope (e.g. built with a differently configured python)?

A sane person will not pull two same libraries into his profile. Maybe we can give a conflict warning/error message under such a abnormal case?

On a standard distro all versions on a system are "kept consistent" but with nix we impose no such restrictions, and some packages don't handle well if you attempt to plug-in "incompatible versions".

This does not mean that nix-env cannot be supported for python libraries.

We do offer alternatives, like using python.buildEnv which is declarative alternative to nix-env imperative installation. They both install packages into an user environment.
I really recommend using nix-shell -p pythonPackages.django and you'll get a shell with $PYTHONPATH set.

I don't like to type nix-shell -p every time I open a new shell.

In addition, according to my understanding, nix-shell -p also does not solve the problem when I need both python2.numpy and python3.numpy at the same time. So does python.buildEnv.

[FRidh]

@zhou13 I also use Python a lot for modelling and such. While a bit off-topic, maybe my setup works for you.

What I have is a shell.nix file which contains all the tools Python libraries I need. Since I am mostly working on a single project, I put this file in the root of my home folder. That way, I just need to type nix-shell.

Furthermore, there are some packages I develop and need and which are also included. Each of them has a release.nix file with buildPythonPackage, etc.

with import <nixpkgs> {};

( let
    python = "python34";
    pythonPackages = pkgs.${python+"Packages"};

    # Not yet available on unstable
    tabulate = pythonPackages.buildPythonPackage rec{
      version = "0.7.5";
      name = "tabulate-${version}";

      src = pkgs.fetchurl {
        url = "https://pypi.python.org/packages/source/t/tabulate/${name}.tar.gz";
        sha256 = "9071aacbd97a9a915096c1aaf0dc684ac2672904cd876db5904085d6dac9810e";
      };

      buildInputs = with pythonPackages; [ nose ];

      # Tests: cannot import common (relative import).
      doCheck = false;

      meta = {
        description = "Pretty-print tabular data";
        homepage = https://bitbucket.org/astanin/python-tabulate;
        license = licenses.mit;
        maintainer = with maintainers; [ fridh ];
      };
    };

    # Packages I develop and depend on
    geometry = callPackage ./Code/geometry/release.nix { pythonPackages = pythonPackages; };
    ism = callPackage ./Code/ism/release.nix { pythonPackages = pythonPackages; geometry = geometry;};
    acoustics = callPackage ./Code/acoustics/release.nix { pythonPackages = pythonPackages; tabulate = tabulate; };
    auraliser = callPackage ./Code/auraliser/release.nix { pythonPackages = pythonPackages; acoustics = acoustics; ism = ism; geometry = geometry; };


in pkgs.${python}.buildEnv.override rec {

  extraLibs = with pythonPackages; [ acoustics auraliser blaze cytoolz dill geometry ipython ism matplotlib memory_profiler nose notebook numba numpy pandas pyqt4 pytest scipy toolz ];
}
).env

What also might be convenient for you is something like

packageOverrides = pkgs: with pkgs; {
    workEnv = pkgs.myEnvFun {
        name = "work";
        buildInputs = [
        python34
        python34Packages.ipython
        python34Packages.numpy
        python34Packages.scipy
        python34Packages.matplotlib
        python34Packages.pandas
        ];
    };

in your .nixpkgs/config.nix. Install this with nix-env-i env-work and use it with load-env-work. Personally I like to avoid nix-env -i and just use nix-shell instead.

I agree it is a bit inconvenient not to be able to access an interpreter directly with all the tools you need. Luckily only a couple of characters are needed :-)

I guess the former (python.buildEnv) can also be used config.nix but I haven't succeeded at that yet.

[CMCDragonkai]

How does things like pip and easy_install factor into this discussion?

[joachifm]

It seems pretty clear to me that the OP wants something that the maintainers are not interested in implementing, so I don't see how this can be resolved without the issue opener providing some code.

[joelmo]

I think the solution to this would be to allow setting propagatedUserEnvPkgs. The python.buildEnv functions will include all propagatedBuildInputs to the env. I think this is wrong because this attribute describes what's needed for the package to be built. I think we should have an attribute or function that tells which packages are needed for running.

The builder for python packages can be extended to scan python imports and tell which package provides them. Test cases should probably not be installed so deps on tests wont be included. And then produce the propagated-user-env-pkgs metadata. Another option is to rename this file and patch python to process these.
I believe this would work for most packages, for others the extra attribute where runtime deps are specified is needed.

[FRidh]

What some of you might be interested in is the following.

Environment defined in separate .nix file

Say we have a build.nix with

with import <nixpkgs> {};
with python35Packages;

python.buildEnv.override rec {

  extraLibs = [ numpy scipy ipython ];
}

then you can install this into your profile with

nix-env -i -f build.nix

Now you have an interpreter with the packages that you added.

Environment defined in ~/.nixpkgs/config.nix

Define the environment

  packageOverrides = pkgs: with pkgs; {
    blogEnv = python35Packages.python.buildEnv.override {
      extraLibs = with python35Packages; [
        pelican
      ];
    };
  };

and install with

nix-env -iA nixos.blogEnv

Note that I'm using the attribute path here.

In #15804 a shorter notation for adding packages to a Python environment was introduced, so now we can also write

blogEnv = python35.withPackages (ps: [ps.pelican]);

Limitations

As you might imagine there is one limitation here, and that's you can install only one environment at a time. You will notice the complaints about collisions when you try to install a second environment.

[joelmo]

@FRidh thanks for sharing this. I believe this is currently the best/only option, but I would not count that method as imperative.

Some time ago I opened a PR for adding runtime dependencies to matplotlib: #13939. This was rejected with the motivation that the right way to install packages was using the buildEnv function.

I think this should be considered again, if people want to imperatively install python packages and submits patches that don't interfere other ways of installing other packages, can't those be accepted?

[FRidh]

@joelmo we have several methods for installing or building environments now, and there are still issues to be solved with those methods as well. Adding additional methods I fear will only make it more confusing and harder to maintain.

[joelmo]

What are those issues? Lots of issues are listed here, #1819. Maybe Circular dependencies, and #2412 (setuptools namespace collisions). Test improvements can also be considered more important #3821 (import tests).

[FRidh]

A big issue for many is that virtualenv and tox aren't working well. We use a lot of wrappers now that set PYTHONPATH or PYTHONHOME as well as symlinks. Depending on how you use a package you need one wrapper, or the other, or none. What needs to be done first is creating a clear overview of what cases we want to support, and how the current implementation looks like.

#13939 was not a generic solution. A generic solution would be something like setting propagatedUserEnvPkgs = propagatedBuildInputs in buildPythonPackage, but if I recall correctly that's not sufficient. If you can come up with a generic solution, great, show it. But before we actually include another method I think we need to do first that which I described in the previous paragraph.

[knedlsepp]

Actually why don't we patch .py files to use hard coded absolute imports? This would be more consistent with the RPATH approach of regular shared libraries and binaries.

[FRidh]

@knedlsepp Such approach definitely would make sense as well.

I suppose with tokenize we could get a long way in converting all import ... and from ... import ... statements. However, how should we deal with code that itself uses importlib? This might work just fine, I don't know. I guess best would be to make an experiment; implement the proposed change and see how it works :-)

[FRidh]

Note that in scripts we also add Python libraries to site. See the wrap Python code.

[CMCDragonkai]

@FRidh I used your method to make the python layer (using anaconda mode) in Spacemacs work. It's easier than loading all "IDE" dependencies in every shell.nix for every Python project I'm working on. However it appears not all anaconda mode dependencies are available, but the layer no longer complains. The building of a custom python environment should be encouraged as the way to make things like this work. Although I wonder if this method is used for other languages like nodejs... etc. This does mean that for a proper dev environment you tend to have a user-profile language interpreter with minimal dependencies to make your IDE work, while having project specific interpreters inside a nix-shell. The downside is that usually layer/mode features that use the interpreter or compiler won't be using the project specific interpreter/compiler specified in the nix-shell, you just have to use a separate terminal to perform those actions, and you can't run everything inside emacs.

FYI, with spacemacs, one can launch a terminal inside, and then run nix-shell inside there.

For reference this is my pythonEnv inside config.nix:

          # custom python environment with python packages embedded
          # nixos isolates each language specific dependency, they are not automatically exposed to the interpreter
          # note that this should not be used for development
          # only for IDE integration and experiments
          pythonEnv = with pkgs; buildEnv {
            name = "pythonEnv";
            paths = [
              (with python27Packages; python.buildEnv.override {
                  extraLibs = [
                    setuptools
                    numpy
                  ];
              })
              (with python35Packages; python.buildEnv.override {
                  extraLibs = [
                    setuptools
                    jedi
                    flake8
                    numpy
                    isort
                    yapf
                    pytest
                  ];
              })
            ];
          };

[FRidh]

In #25985 I propose using a sitecustomize.py for PYTHONPATH manipulation. We could write a sitecustomize.py that searches in $NIX_PROFILES for site-packages and adds those with site.addsitedir (like e.g. #792).

The question would then be, how to enable this feature. There are I think two methods:

1. rebuild Python derivation so it includes the sitecustomize.py. That would require rebuilding all packages as well.
2. composition using e.g. python.buildEnv. Shebangs in packages will however still point to the original Python build.

The suggestion

it'd be easy to have a pure / clean python by simply clearing $NIX_PROFILES

is not acceptable, as that would change the default.

[bjornfor]

Option 3: Only look in $NIX_PROFILES if $PYTHON_USE_NIX_PROFILES (or something) is set. The default could be off, but it'd be trivial for users to turn it on (no rebuild needed).

[FRidh]

The issue with env variables is that they leak. You may want to have it enabled for your Python "environment", but Python applications also become impure right away. Unless we explicitly clear this env var in our wrappers...

[bjornfor]

Well, I want a global switch :-) If you don't then you don't have to enable it.

[siriobalmelli]

Here's my (impure, heathen, infidel, sullied, etc) global switch, in ~/.bashrc:

export PYTHONPATH=$(find -L ~/.nix-profile -path "*/python*/site-packages" -type d | tr '\n' ':')$PYTHONPATH

Notes:

• Haven't tested this with NixOS
• Works on OS X

[coretemp]

Did anyone get pip install and easy_install to work by any chance?

[danbst]

@coretemp see https://gist.github.com/danbst/dd641c696f77f5465ef9c827fcbf1e2c, I am able to use pip install inside nix-shell and it replaced virtualenv for me. There are three lines, which are essential:

            alias pip="PIP_PREFIX='$(pwd)/build/pip_packages' \pip"
            export PYTHONPATH="$(pwd)/build/pip_packages/lib/python2.7/site-packages:$PYTHONPATH"
            unset SOURCE_DATE_EPOCH

[coretemp]

Looks interesting. If it is a general method, which it looks like, I would like to see it in the manual.

[danbst]

@coretemp added a section to Wiki (https://wiki.nixos.org/wiki/Python#Emulating_virtualenv_with_nix-shell). I don't know whether others use this same technique, but it is definitely copypasted somewhere from internet.

[sbourdeauducq]

I don't like to type nix-shell -p every time I open a new shell.

Additionally, -p doesn't work when the Python packages that you want are not in nixpkgs.

[evelineraine]

I have worked around this by adding per-profile, version-specific site-packages paths to sys.path in my local usercustomize.py, which I placed in a $PYTHONPATH directory.

I use Nix on Fedora. I am now able to import nix-installed pygit2 in nix-installed python3.8.

usercustomize.py:

nix_profiles = map(Path, os.environ.get("NIX_PROFILES", "").split())

python_version_majorminor = "{}.{}".format(
    sys.version_info.major, sys.version_info.minor
)
site_packages_path = "lib/python{}/site-packages".format(python_version_majorminor)

profile_site_packages = filter(Path.exists,
    map(lambda p: p / site_packages_path, nix_profiles)
)
sys.path.extend(map(str, profile_site_packages))

See usercustomize.py in my dotfiles for a full implementation.

[stale]

I marked this as stale due to inactivity. → More info