How to use PR in shell.nix ?

[573]

Thought pytorch-bin giving a try.

With shell.nix defined as:

let
  jupyter = import (builtins.fetchGit {
    url = https://github.com/tweag/jupyterWith;
    rev = "";
  }) {};

  iPython = jupyter.kernels.iPythonWith {
    name = "python";
    packages = p: with p; [
            # https://discourse.nixos.org/t/improving-nixos-data-science-infrastructure-ci-for-mkl-cuda/5074/43
            # curl -sL https://github.com/NixOS/nixpkgs/pull/96669.patch | head -n 1 | grep -o -E -e "[0-9a-f]{40}"
            (import (fetchTarball
                    "https://github.com/NixOS/nixpkgs/archive/6cce9c30786442f1fb2b6bd7c2c7c9a089e8648b.tar.gz")
                    {}).python3Packages.pytorch-bin
                  ];
  };

  jupyterEnvironment =
    jupyter.jupyterlabWith {
      kernels = [ iPython ];
    };
in
  jupyterEnvironment.env

I still can't import torch though neither in python nor jupyter lab.
Can't spot the error.

[guaraqe]

Checking from the beginning: are you able to produce a shell.nix just with a python3.withPackages containing your imported package? This is the mechanism JupyterWith uses, so if you are able to do this, JupyterWith works.

I don't know the package, but the -bin would make me thing this is not a library, but I may be wrong.

[573]

@guaraqe you're probably right, IDK if the -bin just refers to the CUDA-binaries bundled in pytorch-bin, maybe I should add pytorch (w/o -bin) as well.

With the python3.withPackages you mentioned, is that related to #95, meaning I had to follow the steps suggested in that ticket's solution right ?

EDIT: Hold on, just saw this section in the readme and will give that a try.

[guaraqe]

For python3.withPackages, I am just talking about Nixpkgs system of giving a Python interpreter with extra libraries, which is independent of JupyterWith. If you are able to do make it work independently of JupyterWith, you should be good.

The PR you mentioned is about JUPYTER_PATH, which is completely independent of the Python kernel, it is about the Jupyter executable being able to see certain executables. What you want is to make it visible to the Python kernel.

[573]

I am out of luck here. The following doesn't seem to work either:

  jupyterWithPath = builtins.fetchGit {
    url = https://github.com/tweag/jupyterWith;
    rev = "";
  };

  # Importing overlays from that path.
  overlays = [
    # Necessary for Jupyter
    (import "${jupyterWithPath}/nix/python-overlay.nix")
    (import "${jupyterWithPath}/nix/overlay.nix")
  ];

  # Your Nixpkgs snapshot, with JupyterWith packages.
  pkgs = import (fetchTarball
                    "https://github.com/NixOS/nixpkgs/archive/6cce9c30786442f1fb2b6bd7c2c7c9a089e8648b.tar.gz") { inherit overlays; };

  # From here, everything happens as in other examples.
  jupyter = pkgs.jupyterWith;

  iPython = jupyter.kernels.iPythonWith {
    name = "python";
    packages = p: with p; [
            pillow
            numpy
            requests
            pandas
            seaborn
            scipy
            matplotlib
            scikitlearn
          ];

          # https://discourse.nixos.org/t/development-with-nix-python/7356/2
          python3 = pkgs.python3.withPackages (ps: [ ps.pytorch ]);
        };

  jupyterEnvironment =
    jupyter.jupyterlabWith {
      kernels = [ iPython ];
    };
in
  jupyterEnvironment.env

gives

ModuleNotFoundError: No module named 'torch'

[GTrunSec]

Notice: my global python(or current JupyterWith nixpkgs channel) version is python3 (python37.withPackages). but this PR python version is python3(python38.withPackages) as default.

So, I'm going to

python37Packages.pytorch-bin;

test commit.

• https://github.com/GTrunSec/Jupyter-data-science-environment/blob/master/overlay/python-overlay.nix#L6
• https://github.com/GTrunSec/Jupyter-data-science-environment/blob/0194e5c71daa74e40b4cdaa8f8724989d372d6fb/my-default.nix#L92
  

[573]

@GTrunSec great, I tried with

# shell.nix
let
  jupyterLib = builtins.fetchGit {
    url = https://github.com/GTrunSec/jupyterWith;
    rev = "d9fe46bce29dcdb026807335f46cb2b8655dbf6b";
    ref = "current";
  };

  dscience = builtins.fetchGit {
    url = https://github.com/GTrunSec/Jupyter-data-science-environment;
    rev = "0194e5c71daa74e40b4cdaa8f8724989d372d6fb";
    ref = "master";
  };

  # Importing overlays from that path.
  overlays = [
    (import "${dscience}/overlay/python-overlay.nix")
    (import "${dscience}/overlay/package-overlay.nix")
  ];

  env = (import (jupyterLib + "/lib/directory.nix")){ inherit pkgs;};

  pkgs = (import "${dscience}/nix/nixpkgs.nix") { inherit overlays; config={ allowUnfree=true; allowBroken=true; };};

  jupyter = import jupyterLib {pkgs=pkgs;};

  iPython = jupyter.kernels.iPythonWith {
    python3 = pkgs.callPackage "${dscience}/overlay/python-self-packages.nix" { inherit pkgs; };
    name = "Python-data-env";
    packages = import "${dscience}/overlay/python-packages-list.nix" {inherit pkgs;};
    ignoreCollisions = true;
  };

  jupyterEnvironment =
    jupyter.jupyterlabWith {
      kernels = [ iPython ];
      extraPackages = p: with p;[ python3Packages.torchBin ];
      extraJupyterPath = p: "${p.python3Packages.torchBin}/lib/python3.7/site-packages";
    };
in
  jupyterEnvironment.env

but stuck here (link to log file)

The following tests FAILED:
2 - plasma-client-tests (Failed)
20 - arrow-io-file-test (Failed)
Errors while running CTest
builder for '/nix/store/g2r3ql5nh9ryf08lsw64l1pfm5dvdmyj-arrow-cpp-0.17.1.drv' failed with exit code 8
copying path '/nix/store/13520lflkakzr457l3c7nl4xp6vsrf2y-python3.7-PyQt5-5.14.2' from 'https://cache.nixos.org'...
cannot build derivation '/nix/store/yg2p2hj0j0ik7rnw1f9fw9sjk4vpxayw-python3.7-pyarrow-0.17.1.drv': 1 dependencies couldn't be built
building '/nix/store/vi2y8ka4wbmczf6qx4zizj2aq965xda0-generate-lockfile.drv'...
cannot build derivation '/nix/store/8h079k0mqkkzihdm2k3h4882fwsm5pln-python3-3.7.7-env.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/jc5ggg3p8h0m47sfxii2vy7ppx0rfg72-ipython-Python-data-env.drv': 1 dependencies couldn't be built
error: build of '/nix/store/7238ci01nrjaghigfmb357wb225is458-python-Python-data-env.drv', '/nix/store/jc5ggg3p8h0m47sfxii2vy7ppx0rfg72-ipython-Python-data-env.drv', '/nix/store/nyi31gsa98c3f80r84gwm73bf627p25v-python3.7-jupyterlab-2.1.2.drv', '/nix/store/vi2y8ka4wbmczf6qx4zizj2aq965xda0-generate-lockfile.drv' failed

[GTrunSec]

@573
first, this issue from my python package which name of zat(The pyarrow got a bug that would not be passed with this nixpkgs channel).
two way to solve it:

1. create the python-packages-list.nix by yourself. (Do not use my python-pkgs-list packages such as zat(this is for security data science) )

{pkgs}:
(with pkgs.python3.withPackages; (p: with p;  [  put your package name here ]

or using my binary cache to speed up (my hydra-CI passing)

nix-shell test.nix --option sandbox false --option substituters "https://cache.nixos.org http://221.4.35.244:8301" --option trusted-public-keys "221.4.35.244:3ehdeUIC5gWzY+I7iF3lrpmxOMyEZQbZlcjOmlOVpeo= cache.nixos.org-1:${yourtrustkey check nix.conf }"

[573]

@GTrunSec Using the cachix option (cachix use nsm-data-analysis)

and this modified shell.nix

# shell.nix
let
  jupyterLib = builtins.fetchGit {
    url = https://github.com/GTrunSec/jupyterWith;
    rev = "d9fe46bce29dcdb026807335f46cb2b8655dbf6b";
    ref = "current";
  };

  dscience = builtins.fetchGit {
    url = https://github.com/GTrunSec/Jupyter-data-science-environment;
    rev = "0194e5c71daa74e40b4cdaa8f8724989d372d6fb";
    ref = "master";
  };

  # Importing overlays from that path.
  overlays = [
    (import "${dscience}/overlay/python-overlay.nix")
    (import "${dscience}/overlay/package-overlay.nix")
  ];

  env = (import (jupyterLib + "/lib/directory.nix")) { inherit pkgs; };

  pkgs = (import "${dscience}/nix/nixpkgs.nix") { inherit overlays; config = { allowUnfree = true; allowBroken = true; }; };

  jupyter = import jupyterLib { pkgs = pkgs; };

  iPython = jupyter.kernels.iPythonWith {
    python3 = pkgs.callPackage "${dscience}/overlay/python-self-packages.nix" { inherit pkgs; };
    name = "Python-data-env";
    packages =
      import
        (pkgs.runCommand "shrinkpkgs"
          { } ''
          sed \
          -e "s!jsondiff\|geoip2\|qtconsole\|ipywidgets\|elasticsearch!!g" \
          -e "s!sympy\|nbdev\|fastai2\|fastai\|zat\|mlfinlab\|tqdm!!g" \
          -e "s!tensorflow\|patsy\|Keras\|dask\|pyfolio!!g" \
          "${dscience}/overlay/python-packages-list.nix" > "$out"
        ''
        ) { inherit pkgs; };
    ignoreCollisions = true;
  };

  jupyterEnvironment =
    jupyter.jupyterlabWith {
      kernels = [ iPython ];
      extraPackages = p: with p;[ python3Packages.torchBin ];
      extraJupyterPath = p: "${p.python3Packages.torchBin}/lib/python3.7/site-packages";
    };
in
jupyterEnvironment.env

I can nix-shell shell.nix --option sandbox false now and

import torch
torch.__version__
'1.6.0'

work.

Concerning jupyter lab running from the nix-shell still got some warnings though

[W 14:22:16.861 LabApp] Error loading server extension pytorch
Traceback (most recent call last):
File "/nix/store/yymb37m7bx1p8gzx332hhigx1aam3p6k-python3.7-notebook-6.0.3/lib/python3.7/site-packages/notebook/notebookapp.py", line 1670, in init_server_extensions
mod = importlib.import_module(modulename)
File "/nix/store/nnfpapxzrhib9gxpj26f0i5h0ww9rfwc-python3-3.7.7/lib/python3.7/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1006, in _gcd_import
File "", line 983, in _find_and_load
File "", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'pytorch'
[W 14:22:16.872 LabApp] Error loading server extension pytorch-bin
Traceback (most recent call last):
File "/nix/store/yymb37m7bx1p8gzx332hhigx1aam3p6k-python3.7-notebook-6.0.3/lib/python3.7/site-packages/notebook/notebookapp.py", line 1670, in init_server_extensions
mod = importlib.import_module(modulename)
File "/nix/store/nnfpapxzrhib9gxpj26f0i5h0ww9rfwc-python3-3.7.7/lib/python3.7/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1006, in _gcd_import
File "", line 983, in _find_and_load
File "", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'pytorch-bin'
[W 14:22:16.880 LabApp] JupyterLab server extension not enabled, manually loading...
[I 14:22:17.411 LabApp] JupyterLab extension loaded from /nix/store/p3ci42w5iw64zwm4lggg4mvyyfnb75ch-python3.7-jupyterlab-2.1.2/lib/python3.7/site-packages/jupyterlab
[I 14:22:17.413 LabApp] JupyterLab application directory is /nix/store/6q0dza9cq2vhb0makd7j8gd7dr6ff3kr-python3.7-jupyterlab-2.1.2/share/jupyter/lab

[GTrunSec]

sorry， I did not get any error from my test machine(virt-box nix-debian).

make sure you have loaded the correct kernel name which in your nix file.
also, I have pushed the binary cache to cachix. Now, you can use the cachix to download binary instead of by yourself(included haseklltorch and TensorFlow binary cache ).

[guaraqe]

Sorry for the delay @573 . As I mentioned above, JupyterWith delegates the Python kernel work to python3.withPackages, which creates a Python interpreter with suitable libraries. The problem is that Python needs all dependencies to be coherent between packages, and therefore, packages from two different Nixpkgs checkouts will not work together. This is made confusing by the fact that this error is completely silent.

However, there is a solution to the problem, which is to change instead the checkout of Nixpkgs that is being used. A way of doing this is using the method described here: https://github.com/tweag/jupyterWith#using-as-an-overlay and replace by the Nixpkgs checkout you are using. Besides some rebuilding, this should work.

[573]

@guaraqe never mind!
With your tip and @GTrunSec 's help I was able to come up with this minimal shell.nix.

let
  jupyterWithPath =
    builtins.fetchGit {
      url = https://github.com/GTrunSec/jupyterWith;
      rev = "d9fe46bce29dcdb026807335f46cb2b8655dbf6b";
      ref = "current";
    };

  # Importing overlays from that path.
  overlays = [
    # Necessary for Jupyter
    (import "${jupyterWithPath}/nix/python-overlay.nix")
    (import "${jupyterWithPath}/nix/overlay.nix")
  ];

  # Your Nixpkgs snapshot, with JupyterWith packages.
  # https://discourse.nixos.org/t/improving-nixos-data-science-infrastructure-ci-for-mkl-cuda/5074/43
  # https://github.com/tweag/jupyterWith/issues/124#issuecomment-692554738
  # curl -sL https://github.com/NixOS/nixpkgs/pull/96669.patch | head -n 1 | grep -o -E -e "[0-9a-f]{40}"
  pkgs = (
    import
      (
        fetchTarball
          "https://github.com/NixOS/nixpkgs/archive/6cce9c30786442f1fb2b6bd7c2c7c9a089e8648b.tar.gz"
      ) { inherit overlays; config = { allowUnfree = true; allowBroken = true; }; }
  );

  # From here, everything happens as in other examples.
  jupyter = pkgs.jupyterWith;

  iPython = jupyter.kernels.iPythonWith {
    # https://github.com/tweag/jupyterWith/issues/124#issuecomment-692554738
    # according to https://github.com/tweag/jupyterWith/issues/124#issuecomment-694109986 doesn't work
    # https://github.com/NixOS/nixpkgs/issues/65351
    #python3 = pkgs.python3.withPackages (_: [ ]);

    name = "github.com_tweag_jupyterWith_issues_124";

    packages = pkgs: with pkgs; [
      pytorch
      pytorch-bin
    ];

    ignoreCollisions = true;
  };

  jupyterEnvironment =
    jupyter.jupyterlabWith {
      kernels = [ iPython ];
    };
in
jupyterEnvironment.env

(notebook attached)

The (minor as the notebook seems to work for me) issue with ModuleNotFoundError: No module named 'pytorch' thus remains for IDK reasons.

[guaraqe]

You working example is essentially the same as the one using overlays, so I am glad it worked! Since this seems solved, I am closing the issue. Feel free to ask if you have further problems.

[573]

Glad it worked!

@guaraqe is there some way to get rid of the [W * LabApp] Error loading server extension pytorch[-bin] messages ?

[guaraqe]

The solution would probably to install the corresponding server extensions, even if I am not sure of what they are. If you want to try to do this, there are some hints here: https://github.com/tweag/jupyterWith#adding-packages-to-the-jupyter-path