Add parameter to ignore collisions possible ?

[573]

Hi,

this project is amazing, thanks for sharing it.

I wonder if I could add nix packages (pyls-mypy and sorts) when they cause collisions as in

collision between /nix/store/y04m7z8k5p2d725v4csw32md7w64j4h9-python3-3.7.6-env/bin/.chardetect-wrapped' and /nix/store/2ry9dqdgjzy8p699ri0sp1rq86bvmdah-python3-3.7.7-env/bin/.chardetect-wrapped'

I hoped something along the lines (snippet incomplete) of

let
  myMachnix =
    let
      mach-nix = import (builtins.fetchGit {
        url = "https://github.com/DavHau/mach-nix/";
        ref = "2.0.0";
      });
    in
    mach-nix.mkPython {
      disable_checks = true;
      requirements = ''
        notebook
        pandas
        numpy

        dropbox
        lockfile

        curlify
        openpyxl
        xlrd
        tqdm
        requests

        cython >= 0.23.5
        bokeh == 0.12.7
        flexx == 0.4.1
        normality == 0.6.1
        dataset == 0.8.0
        tornado
      '';
    };
  in
  {
    home.packages = with pkgs; [
      ((python3.withPackages (pkgs: with pkgs; [
        # for emacs, also https://nixos.wiki/wiki/Vim#Vim_as_a_Python_IDE
        python-language-server
        # the following plugins are optional, they provide type checking, import sorting and code formatting
        pyls-mypy # sind in nixpkgs, nicht in pypi
        pyls-isort
        pyls-black
      ])).override (args: { ignoreCollisions = true; })) # doesn't ignore collisions with mach-nix env though
      myMachnix
    ];
  }

would do. But it seems I had to add the override to myMachnix as well. I just don't know how to do that.

[DavHau]

Hey,
I'm happy you find mach-nix useful!

Basically mkPython of mach-nix just calls python.withPackages. Therefore the exact same overrides should work.
In your specific example that would be:
myMachnix.override (args: { ignoreCollisions = true; })

Though I cannot reproduce your collision. I couldn't find any package named "sorts"

If you find that any list of requirements produces a collision, then this is a potential bug in either mach-nix, nixpkgs, or the package itself. Please file an issue here in this case, and we will see how to proceed.

[573]

Thanks @DavHau,

indeed I get the collision error as well when adding the override as you wrote (snippet, list of requirements as above):

let ...
  in
  {
    home.packages = with pkgs; [
      ((python3.withPackages (pkgs: with pkgs; [
        # for emacs, also https://nixos.wiki/wiki/Vim#Vim_as_a_Python_IDE
        python-language-server
        # the following plugins are optional, they provide type checking, import sorting and code formatting
        pyls-mypy # sind in nixpkgs, nicht in pypi
        pyls-isort
        pyls-black
      ])).override (args: { ignoreCollisions = true; }))
      (myMachnix.override (args: { ignoreCollisions = true; }))
    ];
}

[DavHau]

Could you provide the requirements which cause the collision, so I have a better chance debugging it?

[573]

The ~/.config/nixpkgs/program/python/default.nix I am using in home.nix (imports = [ ./program/python ];), minimal example tested is:

{ # `git ls-remote https://github.com/nixos/nixpkgs-channels nixos-unstable`
  nixpkgs-rev ? "ddf87fb1baf8f5022281dad13fb318fa5c17a7c6"
, pkgsPath ? builtins.fetchTarball {
    name = "nixpkgs-${nixpkgs-rev}";
    url = "https://github.com/nixos/nixpkgs/archive/${nixpkgs-rev}.tar.gz";
  }
  , pkgs ? import pkgsPath {}
  , ...
}:
with import <nixpkgs>{};
let
  myMachnix =
    let
      mach-nix = import (builtins.fetchGit {
        url = "https://github.com/DavHau/mach-nix/";
        ref = "2.0.0";
      });
    in
    mach-nix.mkPython {
      disable_checks = true;
      requirements = ''
        notebook
      '';
    };
  in
  {
    home.packages = with pkgs; [
      ((python3.withPackages (pkgs: with pkgs; [
        # for emacs, also https://nixos.wiki/wiki/Vim#Vim_as_a_Python_IDE
        python-language-server
        # the following plugins are optional, they provide type checking, import sorting and code formatting
        pyls-mypy # sind in nixpkgs, nicht in pypi
        pyls-isort
        pyls-black
      ])).override (args: { ignoreCollisions = true; }))
      (myMachnix.override (args: { ignoreCollisions = true; }))
    ];
}

Getting:

collision between /nix/store/mvxpll6v3a11q5m1w5khsf77w7w98zr9-python3-3.7.6-env/bin/.easy_install-3.7-wrapped' and /nix/store/2ry9dqdgjzy8p699ri0sp1rq86bvmdah-python3-3.7.7-env/bin/.easy_install-3.7-wrapped'

[DavHau]

OK, I think I get the problem now. You are trying to mix two different python environments. The ignoreColisions only applies to each of the environments individually, but not for the combination of both. I guess the way you try to mix environments doesn't work in general. But there are other ways to do it.

First of all, it is not really necessary to manually mix python packages from nixpkgs into mach-nix. All python packages from nixpkgs are available to mach-nix already, just put them into the requirements of mach-nix like this:
(note: python-jsonrpc-server causes some strange bug which is fixed by switching the provider)

let
  pkgs = import <unstable>;
  myMachnix =
    let
      mach-nix = import (builtins.fetchGit {
        url = "https://github.com/DavHau/mach-nix/";
        ref = "2.0.0";
      });
    in
    mach-nix.mkPython {
      disable_checks = true;
      requirements = ''
        notebook
        pyls-mypy
        pyls-isort
        pyls-black
      '';
      providers = {
        # python-jsonrpc-server seems to cause a strange bug when installing from pypi.
        # We change its provider to nixpkgs
        python-jsonrpc-server = "nixpkgs";
      };
    };
in
myMachnix

---

If you really need to mix two environments, then you could use the lower level mach nix function machnix.machNix to generate pythonOverrides and a package selector for the mach-nix part, and then mix this together with some packages you take from nixpkgs:
(I don't really like this interface yet and will probably change it in an upcoming release)

let
  pkgs = import <unstable> {};
  machnix_src = builtins.fetchGit {
    url = "https://github.com/DavHau/mach-nix/";
    ref = "2.0.0";
  };
  machnix = import machnix_src;
  autoPatchelfHook = import "${machnix_src}/mach_nix/nix/auto_patchelf_hook.nix" {inherit (pkgs) fetchurl makeSetupHook writeText;};
  result = machnix.machNix {
    requirements = ''
      notebook
    '';
    python = pkgs.python3;
  };
  overrides_machnix = result.overrides pkgs.pythonManylinuxPackages.manylinux1 pkgs.autoPatchelfHook;
  my_python = pkgs.python3.override {
    packageOverrides = overrides_machnix;
  };
in
my_python.withPackages (ps: with ps; (result.select_pkgs ps) ++ [
  python-language-server
  pyls-mypy
  pyls-isort
  pyls-black
])

EDIT: Another bug prevents prevents this from working: #27

[573]

Some of the checks despite setting disable_checks = true; still are run and fail, how can I disable those as well, low level api ?
I used the first approach you describe.

[573]

I think I got your example now:

{ # `git ls-remote https://github.com/nixos/nixpkgs-channels nixos-unstable`
  nixpkgs-rev ? "ddf87fb1baf8f5022281dad13fb318fa5c17a7c6"
, pkgsPath ? builtins.fetchTarball {
    name = "nixpkgs-${nixpkgs-rev}";
    url = "https://github.com/nixos/nixpkgs/archive/${nixpkgs-rev}.tar.gz";
  }
  , pkgs ? import pkgsPath {}
  , ...
}:
with import <nixpkgs>{};
let
  myMachnix =
    let
      mach-nix = import (builtins.fetchGit {
        url = "https://github.com/DavHau/mach-nix/";
        ref = "2.0.0";
      });
    in
    mach-nix.mkPython {
      disable_checks = true;
      requirements = ''
        cffi
        aiohttp
        notebook
        # ... all my other reqs.

        # for emacs, also https://nixos.wiki/wiki/Vim#Vim_as_a_Python_IDE
        python-language-server

        # https://github.com/DavHau/mach-nix/issues/24
        pyls-mypy
        pyls-isort
        pyls-black
      '';
      providers = {
        # python-jsonrpc-server seems to cause a strange bug when installing from pypi.
        # We change its provider to nixpkgs
        python-jsonrpc-server = "nixpkgs";
      };
    };
    aiohttp = pkgs.python3Packages.aiohttp.overridePythonAttrs {
      doCheck = false;
      doInstallCheck = false;
    };
    cffi = pkgs.python3Packages.cffi.overridePythonAttrs {
      doCheck = false;
      doInstallCheck = false;
    };
  in
  {
    home.packages = with pkgs; [
      (myMachnix.override (args: {
        # see https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/interpreters/python/wrapper.nix
        ignoreCollisions = true;
        }))
    ];
}

is working no problems.