mkl: init at 2019.0.117 #47182

bhipple · 2018-09-22T19:33:45Z

This packags the Intel Math Kernel library on x86-64 platforms, which is a
dependency for many data science and machine learning packages.

Upstream, Intel provides proprietary binary RPMs with a permissive
redistribution license. These have been repackaged in both Debian and Anaconda,
so we are not the first distribution to redistribute.

Motivation for this change

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

This packags the Intel Math Kernel library on x86-64 platforms, which is a dependency for many data science and machine learning packages. Upstream, Intel provides proprietary binary RPMs with a permissive redistribution license. These have been repackaged in both Debian and Anaconda, so we are not the first distribution to redistribute.

bhipple · 2018-09-22T19:38:56Z

@GrahamcOfBorg build mkl

GrahamcOfBorg · 2018-09-22T19:39:15Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: mkl

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-09-22T19:39:32Z

No attempt on x86_64-linux (full log)

The following builds were skipped because they don't evaluate on x86_64-linux: mkl

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-09-22T19:48:02Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: mkl

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-09-22T19:48:31Z

No attempt on x86_64-linux (full log)

The following builds were skipped because they don't evaluate on x86_64-linux: mkl

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.

bhipple · 2018-09-22T20:00:27Z

I think borg doesn't like it because it's an unfree package? The platform matches on NixOS.

orivej · 2018-09-22T20:00:34Z

pkgs/development/libraries/science/math/mkl/default.nix

+      threading models.
+    '';
+    homepage = https://software.intel.com/en-us/mkl;
+    license = licenses.issl;


We should probably use licenses.unfreeRedistributable to build this on Hydra, or extend isUnfree to support arbitrary unfree redistributable licenses.

I agree with

We should probably use licenses.unfreeRedistributable to build this on Hydra

but not with

or extend isUnfree to support arbitrary unfree redistributable licenses.

because then they would be probably be installed with allowUnfree = false;, which is not what users expect.

I meant adding an attribute redistributable = true alongside unfree = false; allowUnfree = false will not install such packages.

That's fine, sorry I misunderstood.

Actually, are we sure unfreeRedistributable is working as intended? I've started digging in, and it looks like Hydra doesn't actually build any of these things. I spot-checked 5-10 packages and didn't see any of them on the hydra nixpkgs jobset. For example, consider the package aliza, which is nowhere to be found on Hydra despite having an unfreeRedistributable license. Looking at the source code on hydra and the meta evaluator, it looks like there's no extra condition for free = false but redistributable.

There's another license, unfreeRedistributableFirmware, that is being built and distributed -- but the reason for that is that it declares itself as free = true.

At any rate, I think we can cleanup some of this logic and fix these issues by updating Hydra with an unfreePredicate filter that allows through redistributable packages. There's an initial WIP RFC here

bhipple · 2018-09-23T16:50:34Z

I like the idea of adding a redistribute boolean attribute to licenses for Hydra. Where is that checked? In the meantime I submitted #47230, which I think is a no-op cleanup of the existing logic (at least as far as I'm reading it; please correct if mistaken).

EDIT: I found it and will take a look at making the redistribute attribute for distribution: https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/release.nix#L18-L19

GrahamcOfBorg · 2018-09-23T16:51:15Z

No attempt on x86_64-linux (full log)

The following builds were skipped because they don't evaluate on x86_64-linux: mkl

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-09-23T16:54:43Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: mkl

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.

bhipple · 2018-09-26T01:13:46Z

I opened a separate PR to start taking a look at unfreeRedistributable and some potential license refactorings, but in the meantime I don't think it's much of a problem if Hydra does not build and distribute this. After all, the only thing the "build" is doing is transmuting one binary package into another.

GrahamcOfBorg · 2018-09-26T01:26:34Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: mkl

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-09-26T01:27:01Z

No attempt on x86_64-linux (full log)

The following builds were skipped because they don't evaluate on x86_64-linux: mkl

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.

xeji · 2018-09-26T07:24:36Z

After all, the only thing the "build" is doing is transmuting one binary package into another.

I suggest to set preferLocalBuild = true; to disable Hydra builds even if Hydra would build packages with an unfree-redistributable license. It's >800M download after all, and the "build" operations are really trivial. It makes no sense to put it in the binary cache.

orivej · 2018-09-26T19:03:16Z

It's >800M download after all, and the "build" operations are really trivial. It makes no sense to put it in the binary cache.

I don't know which is better, but there is a benefit of having it in the binary cache in saving user's disk space: if they build it themselves, they'll store both the source archives and the unpacked content.

Cache efficiency can be improved by moving the extracted content into a fixed output derivation (e.g. by replacing fetchurl with fetchzip with a custom postFetch script, as is common in fonts) and symlinking the content into the final openmp-dependent derivation.

xeji · 2018-09-26T19:41:59Z

Cache efficiency can be improved by moving the extracted content into a fixed output derivation (e.g. by replacing fetchurl with fetchzip with a custom postFetch script, as is common in fonts) and symlinking the content into the final openmp-dependent derivation.

Good idea! Couldn't we do that locally to avoid duplication in the local store without involving the binary cache?

orivej · 2018-09-26T20:30:55Z

Good idea! Couldn't we do that locally to avoid duplication in the local store without involving the binary cache?

My point was that Hydra would cache the fixed-output derivation once per version (rather than once per stdenv/openmp rebuild), but yes, this will also be space-efficient with preferLocalBuild = true;.

bhipple · 2018-09-27T04:34:14Z

It's >800M download after all, and the "build" operations are really trivial. It makes no sense to put it in the binary cache.

Aside from the benefit @orivej mentioned with the double-space use, there's a second benefit for this particular use case: Intel distributes the src package as a massive tar of all the architecture binaries, and we're filtering it to a subset of what we need -- so the final binary package is actually smaller than the starting "source" package (though still huge):

λ brh nixpkgs →  du -sh $(nix-build . -A mkl.src)
836M    /nix/store/xlwgac8h9s1rpqjp3ycwzza2bnzm61yv-l_mkl_2019.0.117.tgz

λ brh nixpkgs →  du -sh $(nix-build . -A mkl)
626M    /nix/store/p2qhmh8fn94p8fwy24d46y7fl5lcyh8k-mkl-2019.0.117

I'd prefer to omit preferLocalBuild for that reason, plus I'd like my own machines to share the binary with each other, plus it's irrelevant (for now) for Hydra. The idea with the fixed output fetch for the initial src tarball seems nice; I've never looked at those and have been meaning to investigate how they work for things like TexLive. Is there some documentation I can read up on them? It looks to me like that fetchzip package still depends on stdenv and would be rebuilt on a change?

smaret · 2018-09-27T14:46:18Z

I was able to build this expression on Darwin with the following changes:

  src = fetchurl {
    url = "http://registrationcenter-download.intel.com/akdlm/irc_nas/tec/13565/m_mkl_${version}.dmg";
    sha256 = "1f1jppac7vqwn00hkws0p4njx38ajh0n25bsjyb5d7jcacwfvm02";
  };

  buildPhase = ''for f in Contents/Resources/pkg/*.tgz; do tar xzvf $f; done'' ;

 installPhase = ''
     mkdir -p $out/lib
     cp -r compilers_and_libraries_${version}/mac/mkl/include $out/
     cp -r compilers_and_libraries_${version}/mac/mkl/lib/* $out/lib/
     cp -r compilers_and_libraries_${version}/licensing/mkl/en/license.txt $out/lib/
  '';

undmg also needs to be added to the buildInputs.

Perhaps the package could be modified to support both Linux and Darwin? I'm not familiar enough with nix expressions to do it myself, unfortunately.

bhipple · 2018-10-03T02:55:40Z

I tossed in Darwin support as well, using your commands. I can confirm this still works on Linux, but I don't have access to a Darwin machine to test; would you mind giving it a try to make sure I didn't miss something in the conditionals?

I also swapped this to use rpmextract (which is a tiny script that does the command I was piping to cpio) and added both licenses for the redistributable (even though it's currently a no-op).

Regarding openmp, we could omit it entirely, which would simplify the expression and reduce the dependencies. Not everything that needs mkl needs a section that needs openmp, so we could have libraries/applications themselves add to LD_LIBRARY_PATH with wrapProgram. Is that preferable? That said, since the need of openmp is actually in mkl itself, it seems sensible to export it and have it "just work" without the caller having to know that.

GrahamcOfBorg · 2018-10-03T03:06:29Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: mkl

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-10-03T03:07:09Z

No attempt on x86_64-linux (full log)

The following builds were skipped because they don't evaluate on x86_64-linux: mkl

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.

smaret · 2018-10-03T08:08:54Z

I can confirm this still works on Linux, but I don't have access to a Darwin machine to test; would you mind giving it a try to make sure I didn't miss something in the conditionals?

Done, it works fine on Darwin. Thanks!

orivej · 2018-10-03T12:12:06Z

Is there some documentation I can read up on them?

There is https://nixos.org/nix/manual/#fixed-output-drvs

It looks to me like that fetchzip package still depends on stdenv and would be rebuilt on a change?

Derivations that specify outputHash are not rebuilt when their dependencies change.

bhipple · 2018-10-04T02:25:14Z

pkgs/development/libraries/science/math/mkl/default.nix

+  outputHashAlgo = "sha256";
+  outputHashMode = "recursive";
+  outputHash = if stdenvNoCC.isDarwin
+    then "0000000000000000000000000000000000000000000000000000"


@smaret can you try building this again and let me know what the fixed-output hash is for Mac?

Here it is: "1224dln7n8px1rk8biiggf77wjhxh8mzw0hd8zlyjm8i6j8w7i12"

bhipple · 2018-10-04T02:28:43Z

That was easy! Since the majority of mkl doesn't actually need libomp and applications that use those portions of mkl are already expected to manipulate LD_LIBRARY_PATH [1], I think we can just make this entire thing a fixed-output derivation and save a ton of build time + space. This seems a lot cleaner to me; let me know what you guys think.

[1] For example, see: https://github.com/oxfordcontrol/osqp/blob/master/docs/get_started/linear_system_solvers.rst#install-with-anaconda

GrahamcOfBorg · 2018-10-04T02:40:49Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: mkl

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-10-04T02:41:11Z

No attempt on x86_64-linux (full log)

The following builds were skipped because they don't evaluate on x86_64-linux: mkl

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-10-04T13:33:11Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: mkl

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-10-04T13:33:44Z

No attempt on x86_64-linux (full log)

The following builds were skipped because they don't evaluate on x86_64-linux: mkl

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.

bhipple · 2018-10-15T20:07:10Z

CC @xeji @orivej @matthewbauer anyone have any further review feedback? I think this is much polished from the initial cut and should be good to go now, but as always am interested if there are ways to improve it.

matthewbauer · 2018-10-15T20:17:53Z

Looks alright to me!

GrahamcOfBorg added 8.has: package (new) 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 labels Sep 22, 2018

orivej reviewed Sep 22, 2018

View reviewed changes

bhipple force-pushed the add/mkl branch from e13ffe8 to d7c1d04 Compare September 26, 2018 01:12

Add Darwin as a supported platform

ba02a2c

GrahamcOfBorg added 10.rebuild-darwin: 1-10 and removed 10.rebuild-darwin: 0 labels Oct 3, 2018

bhipple commented Oct 4, 2018

View reviewed changes

bhipple mentioned this pull request Oct 4, 2018

Docs: update fixed output hashes NixOS/nix#2468

Merged

Convert entire mkl package to a fixed-output derivation

0b82df8

bhipple force-pushed the add/mkl branch from 434637d to 0b82df8 Compare October 4, 2018 13:16

markuskowa approved these changes Oct 13, 2018

View reviewed changes

matthewbauer merged commit 0b298d5 into NixOS:master Oct 15, 2018

bhipple mentioned this pull request Dec 19, 2018

mkl: include Intel's libiomp.so in the MKL RPM unpack #52543

Merged

10 tasks

bhipple deleted the add/mkl branch February 27, 2019 23:04

mkl: init at 2019.0.117 #47182

mkl: init at 2019.0.117 #47182

Conversation

bhipple commented Sep 22, 2018 • edited

Motivation for this change

Things done

bhipple commented Sep 22, 2018

GrahamcOfBorg commented Sep 22, 2018

GrahamcOfBorg commented Sep 22, 2018

GrahamcOfBorg commented Sep 22, 2018

GrahamcOfBorg commented Sep 22, 2018

bhipple commented Sep 22, 2018

orivej Sep 22, 2018 • edited

Choose a reason for hiding this comment

xeji Sep 22, 2018

Choose a reason for hiding this comment

orivej Sep 23, 2018

Choose a reason for hiding this comment

xeji Sep 23, 2018

Choose a reason for hiding this comment

bhipple Sep 23, 2018 • edited

Choose a reason for hiding this comment

bhipple commented Sep 23, 2018 • edited

GrahamcOfBorg commented Sep 23, 2018

GrahamcOfBorg commented Sep 23, 2018

bhipple commented Sep 26, 2018

GrahamcOfBorg commented Sep 26, 2018

GrahamcOfBorg commented Sep 26, 2018

xeji commented Sep 26, 2018

orivej commented Sep 26, 2018

xeji commented Sep 26, 2018

orivej commented Sep 26, 2018

bhipple commented Sep 27, 2018 • edited

smaret commented Sep 27, 2018 • edited

bhipple commented Oct 3, 2018 • edited

GrahamcOfBorg commented Oct 3, 2018

GrahamcOfBorg commented Oct 3, 2018

smaret commented Oct 3, 2018

orivej commented Oct 3, 2018

bhipple Oct 4, 2018

Choose a reason for hiding this comment

smaret Oct 4, 2018

Choose a reason for hiding this comment

bhipple commented Oct 4, 2018

GrahamcOfBorg commented Oct 4, 2018

GrahamcOfBorg commented Oct 4, 2018

GrahamcOfBorg commented Oct 4, 2018

GrahamcOfBorg commented Oct 4, 2018

bhipple commented Oct 15, 2018

matthewbauer commented Oct 15, 2018

bhipple commented Sep 22, 2018 •

edited

orivej Sep 22, 2018 •

edited

bhipple Sep 23, 2018 •

edited

bhipple commented Sep 23, 2018 •

edited

bhipple commented Sep 27, 2018 •

edited

smaret commented Sep 27, 2018 •

edited

bhipple commented Oct 3, 2018 •

edited