WIP: shell.nix that provides all the dependencies

[wavewave]

To make hasktorch buildable simply.

$ nix-shell shell.nix
$ cabal new-build hasktorch

Note: this is not yet finalized since I disabled mkl and adjusted C++11 ABI flag. I would like to have some discussion about them.

[junjihashimoto]

-optc-D_GLIBCXX_USE_CXX11_ABI=1 is a default option of gcc.
So we can just remove it to use CXX11_ABI.

[junjihashimoto]

Official page of pytorch provides libtorch of CXX11ABI.
So It ’s a good time to change.

[wavewave]

Thanks! now I removed them.

[junjihashimoto]

@wavewave
Thx!
I see NixOS/nixpkgs#65041.
I want to add nix-build on CircleCI.
Could you try it?

[wavewave]

@junjihashimoto Yes. First, I am planning to fully nixify hasktorch and libtorch-ffi (as individual nix package), so it can be easier to be integrated.

[wavewave]

I enabled mkl now but i cannot figure out linking against mklml though, so still commented out mklml linking in libtorch-ffi. Maybe the pytorch packaging in nixpkgs does not yet incorporate mkl-dnn?

[stites]

Currently tied up and can't get too in depth, but @wavewave can you check if nix's mkl pulls in mklml/mkldnn? It might need to be added separately. If that is the case you could attempt to just use the older libtorch folder (which just pulls down the zip)? It will have all the binaries, but might not get hooked up correctly - verifiable by running hasktorch tests.

[stites]

Oh, also! PyTorch actually generates a caffe python module - could you check those outputs as well?

[tscholak]

@wavewave cool! I’ll try your shell now.
I’ve started to package up the hasktorch projects, see https://github.com/tscholak/hasktorch/blob/nix/nix/shared.nix

[wavewave]

This PR is subsumed by #164 which packages up hasktorch as a nix package.
shell.nix is for hasktorch developer (preparing for dependency needed for hasktorch build) and user-shell.nix is for hasktorch user (hasktorch is a visible package to ghc).

[wavewave]

@tscholak oh. the nix packaging was there too. cool. thanks!

[wavewave]

close this PR, and let's unify the effort in #164.

[wavewave]

@stites As for caffe, I found that the following python modules are packaged:

$ ls /nix/store/qlqyc5jykmcc0rab0mg7q1pzmnkxkc26-python3.7-pytorch-1.2.0/lib/python3.7/site-packages/
caffe2  torch  torch-1.2.0.dist-info

and its C-dependencies libcaffe2* files are also present.

$ ls /nix/store/nzg8m241q9yymv8b1r5f4g7aii16is92-python3.7-pytorch-1.2.0-dev/lib/
libc10.so  libcaffe2_detectron_ops.so  libcaffe2_module_test_dynamic.so  libcaffe2_observers.so  libshm.so  libtorch_python.so  libtorch.so

[wavewave]

@stites As for mkl, basic mkl libraries are packaged but not mklml, mkldnn stuff:

$ ls /nix/store/q9x08zm24hf4qyd2sdgcs8kilxj212ad-mkl-2019.3.199/lib
libiomp5.a      libiompstubs5.a       libmkl_avx512.so  libmkl_intel_ilp64.so   libmkl_mc.so          libmkl_vml_avx512_mic.so  libmkl_vml_def.so  license.txt
libiomp5.dbg    libiompstubs5.so      libmkl_avx.so     libmkl_intel_lp64.so    libmkl_rt.so          libmkl_vml_avx512.so      libmkl_vml_mc2.so  locale
libiomp5_db.so  libmkl_avx2.so        libmkl_core.so    libmkl_intel_thread.so  libmkl_sequential.so  libmkl_vml_avx.so         libmkl_vml_mc3.so
libiomp5.so     libmkl_avx512_mic.so  libmkl_def.so     libmkl_mc3.so           libmkl_vml_avx2.so    libmkl_vml_cmpt.so        libmkl_vml_mc.so

[tscholak]

@wavewave reading through https://intel.github.io/mkl-dnn/index.html... I don’t think mkldnn is part of mkl at all.