stack install hmatrix is ugly #1463

Open
ndmitchell opened this Issue Dec 2, 2015 · 13 comments

Comments

Projects
None yet
8 participants
@ndmitchell
Contributor

ndmitchell commented Dec 2, 2015

Currently installing hmatrix with stack is difficult. The advice is to download some files then do:

stack install hmatrix --flag hmatrix:openblas --extra-lib-dirs=C:\...\OpenBLAS\lib --extra-include-dir=C:\...\OpenBLAS\include

Alas, that doesn't work because extra-include-dir is not a valid flag anymore. But either way, it would be nice if it was simpler and didn't require any download or complex messing around.

@mboes

This comment has been minimized.

Show comment
Hide comment
@mboes

mboes Dec 2, 2015

Contributor

This category of use case is pretty much what my colleague @YPares's @tweag work on Nix support for Stack is meant to help. The idea is: leverage an existing lightweight system to install system packages (such as OpenBLAS) with project-local scope (so no global install and then pointing to Stack the exact location required). See PR #1285.

It's a cross platform solution, so should work where Docker does not (stack --docker is another solution to your problem I think). But I'm not sure yet just how good Nix's support for Windows really is at the moment. So in your particular case it might not help. Yves and I would be curious to get feedback from Windows users such as yourself on actual applications.

Contributor

mboes commented Dec 2, 2015

This category of use case is pretty much what my colleague @YPares's @tweag work on Nix support for Stack is meant to help. The idea is: leverage an existing lightweight system to install system packages (such as OpenBLAS) with project-local scope (so no global install and then pointing to Stack the exact location required). See PR #1285.

It's a cross platform solution, so should work where Docker does not (stack --docker is another solution to your problem I think). But I'm not sure yet just how good Nix's support for Windows really is at the moment. So in your particular case it might not help. Yves and I would be curious to get feedback from Windows users such as yourself on actual applications.

@YPares

This comment has been minimized.

Show comment
Hide comment
@YPares

YPares Dec 2, 2015

Collaborator

@ndmitchell It's actually --extra-include-dirs, you forgot an s.

Collaborator

YPares commented Dec 2, 2015

@ndmitchell It's actually --extra-include-dirs, you forgot an s.

@ndmitchell

This comment has been minimized.

Show comment
Hide comment
@ndmitchell

ndmitchell Dec 2, 2015

Contributor

@YPares - thanks - that's pulled from the hmatrix website, and in this case, I think the command line parser should be smart enough to default if there is only one prefix.

@mboes / @YPares - what would this look like as someone who wants to use a package the depends on hmatrix? Would I have to do anything, or would hmatrix have to tweak their stack.yaml to add Nix info? My guess is that Nix on Windows is not that great (it's a weird intersection of people), but only way to be sure is try it and find out.

Contributor

ndmitchell commented Dec 2, 2015

@YPares - thanks - that's pulled from the hmatrix website, and in this case, I think the command line parser should be smart enough to default if there is only one prefix.

@mboes / @YPares - what would this look like as someone who wants to use a package the depends on hmatrix? Would I have to do anything, or would hmatrix have to tweak their stack.yaml to add Nix info? My guess is that Nix on Windows is not that great (it's a weird intersection of people), but only way to be sure is try it and find out.

@YPares

This comment has been minimized.

Show comment
Hide comment
@YPares

YPares Dec 3, 2015

Collaborator

The Nix support conforms to the existing Stack model. That is, if you want to build a project, it'll do its magic and transparently deal with all dependencies (haskell or system) for you. It doesn't deal with libraries or apps installed using the global stack config. And owing to the fact that projects are non-compositional, if you're building your own project, you'll have to copy the relevant snippets (e.g. the nix: section in the upstream projects if any) in your own stack.yaml. That's the best we can do without extending the .cabal format proper.

So if you want to build an app that uses hmatrix, if the author of that app added a nix: section to the stack.yaml, stack build should just work (no need to install OpenBLAS manually). If you have a project of your own that depends on hmatrix, you can add such a section to your stack.yaml so that others can build your project without hassle. If you stack install using the global config (ie not in the scope of any project), then you're out of luck: create a stack.yaml first or add the section to your ~/.stack/config.yaml.

Collaborator

YPares commented Dec 3, 2015

The Nix support conforms to the existing Stack model. That is, if you want to build a project, it'll do its magic and transparently deal with all dependencies (haskell or system) for you. It doesn't deal with libraries or apps installed using the global stack config. And owing to the fact that projects are non-compositional, if you're building your own project, you'll have to copy the relevant snippets (e.g. the nix: section in the upstream projects if any) in your own stack.yaml. That's the best we can do without extending the .cabal format proper.

So if you want to build an app that uses hmatrix, if the author of that app added a nix: section to the stack.yaml, stack build should just work (no need to install OpenBLAS manually). If you have a project of your own that depends on hmatrix, you can add such a section to your stack.yaml so that others can build your project without hassle. If you stack install using the global config (ie not in the scope of any project), then you're out of luck: create a stack.yaml first or add the section to your ~/.stack/config.yaml.

@ndmitchell

This comment has been minimized.

Show comment
Hide comment
@ndmitchell

ndmitchell Dec 3, 2015

Contributor

That's a little unfortunate. If I am using hmatrix, as a transitive dependency (e.g. I'm really using hnn), then I don't want to care about hmatrix at all - it's entirely an implementation detail of a library I care about. Couldn't Stack/Stackage suck the Nix bits out of the stack.yaml if the hmatrix author included it? Or maybe have some extra metadata somewhere? You can put custom metadata in a Cabal project I think (something like beginning with X or some other prefix) which might work too.

Until stack install "just works" for projects depending on hmatrix there will remain a non-zero cost to making use of hmatrix, which is a bit sad.

Contributor

ndmitchell commented Dec 3, 2015

That's a little unfortunate. If I am using hmatrix, as a transitive dependency (e.g. I'm really using hnn), then I don't want to care about hmatrix at all - it's entirely an implementation detail of a library I care about. Couldn't Stack/Stackage suck the Nix bits out of the stack.yaml if the hmatrix author included it? Or maybe have some extra metadata somewhere? You can put custom metadata in a Cabal project I think (something like beginning with X or some other prefix) which might work too.

Until stack install "just works" for projects depending on hmatrix there will remain a non-zero cost to making use of hmatrix, which is a bit sad.

@YPares YPares closed this Dec 3, 2015

@YPares YPares reopened this Dec 3, 2015

@YPares

This comment has been minimized.

Show comment
Hide comment
@YPares

YPares Dec 3, 2015

Collaborator

I agree. That's just the limit of the model that consists in pushing more and more metadata into the stack.yaml, which is for project metadata, not package metadata. If you want to make ​packages​ self contained, then as you say, the right place to push the metadata is in the .cabal I think, and to ideally have the Cabal library deal with it (it just takes invoking nix-shell during evaluation of Setup.lhs, which would transparently make the required system libraries available).

Collaborator

YPares commented Dec 3, 2015

I agree. That's just the limit of the model that consists in pushing more and more metadata into the stack.yaml, which is for project metadata, not package metadata. If you want to make ​packages​ self contained, then as you say, the right place to push the metadata is in the .cabal I think, and to ideally have the Cabal library deal with it (it just takes invoking nix-shell during evaluation of Setup.lhs, which would transparently make the required system libraries available).

@ndmitchell

This comment has been minimized.

Show comment
Hide comment
@ndmitchell

ndmitchell Dec 3, 2015

Contributor

Maybe if a custom Setup.hs detected an environment variable STACK and then wrote out some information in a predictable file location (perhaps given by the env var), then Stack would know what needed to be done?

Contributor

ndmitchell commented Dec 3, 2015

Maybe if a custom Setup.hs detected an environment variable STACK and then wrote out some information in a predictable file location (perhaps given by the env var), then Stack would know what needed to be done?

@ocramz

This comment has been minimized.

Show comment
Hide comment
@ocramz

ocramz Dec 3, 2015

IIUC, the .cabal file can't accept relative paths; I mention this because I'm working on a package that has to build an FFI dependency and link the dynlibs from GHC afterwards. I use the "ugly" workaround of specifying --extra-lib-dirs and --extra-include-dirs in the stack build command using a combination of shell script and makefile. There has to be a platform-independent way to solve this..

ocramz commented Dec 3, 2015

IIUC, the .cabal file can't accept relative paths; I mention this because I'm working on a package that has to build an FFI dependency and link the dynlibs from GHC afterwards. I use the "ugly" workaround of specifying --extra-lib-dirs and --extra-include-dirs in the stack build command using a combination of shell script and makefile. There has to be a platform-independent way to solve this..

@hsenag

This comment has been minimized.

Show comment
Hide comment
@hsenag

hsenag Dec 9, 2015

Isn't "pkgconfig" the way for cabal files to specify these dependencies?

hsenag commented Dec 9, 2015

Isn't "pkgconfig" the way for cabal files to specify these dependencies?

@mboes

This comment has been minimized.

Show comment
Hide comment
@mboes

mboes Dec 10, 2015

Contributor

Yes and no. pkgconfig names don't always correspond to the distro package names. Which in any case change from distro to distro.

Contributor

mboes commented Dec 10, 2015

Yes and no. pkgconfig names don't always correspond to the distro package names. Which in any case change from distro to distro.

@peti

This comment has been minimized.

Show comment
Hide comment
@peti

peti Dec 10, 2015

Contributor

Distros could provide a (system-wide) config file that maps the pkgconfig names used on Hackage to the package name that's appropriate for that particular distro. That is what cabal2nix does internally, too.

Contributor

peti commented Dec 10, 2015

Distros could provide a (system-wide) config file that maps the pkgconfig names used on Hackage to the package name that's appropriate for that particular distro. That is what cabal2nix does internally, too.

@sindikat

This comment has been minimized.

Show comment
Hide comment
@sindikat

sindikat May 12, 2016

For whom it may be useful:

I use NixOS and wanted to install SubHask, which requires HMatrix to be exactly 0.16.1.5. This is how my stack.yaml looks like (with some packages omitted for brevity):

resolver: lts-5.13
packages:
- '.'
extra-deps:
- subhask-0.1.1.0
- hmatrix-0.16.1.5
flags: {}
extra-package-dbs: []
# extra-include-dirs: [/path/to/dir]
extra-lib-dirs:
- /nix/store/mlpy5chf205z48nl2g089g6y7a6g32q0-gfortran-4.9.3/lib/
nix:
  enable: true
  packages:
  - blas
  - liblapack
  - gfortran

For whom it may be useful:

I use NixOS and wanted to install SubHask, which requires HMatrix to be exactly 0.16.1.5. This is how my stack.yaml looks like (with some packages omitted for brevity):

resolver: lts-5.13
packages:
- '.'
extra-deps:
- subhask-0.1.1.0
- hmatrix-0.16.1.5
flags: {}
extra-package-dbs: []
# extra-include-dirs: [/path/to/dir]
extra-lib-dirs:
- /nix/store/mlpy5chf205z48nl2g089g6y7a6g32q0-gfortran-4.9.3/lib/
nix:
  enable: true
  packages:
  - blas
  - liblapack
  - gfortran
@mgsloan

This comment has been minimized.

Show comment
Hide comment
@mgsloan

mgsloan Jul 26, 2016

Collaborator

#2122 is also somewhat relevant. It would allow stack install hmatrix to use the stack.yaml from the tarball.

Collaborator

mgsloan commented Jul 26, 2016

#2122 is also somewhat relevant. It would allow stack install hmatrix to use the stack.yaml from the tarball.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment