juice does not compile on recent OS X

[bpr]

macOS High Sierra version 10.3.6

Ran cargo build --release in juice-examples dir as per instructions. Last few lines of error look like

          ld: library not found for -lcudnn
          clang: error: linker command failed with exit code 1 (use -v to see invocation)

I thought that juice could compile and run without a GPU; is that wrong?

[drahnr]

Default feature includes cuda, --no-default-features --features=native should work

[bpr]

Same error when building with --no-default-features --features=native.

EDIT:
Actually, same error if I issued the cargo build command in the juice dir. In the juice-examples dir, the build fails with

          ld: library not found for -lopenblas
          clang: error: linker command failed with exit code 1 (use -v to see invocation)

even though I have openblas installed

bpr@macbook (master) juice-examples: brew info openblas
openblas: stable 0.3.6 (bottled), HEAD [keg-only]
Optimized BLAS library
https://www.openblas.net/
/usr/local/Cellar/openblas/0.3.6_1 (21 files, 120.3MB)
  Poured from bottle on 2019-05-30 at 11:12:31
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/openblas.rb
==> Dependencies
Required: gcc ✔
==> Options
--HEAD
	Install HEAD version
==> Caveats
openblas is keg-only, which means it was not symlinked into /usr/local,
because macOS provides BLAS and LAPACK in the Accelerate framework.

For compilers to find openblas you may need to set:
  export LDFLAGS="-L/usr/local/opt/openblas/lib"
  export CPPFLAGS="-I/usr/local/opt/openblas/include"

For pkg-config to find openblas you may need to set:
  export PKG_CONFIG_PATH="/usr/local/opt/openblas/lib/pkgconfig"

==> Analytics
install: 27,388 (30 days), 102,890 (90 days), 328,507 (365 days)
install_on_request: 1,900 (30 days), 8,435 (90 days), 33,465 (365 days)
build_error: 0 (30 days)

and I set both the LDFLAGS and CPPFLAGS as mentioned in the brew info

[drahnr]

I need the rustc version, cargo version, brew version openblas version to dig into this

[bpr]

bpr@macbook (master) juice-examples: rustc --version
rustc 1.36.0 (a53f9df32 2019-07-03)
bpr@macbook (master) juice-examples: cargo --version
cargo 1.36.0 (c4fcfb725 2019-05-15)
bpr@macbook (master) juice-examples: brew --version
Homebrew 2.1.7
Homebrew/homebrew-core (git revision 0f55ed; last commit 2019-07-18)
Homebrew/homebrew-cask (git revision 1adfb; last commit 2019-07-18)
bpr@macbook (master) juice-examples: brew info openblas
openblas: stable 0.3.6 (bottled), HEAD [keg-only]
Optimized BLAS library
https://www.openblas.net/
/usr/local/Cellar/openblas/0.3.6_1 (21 files, 120.3MB)
  Poured from bottle on 2019-05-30 at 11:12:31
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/openblas.rb

[pheki]

I managed to get it to compile by adding this to .cargo/config:

[target.x86_64-apple-darwin]
rustflags = ["-C", "link-args=-L/usr/local/opt/openblas/lib"]

On the other side, examples will not run with openblas anyway...

$ ../target/release/juice-examples mnist linear --batch-size 10
Right now, you really need cuda! Not all features are available for all backends and as such, this one -as of now - only works with cuda.
thread 'main' panicked at 'explicit panic', juice-examples/src/main.rs:189:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

Want me to open a PR?

[drahnr]

@pheki that is correct, but tbh, I think this should be part of the rust-blas build.rs https://github.com/spearow/rust-blas/blob/master/build.rs and not part of juice. It also looks like setting BLAS_LIB_DIR should suffice to make it work without any additional changes.

Since my primary machine is Fedora and I only have very dated mac os x machines available, testing is somewhat limited (primary testing is always done on Fedora)

Could you potentially give the above a go on Mac OS X? (I never got around to testing @bpr 's issue, sorry for that @bpr )

Is /usr/local/opt/openblas the default location for libs or is it system specific thing?
Is that location consistent across Mac OS X?
Your openblas version is installed via brew I guess?
Is openblas considered a framework?

Depending on the answers, adding the library path if the TARGET tripled contains Mac OS X via

if let Some(triplet) = std::env::var("TARGET") {
    if triplet == "x86_64-apple-darwin" {
        println!("rustc-link-search=native=/usr/local/opt/openblas/lib");
    }
}

https://doc.rust-lang.org/cargo/reference/environment-variables.html
https://doc.rust-lang.org/cargo/reference/build-scripts.html

[pheki]

@drahnr

Your openblas version is installed via brew I guess?

Yes, installed via homebrew, which I'd say is the de facto standard package manager for MacOS.

Is /usr/local/opt/openblas the default location for libs or is it system specific thing?

It's the default location for homebrew installed packages (I'll explain more below)

Is that location consistent across Mac OS X?

Yes.

Is openblas considered a framework?

No, because it is not bundled as a Framework that seems follows a very specific directory structure.

I haven't been able to find the exact directories that are searched for libraries in MacOS, but from experience I can tell:
/usr/lib <- lots of native MacOS libraries and symlinks
/Library/Frameworks
/System/Library/Frameworks
/usr/local/lib <- homebrew will symlink all of its libraries here
/usr/local/Frameworks <- homebrew symlinks frameworks here (for example qt)

brew installs all of its packages into /usr/local/Cellar/<package-name>/<version> and symlinks binaries into /usr/local/bin, libraries into /usr/local/lib, includes (.h) into /usr/local/includes, and so on...

It will also symlink /usr/local/Cellar/<package-name>/<last-version> into /usr/local/opt/<package-name>, so its always the current version.

Example:

$ ls -l /usr/local/opt/ | grep blas
lrwxr-xr-x  1 pheki  admin  24 Sep 17 17:39 openblas -> ../Cellar/openblas/0.3.7
$ ls /usr/local/Cellar/openblas/0.3.7/
Changelog.txt		INSTALL_RECEIPT.json	LICENSE			README.md		include			lib

It seems that the system Accelerate framework contains libblas (the MacOS fs is by default not case-sensitive):

$ ls /System/Library/Frameworks/Accelerate.framework/Versions/Current/Frameworks/vecLib.framework/Versions/Current | grep .dylib
libBLAS.dylib
libBNNS.dylib
libLAPACK.dylib
libLinearAlgebra.dylib
libQuadrature.dylib
libSparse.dylib
libSparseBLAS.dylib
libvDSP.dylib
libvMisc.dylib

And that's why homebrew will not symlink (as it would override) openblas to /usr/local/lib...

---

Setting BLAS_LIB_DIR is enough, but I believe that adding it to the the build script would be better (I've already tested it and can open a PR).

An alternative would be to instruct users to run brew link --force openblas, which would then force homebrew to symlink openblas libraries to /usr/local/lib. I find it bad because it would then override the native libBLAS, which could have unintended consequences.

What do you think?

[drahnr]

If blas is already part of the accelerate framework, I guess it would make sense to link to that by default, granted it is compatible with openblas.

In case the compatibiity is not given, instructing the user to do brew link --force openblas, though I am little wary of side effects.

I agree that usability with native only needs to be improved, usually that's the first step in checking juice out.

I'd very much appreciate a PR!