Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Documentation clarification for installing MKL on Mac OS X #31

Closed
JulioBarros opened this issue Sep 12, 2017 · 39 comments
Closed

Documentation clarification for installing MKL on Mac OS X #31

JulioBarros opened this issue Sep 12, 2017 · 39 comments

Comments

@JulioBarros
Copy link

I stumbled around trying to install MKL and figured out setting DYLD_LIBRARY_PATH on MAC OS X doesn’t work as one might expect anymore because because of system integrity protection (SIP) http://osxdaily.com/2015/10/05/disable-rootless-system-integrity-protection-mac-os-x/

I instead created links from the MKL libs to an existing allowed directory and that seems to work. For example:
ln -s /opt/intel/lib* /usr/local/lib
ln -s /opt/intel/mkl/lib/* /usr/local/lib

This is mentioned the docs (thanks!) but wanted to point out that it may be necessary if you don't want to relax the SIP settings and I thought clearer/stronger suggestion would alert people that this is an option. Thanks again.

@blueberry
Copy link
Member

Added to the docs. Thanks.

@QQism
Copy link

QQism commented Sep 22, 2017

@JulioBarros Thanks, I think there is a typo in the first command which should have the forward slash / before *

ln -s /opt/intel/lib/* /usr/local/lib
ln -s /opt/intel/mkl/lib/* /usr/local/lib 

@blueberry blueberry pinned this issue Feb 21, 2019
@malloc82
Copy link

Hi, I just want to comment on that linking

/opt/intel/lib/*
/opt/intel/mkl/lib/* 

does not work for me on Mojave with MKL 2019.5.281. I have to link:

/opt/intel/compilers_and_libraries_2019.5.281/mac/mkl/lib/*
/opt/intel/compilers_and_libraries_2019.5.281/mac/compiler/lib/* 

Got the idea from this post (second problem).

@BorisVSchmid
Copy link

BorisVSchmid commented May 12, 2020

Couldn't get the above tricks to work anymore (on Mac OS Catalina 10.15.4, and Intel MKL 2020.1, and oracle-jdk installed through brew cask). Everything worked fine if I disabled SIP, so that DYLD_LIBRARY_PATH became visible to the repl, but nothing else seemed to work: had the relevant dylib files in usr/local/bin, had the files in /Library/Java/Extensions, etc, to no effect. Not really satisfactory, but spend more time than I wanted on trying to get this to work.

One thing to get things to work with SIP disabled after installing the Intel MKL is to add these three lines to your .zshenv:

# MKL Environment Variables:
export INTEL_COMPILERS_AND_LIBS=/opt/intel/compilers_and_libraries_2020/mac
source $INTEL_COMPILERS_AND_LIBS/mkl/bin/mklvars.sh intel64

That will set several variables, including DYLD_LIBRARY_PATH.

edit - should have said I tried "usr/local/lib" - was mucking around with the java binary later on, and confused "usr/local/lib" and "usr/local/bin" in the writeup.

@blueberry
Copy link
Member

Thank you. Important: dylib files should be in /usr/local/lib, and NOT in /usr/local/bin. (this might be a typo in your message, or you were using the wrong directory?)

@mars0i
Copy link

mars0i commented Jun 30, 2020

On MacOS 10.5.5 I also had to run

ln -s /opt/intel/compilers_and_libraries_2020.1.216/mac/compiler/lib/libiomp5.dylib /usr/local/lib

@jasonwaack
Copy link

jasonwaack commented Aug 18, 2020

Worked on Mac 10.14.6 ...

  1. Install Intel MKL

  2. Per BorisVSchmid disable SIP and add three lines to .zshenv

# MKL Environment Variables:
export INTEL_COMPILERS_AND_LIBS=/opt/intel/compilers_and_libraries_2020/mac
source $INTEL_COMPILERS_AND_LIBS/mkl/bin/mklvars.sh intel64
  1. Per mars0i add simlink to libiomp5.dylib in /usr/local/lib. NOTE: change path to reflect version of MKL installed.
ln -s /opt/intel/compilers_and_libraries_2020.2.258/mac/compiler/lib/libiomp5.dylib /usr/local/lib

@blueberry
Copy link
Member

Since 0.35.0 there is also the zero-install option:

https://dragan.rocks/articles/20/Clojure-Neanderthal-MKL-acceleration-out-of-the-box

@BorisVSchmid
Copy link

Looks great, but I might still be missing a step somewhere. Downloaded hello-world and tried to run it, and I get

  #_=>   (:require [uncomplicate.neanderthal
  #_=>              [core :refer :all]
  #_=>              [native :refer :all]]))
Syntax error compiling at (uncomplicate/neanderthal/native.clj:1:1).
No namespace: uncomplicate.neanderthal.internal.host.mkl```

No errors when starting up the repl or anything.

Of course, I came from a SIP-disabled Mac OSX install. Now deleted that install and re-enabled SIP. Maybe that mucked up something somewhere.

@blueberry
Copy link
Member

Do you use 0.35.0, and do you have this dependency enabled?

;;Optional. If bytedeco is not present, a system-wide MKL is used.
[org.bytedeco/mkl-platform-redist "2020.1-1.5.3"]]

BTW. Can you please run lein repl in terminal, and paste the complete output, so I can try to see why it fails on your machine?

@jasonwaack
Copy link

jasonwaack commented Aug 18, 2020

I could not get MKL jar to work on my Mac so I installed the full MLK as described above successfully. If I now simply add the jar dependency back...

[org.bytedeco/mkl-platform-redist "2020.1-1.5.3"] 

I get the following error in my REPL when I load a namespace requiring uncomplicate.neanderthal.native

amalgamation.core=> dyld: lazy symbol binding failed: Symbol not found: _mkl_serv_core_register_cleanup
  Referenced from: /opt/intel/compilers_and_libraries_2020.2.258/mac/mkl/lib/libmkl_intel_thread.dylib
  Expected in: flat namespace

dyld: Symbol not found: _mkl_serv_core_register_cleanup
  Referenced from: /opt/intel/compilers_and_libraries_2020.2.258/mac/mkl/lib/libmkl_intel_thread.dylib
  Expected in: flat namespace

Before I fully installed MLK and only added the jar dependency I got an error referencing the /private directory cache on my machine.

@blueberry
Copy link
Member

I only have access to an old macOS from 5 years ago, where everything works. Later Apple started throwing wrenches in the wheels, so each new macOS disables something. Any info with specific versions and combinations is appreciated. Unfortunately, I can't help much in straightening it completely.

@BorisVSchmid
Copy link

BorisVSchmid commented Aug 20, 2020

Hi Dragan,

Thanks for all your efforts - got it to work straight from the start on Windows 10, but I tried both on my own macbook air 2013 (MacOSX 10.15.6), and my gf's mac (idem, but with macOS 10.14.6, and as she doesn't muck about with clojure or MKLs, it was a clean mac as far as clojure/intel_MKL was concerned), and got the same error.

Here is on my mac:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Execution error (UnsatisfiedLinkError) at java.lang.ClassLoader$NativeLibrary/load0 (ClassLoader.java:-2).
/private/var/folders/5q/3bz1kqxd7030w_84_7f4p17m0000gn/T/libneanderthal-mkl-0.33.08576346321569376451.jnilib: dlopen(/private/var/folders/5q/3bz1kqxd7030w_84_7f4p17m0000gn/T/libneanderthal-mkl-0.33.08576346321569376451.jnilib, 1): Library not loaded: @rpath/libmkl_intel_lp64.dylib
  Referenced from: /private/var/folders/5q/3bz1kqxd7030w_84_7f4p17m0000gn/T/libneanderthal-mkl-0.33.08576346321569376451.jnilib
  Reason: image not found

user=>

user=> *e
#error {
 :cause "/private/var/folders/5q/3bz1kqxd7030w_84_7f4p17m0000gn/T/libneanderthal-mkl-0.33.08576346321569376451.jnilib: dlopen(/private/var/folders/5q/3bz1kqxd7030w_84_7f4p17m0000gn/T/libneanderthal-mkl-0.33.08576346321569376451.jnilib, 1): Library not loaded: @rpath/libmkl_intel_lp64.dylib\n  Referenced from: /private/var/folders/5q/3bz1kqxd7030w_84_7f4p17m0000gn/T/libneanderthal-mkl-0.33.08576346321569376451.jnilib\n  Reason: image not found"
 :via
 [{:type clojure.lang.Compiler$CompilerException
   :message "Syntax error macroexpanding at (mkl.clj:86:35)."
   :data #:clojure.error{:phase :execution, :line 86, :column 35, :source "mkl.clj"}
   :at [clojure.lang.Compiler$InvokeExpr eval "Compiler.java" 3707]}
  {:type java.lang.UnsatisfiedLinkError
   :message "/private/var/folders/5q/3bz1kqxd7030w_84_7f4p17m0000gn/T/libneanderthal-mkl-0.33.08576346321569376451.jnilib: dlopen(/private/var/folders/5q/3bz1kqxd7030w_84_7f4p17m0000gn/T/libneanderthal-mkl-0.33.08576346321569376451.jnilib, 1): Library not loaded: @rpath/libmkl_intel_lp64.dylib\n  Referenced from: /private/var/folders/5q/3bz1kqxd7030w_84_7f4p17m0000gn/T/libneanderthal-mkl-0.33.08576346321569376451.jnilib\n  Reason: image not found"
   :at [java.lang.ClassLoader$NativeLibrary load0 "ClassLoader.java" -2]}]
 :trace
 [[java.lang.ClassLoader$NativeLibrary load0 "ClassLoader.java" -2]
  [java.lang.ClassLoader$NativeLibrary load "ClassLoader.java" 2452]
  [java.lang.ClassLoader$NativeLibrary loadLibrary "ClassLoader.java" 2508]
  [java.lang.ClassLoader loadLibrary0 "ClassLoader.java" 2704]
  [java.lang.ClassLoader loadLibrary "ClassLoader.java" 2637]
  [java.lang.Runtime load0 "Runtime.java" 745]
  [java.lang.System load "System.java" 1871]
  [uncomplicate.neanderthal.internal.host.NarSystem loadLibrary "NarSystem.java" 48]
  [uncomplicate.neanderthal.internal.host.MKL <clinit> "MKL.java" 16]
  [uncomplicate.neanderthal.internal.host.mkl$create_stream_ars5 invokeStatic "mkl.clj" 83]
  [uncomplicate.neanderthal.internal.host.mkl$create_stream_ars5 invoke "mkl.clj" 81]
  [clojure.lang.AFn applyToHelper "AFn.java" 154]
  [clojure.lang.AFn applyTo "AFn.java" 144]
  [clojure.lang.Compiler$InvokeExpr eval "Compiler.java" 3702]
  [clojure.lang.Compiler$DefExpr eval "Compiler.java" 457]
  [clojure.lang.Compiler eval "Compiler.java" 7182]
  [clojure.lang.Compiler load "Compiler.java" 7636]
  [clojure.lang.RT loadResourceScript "RT.java" 381]
  [clojure.lang.RT loadResourceScript "RT.java" 372]
  [clojure.lang.RT load "RT.java" 459]
  [clojure.lang.RT load "RT.java" 424]
  [clojure.core$load$fn__6839 invoke "core.clj" 6126]
  [clojure.core$load invokeStatic "core.clj" 6125]
  [clojure.core$load doInvoke "core.clj" 6109]
  [clojure.lang.RestFn invoke "RestFn.java" 408]
  [clojure.core$load_one invokeStatic "core.clj" 5908]
  [clojure.core$load_one invoke "core.clj" 5903]
  [clojure.core$load_lib$fn__6780 invoke "core.clj" 5948]
  [clojure.core$load_lib invokeStatic "core.clj" 5947]
  [clojure.core$load_lib doInvoke "core.clj" 5928]
  [clojure.lang.RestFn applyTo "RestFn.java" 142]
  [clojure.core$apply invokeStatic "core.clj" 667]
  [clojure.core$load_libs invokeStatic "core.clj" 5989]
  [clojure.core$load_libs doInvoke "core.clj" 5969]
  [clojure.lang.RestFn applyTo "RestFn.java" 137]
  [clojure.core$apply invokeStatic "core.clj" 667]
  [clojure.core$require invokeStatic "core.clj" 6007]
  [clojure.core$require doInvoke "core.clj" 6007]
  [clojure.lang.RestFn invoke "RestFn.java" 436]
  [uncomplicate.neanderthal.native$eval6207$loading__6721__auto____6208 invoke "native.clj" 9]
  [uncomplicate.neanderthal.native$eval6207 invokeStatic "native.clj" 9]
  [uncomplicate.neanderthal.native$eval6207 invoke "native.clj" 9]
  [clojure.lang.Compiler eval "Compiler.java" 7177]
  [clojure.lang.Compiler eval "Compiler.java" 7166]
  [clojure.lang.Compiler load "Compiler.java" 7636]
  [clojure.lang.RT loadResourceScript "RT.java" 381]
  [clojure.lang.RT loadResourceScript "RT.java" 372]
  [clojure.lang.RT load "RT.java" 459]
  [clojure.lang.RT load "RT.java" 424]
  [clojure.core$load$fn__6839 invoke "core.clj" 6126]
  [clojure.core$load invokeStatic "core.clj" 6125]
  [clojure.core$load doInvoke "core.clj" 6109]
  [clojure.lang.RestFn invoke "RestFn.java" 408]
  [clojure.core$load_one invokeStatic "core.clj" 5908]
  [clojure.core$load_one invoke "core.clj" 5903]
  [clojure.core$load_lib$fn__6780 invoke "core.clj" 5948]
  [clojure.core$load_lib invokeStatic "core.clj" 5947]
  [clojure.core$load_lib doInvoke "core.clj" 5928]
  [clojure.lang.RestFn applyTo "RestFn.java" 142]
  [clojure.core$apply invokeStatic "core.clj" 667]
  [clojure.core$load_libs invokeStatic "core.clj" 5989]
  [clojure.core$load_libs doInvoke "core.clj" 5969]
  [clojure.lang.RestFn applyTo "RestFn.java" 137]
  [clojure.core$apply invokeStatic "core.clj" 667]
  [clojure.core$require invokeStatic "core.clj" 6007]
  [clojure.core$require doInvoke "core.clj" 6007]
  [clojure.lang.RestFn invoke "RestFn.java" 408]
  [hello_world.native$eval1589$loading__6721__auto____1590 invoke "native.clj" 9]
  [hello_world.native$eval1589 invokeStatic "native.clj" 9]
  [hello_world.native$eval1589 invoke "native.clj" 9]
  [clojure.lang.Compiler eval "Compiler.java" 7177]
  [clojure.lang.Compiler eval "Compiler.java" 7166]
  [clojure.lang.Compiler load "Compiler.java" 7636]
  [clojure.lang.Compiler loadFile "Compiler.java" 7574]
  [clojure.lang.RT$3 invoke "RT.java" 327]
  [user$eval1585 invokeStatic "form-init13553170854950281782.clj" 1]
  [user$eval1585 invoke "form-init13553170854950281782.clj" 1]
  [clojure.lang.Compiler eval "Compiler.java" 7177]
  [clojure.lang.Compiler eval "Compiler.java" 7132]
  [clojure.core$eval invokeStatic "core.clj" 3214]
  [clojure.core$eval invoke "core.clj" 3210]
  [nrepl.middleware.interruptible_eval$evaluate$fn__917 invoke "interruptible_eval.clj" 91]
  [clojure.main$repl$read_eval_print__9086$fn__9089 invoke "main.clj" 437]
  [clojure.main$repl$read_eval_print__9086 invoke "main.clj" 437]
  [clojure.main$repl$fn__9095 invoke "main.clj" 458]
  [clojure.main$repl invokeStatic "main.clj" 458]
  [clojure.main$repl doInvoke "main.clj" 368]
  [clojure.lang.RestFn invoke "RestFn.java" 1523]
  [nrepl.middleware.interruptible_eval$evaluate invokeStatic "interruptible_eval.clj" 84]
  [nrepl.middleware.interruptible_eval$evaluate invoke "interruptible_eval.clj" 56]
  [nrepl.middleware.interruptible_eval$interruptible_eval$fn__943$fn__947 invoke "interruptible_eval.clj" 155]
  [clojure.lang.AFn run "AFn.java" 22]
  [nrepl.middleware.session$session_exec$main_loop__1044$fn__1048 invoke "session.clj" 190]
  [nrepl.middleware.session$session_exec$main_loop__1044 invoke "session.clj" 189]
  [clojure.lang.AFn run "AFn.java" 22]
  [java.lang.Thread run "Thread.java" 832]]}
user=> 

Java version:

boris@Air hello-world % java -version
java version "14.0.1" 2020-04-14
Java(TM) SE Runtime Environment (build 14.0.1+7)
Java HotSpot(TM) 64-Bit Server VM (build 14.0.1+7, mixed mode, sharing)
boris@Air hello-world % 

the project.clj:

(defproject hello-world "0.35.0"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.10.1"]
                 [uncomplicate/neanderthal "0.35.0"]
                 ;;Optional. If bytedeco is not present, a system-wide MKL is used.
                 [org.bytedeco/mkl-platform-redist "2020.1-1.5.3"]]
  ;; If on Java 9+, you have to uncomment the following JVM option.
  :jvm-opts ^:replace ["--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED"])

And if I do a sudo find * | grep mkl, I see that mkl is located in the places you might expect it:

Users/boris/.m2/repository/org/bytedeco/mkl-platform/2020.1-1.5.3
Users/boris/.m2/repository/org/bytedeco/mkl-platform/2020.1-1.5.3/mkl-platform-2020.1-1.5.3.pom
Users/boris/.m2/repository/org/bytedeco/mkl-platform/2020.1-1.5.3/mkl-platform-2020.1-1.5.3.pom.sha1
Users/boris/.m2/repository/org/bytedeco/mkl-platform/2020.1-1.5.3/mkl-platform-2020.1-1.5.3.jar
Users/boris/.m2/repository/org/bytedeco/mkl-platform/2020.1-1.5.3/mkl-platform-2020.1-1.5.3.jar.sha1
Users/boris/.m2/repository/org/bytedeco/mkl-platform/2020.1-1.5.3/_remote.repositories
Users/boris/.m2/repository/org/bytedeco/mkl-platform-redist
Users/boris/.m2/repository/org/bytedeco/mkl-platform-redist/2020.1-1.5.3
Users/boris/.m2/repository/org/bytedeco/mkl-platform-redist/2020.1-1.5.3/mkl-platform-redist-2020.1-1.5.3.pom.sha1
Users/boris/.m2/repository/org/bytedeco/mkl-platform-redist/2020.1-1.5.3/mkl-platform-redist-2020.1-1.5.3.jar
Users/boris/.m2/repository/org/bytedeco/mkl-platform-redist/2020.1-1.5.3/_remote.repositories
Users/boris/.m2/repository/org/bytedeco/mkl-platform-redist/2020.1-1.5.3/mkl-platform-redist-2020.1-1.5.3.pom
Users/boris/.m2/repository/org/bytedeco/mkl-platform-redist/2020.1-1.5.3/mkl-platform-redist-2020.1-1.5.3.jar.sha1
Users/boris/.m2/repository/org/bytedeco/mkl
Users/boris/.m2/repository/org/bytedeco/mkl/2020.1-1.5.3
Users/boris/.m2/repository/org/bytedeco/mkl/2020.1-1.5.3/mkl-2020.1-1.5.3-windows-x86_64-redist.jar.sha1
Users/boris/.m2/repository/org/bytedeco/mkl/2020.1-1.5.3/mkl-2020.1-1.5.3-windows-x86.jar
Users/boris/.m2/repository/org/bytedeco/mkl/2020.1-1.5.3/mkl-2020.1-1.5.3-windows-x86_64-redist.jar
Users/boris/.m2/repository/org/bytedeco/mkl/2020.1-1.5.3/mkl-2020.1-1.5.3-macosx-x86_64.jar.sha1
Users/boris/.m2/repository/org/bytedeco/mkl/2020.1-1.5.3/mkl-2020.1-1.5.3-linux-x86-redist.jar
Users/boris/.m2/repository/org/bytedeco/mkl/2020.1-1.5.3/mkl-2020.1-1.5.3.pom
Users/boris/.m2/repository/org/bytedeco/mkl/2020.1-1.5.3/mkl-2020.1-1.5.3.jar.sha1
Users/boris/.m2/repository/org/bytedeco/mkl/2020.1-1.5.3/mkl-2020.1-1.5.3-windows-x86.jar.sha1
Users/boris/.m2/repository/org/bytedeco/mkl/2020.1-1.5.3/mkl-2020.1-1.5.3-macosx-x86_64-redist.jar
Users/boris/.m2/repository/org/bytedeco/mkl/2020.1-1.5.3/mkl-2020.1-1.5.3.jar
Users/boris/.m2/repository/org/bytedeco/mkl/2020.1-1.5.3/mkl-2020.1-1.5.3-linux-x86.jar.sha1
Users/boris/.m2/repository/org/bytedeco/mkl/2020.1-1.5.3/mkl-2020.1-1.5.3-linux-x86_64-redist.jar.sha1
Users/boris/.m2/repository/org/bytedeco/mkl/2020.1-1.5.3/mkl-2020.1-1.5.3-linux-x86_64.jar
Users/boris/.m2/repository/org/bytedeco/mkl/2020.1-1.5.3/mkl-2020.1-1.5.3-windows-x86_64.jar.sha1
Users/boris/.m2/repository/org/bytedeco/mkl/2020.1-1.5.3/mkl-2020.1-1.5.3-linux-x86_64-redist.jar
Users/boris/.m2/repository/org/bytedeco/mkl/2020.1-1.5.3/mkl-2020.1-1.5.3-windows-x86_64.jar
Users/boris/.m2/repository/org/bytedeco/mkl/2020.1-1.5.3/mkl-2020.1-1.5.3.pom.sha1
Users/boris/.m2/repository/org/bytedeco/mkl/2020.1-1.5.3/mkl-2020.1-1.5.3-windows-x86-redist.jar
Users/boris/.m2/repository/org/bytedeco/mkl/2020.1-1.5.3/_remote.repositories
Users/boris/.m2/repository/org/bytedeco/mkl/2020.1-1.5.3/mkl-2020.1-1.5.3-macosx-x86_64-redist.jar.sha1
Users/boris/.m2/repository/org/bytedeco/mkl/2020.1-1.5.3/mkl-2020.1-1.5.3-linux-x86.jar
Users/boris/.m2/repository/org/bytedeco/mkl/2020.1-1.5.3/mkl-2020.1-1.5.3-macosx-x86_64.jar
Users/boris/.m2/repository/org/bytedeco/mkl/2020.1-1.5.3/mkl-2020.1-1.5.3-windows-x86-redist.jar.sha1
Users/boris/.m2/repository/org/bytedeco/mkl/2020.1-1.5.3/mkl-2020.1-1.5.3-linux-x86_64.jar.sha1
Users/boris/.m2/repository/org/bytedeco/mkl/2020.1-1.5.3/mkl-2020.1-1.5.3-linux-x86-redist.jar.sha1
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64-redist.jar
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64-redist.jar/org
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64-redist.jar/org/bytedeco
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64-redist.jar/org/bytedeco/mkl
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64-redist.jar/org/bytedeco/mkl/macosx-x86_64
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64-redist.jar/org/bytedeco/mkl/macosx-x86_64/libmkl_rt.dylib
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64-redist.jar/org/bytedeco/mkl/macosx-x86_64/libmkl_vml_avx2.dylib
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64-redist.jar/org/bytedeco/mkl/macosx-x86_64/libmkl_vml_avx.dylib
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64-redist.jar/org/bytedeco/mkl/macosx-x86_64/libmkl_core.dylib
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64-redist.jar/org/bytedeco/mkl/macosx-x86_64/libmkl_vml_avx512.dylib
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64-redist.jar/org/bytedeco/mkl/macosx-x86_64/libmkl_avx.dylib
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64-redist.jar/org/bytedeco/mkl/macosx-x86_64/libiomp5.dylib
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64-redist.jar/org/bytedeco/mkl/macosx-x86_64/libmkl_mc3.dylib
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64-redist.jar/org/bytedeco/mkl/macosx-x86_64/libmkl_intel_lp64.dylib
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64-redist.jar/org/bytedeco/mkl/macosx-x86_64/libmkl_avx512.dylib
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64-redist.jar/org/bytedeco/mkl/macosx-x86_64/libmkl_vml_mc2.dylib
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64-redist.jar/org/bytedeco/mkl/macosx-x86_64/libmkl_intel_thread.dylib
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64-redist.jar/org/bytedeco/mkl/macosx-x86_64/libmkl_vml_mc3.dylib
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64-redist.jar/org/bytedeco/mkl/macosx-x86_64/libmkl_avx2.dylib
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64.jar
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64.jar/org
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64.jar/org/bytedeco
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64.jar/org/bytedeco/mkl
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64.jar/org/bytedeco/mkl/macosx-x86_64
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64.jar/org/bytedeco/mkl/macosx-x86_64/libmkl_rt.dylib
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64.jar/org/bytedeco/mkl/macosx-x86_64/libmkl_vml_avx2.dylib
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64.jar/org/bytedeco/mkl/macosx-x86_64/libjnimkl_rt.dylib
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64.jar/org/bytedeco/mkl/macosx-x86_64/libmkl_vml_avx.dylib
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64.jar/org/bytedeco/mkl/macosx-x86_64/libmkl_core.dylib
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64.jar/org/bytedeco/mkl/macosx-x86_64/libmkl_vml_avx512.dylib
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64.jar/org/bytedeco/mkl/macosx-x86_64/libiomp5.dylib
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64.jar/org/bytedeco/mkl/macosx-x86_64/libmkl_vml_mc2.dylib
Users/boris/.javacpp/cache/mkl-2020.1-1.5.3-macosx-x86_64.jar/org/bytedeco/mkl/macosx-x86_64/libmkl_vml_mc3.dylib```

@genovese
Copy link

I have the same problem on Mac OS X 10.14.5 running Java 13.0.1.
Used the hello world example project, as below, but added the mkl jar (and reflection warning).
Error comes when requiring neanderthal native from within the repl; repl trace and error output below.
The outcome looks similar to the previous comment.

Thanks for your help!

(defproject hello-world "0.35.0"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.10.1"]
                 [uncomplicate/neanderthal "0.35.0"]
                 ;;Optional. If bytedeco is not present, a system-wide MKL is used.
                 [org.bytedeco/mkl-platform-redist "2020.1-1.5.3"]]
  ;; If on Java 9+, you have to uncomment the following JVM option.
  :jvm-opts ^:replace ["--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED"]

  :global-vars {*warn-on-reflection* true
                *print-length* 128})

user=> (require '[uncomplicate.neanderthal.native :refer [dv dge]])
Reflection warning, uncomplicate/neanderthal/internal/host/buffer_block.clj:149:9 - call to method put on java.nio.ByteBuffer can't be resolved (argument types: long, double).
Reflection warning, uncomplicate/neanderthal/internal/host/buffer_block.clj:171:5 - call to method put on java.nio.ByteBuffer can't be resolved (argument types: long, long).
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Execution error (UnsatisfiedLinkError) at java.lang.ClassLoader$NativeLibrary/load0 (ClassLoader.java:-2).
/private/var/folders/r1/rmk22wkx4qx5vlbjslm8tjdh0000gp/T/libneanderthal-mkl-0.33.01688729730332291564.jnilib: dlopen(/private/var/folders/r1/rmk22wkx4qx5vlbjslm8tjdh0000gp/T/libneanderthal-mkl-0.33.01688729730332291564.jnilib, 1): Library not loaded: @rpath/libmkl_intel_lp64.dylib
Referenced from: /private/var/folders/r1/rmk22wkx4qx5vlbjslm8tjdh0000gp/T/libneanderthal-mkl-0.33.01688729730332291564.jnilib
Reason: image not found

user=> *e
#error {
:cause "/private/var/folders/r1/rmk22wkx4qx5vlbjslm8tjdh0000gp/T/libneanderthal-mkl-0.33.01688729730332291564.jnilib: dlopen(/private/var/folders/r1/rmk22wkx4qx5vlbjslm8tjdh0000gp/T/libneanderthal-mkl-0.33.01688729730332291564.jnilib, 1): Library not loaded: @rpath/libmkl_intel_lp64.dylib\n Referenced from: /private/var/folders/r1/rmk22wkx4qx5vlbjslm8tjdh0000gp/T/libneanderthal-mkl-0.33.01688729730332291564.jnilib\n Reason: image not found"
:via
[{:type clojure.lang.Compiler$CompilerException
:message "Syntax error macroexpanding at (mkl.clj:86:35)."
:data #:clojure.error{:phase :execution, :line 86, :column 35, :source "mkl.clj"}
:at [clojure.lang.Compiler$InvokeExpr eval "Compiler.java" 3707]}
{:type java.lang.UnsatisfiedLinkError
:message "/private/var/folders/r1/rmk22wkx4qx5vlbjslm8tjdh0000gp/T/libneanderthal-mkl-0.33.01688729730332291564.jnilib: dlopen(/private/var/folders/r1/rmk22wkx4qx5vlbjslm8tjdh0000gp/T/libneanderthal-mkl-0.33.01688729730332291564.jnilib, 1): Library not loaded: @rpath/libmkl_intel_lp64.dylib\n Referenced from: /private/var/folders/r1/rmk22wkx4qx5vlbjslm8tjdh0000gp/T/libneanderthal-mkl-0.33.01688729730332291564.jnilib\n Reason: image not found"
:at [java.lang.ClassLoader$NativeLibrary load0 "ClassLoader.java" -2]}]
:trace
[[java.lang.ClassLoader$NativeLibrary load0 "ClassLoader.java" -2]
[java.lang.ClassLoader$NativeLibrary load "ClassLoader.java" 2440]
[java.lang.ClassLoader$NativeLibrary loadLibrary "ClassLoader.java" 2497]
[java.lang.ClassLoader loadLibrary0 "ClassLoader.java" 2694]
[java.lang.ClassLoader loadLibrary "ClassLoader.java" 2627]
[java.lang.Runtime load0 "Runtime.java" 744]
[java.lang.System load "System.java" 1873]
[uncomplicate.neanderthal.internal.host.NarSystem loadLibrary "NarSystem.java" 48]
[uncomplicate.neanderthal.internal.host.MKL "MKL.java" 16]
[uncomplicate.neanderthal.internal.host.mkl$create_stream_ars5 invokeStatic "mkl.clj" 83]
[uncomplicate.neanderthal.internal.host.mkl$create_stream_ars5 invoke "mkl.clj" 81]
[clojure.lang.AFn applyToHelper "AFn.java" 154]
[clojure.lang.AFn applyTo "AFn.java" 144]
[clojure.lang.Compiler$InvokeExpr eval "Compiler.java" 3702]
[clojure.lang.Compiler$DefExpr eval "Compiler.java" 457]
[clojure.lang.Compiler eval "Compiler.java" 7182]
[clojure.lang.Compiler load "Compiler.java" 7636]
[clojure.lang.RT loadResourceScript "RT.java" 381]
[clojure.lang.RT loadResourceScript "RT.java" 372]
[clojure.lang.RT load "RT.java" 459]
[clojure.lang.RT load "RT.java" 424]
[clojure.core$load$fn__6839 invoke "core.clj" 6126]
[clojure.core$load invokeStatic "core.clj" 6125]
[clojure.core$load doInvoke "core.clj" 6109]
[clojure.lang.RestFn invoke "RestFn.java" 408]
[clojure.core$load_one invokeStatic "core.clj" 5908]
[clojure.core$load_one invoke "core.clj" 5903]
[clojure.core$load_lib$fn__6780 invoke "core.clj" 5948]
[clojure.core$load_lib invokeStatic "core.clj" 5947]
[clojure.core$load_lib doInvoke "core.clj" 5928]
[clojure.lang.RestFn applyTo "RestFn.java" 142]
[clojure.core$apply invokeStatic "core.clj" 667]
[clojure.core$load_libs invokeStatic "core.clj" 5989]
[clojure.core$load_libs doInvoke "core.clj" 5969]
[clojure.lang.RestFn applyTo "RestFn.java" 137]
[clojure.core$apply invokeStatic "core.clj" 667]
[clojure.core$require invokeStatic "core.clj" 6007]
[clojure.core$require doInvoke "core.clj" 6007]
[clojure.lang.RestFn invoke "RestFn.java" 436]
[uncomplicate.neanderthal.native$eval6207$loading__6721__auto____6208 invoke "native.clj" 9]
[uncomplicate.neanderthal.native$eval6207 invokeStatic "native.clj" 9]
[uncomplicate.neanderthal.native$eval6207 invoke "native.clj" 9]
[clojure.lang.Compiler eval "Compiler.java" 7177]
[clojure.lang.Compiler eval "Compiler.java" 7166]
[clojure.lang.Compiler load "Compiler.java" 7636]
[clojure.lang.RT loadResourceScript "RT.java" 381]
[clojure.lang.RT loadResourceScript "RT.java" 372]
[clojure.lang.RT load "RT.java" 459]
[clojure.lang.RT load "RT.java" 424]
[clojure.core$load$fn__6839 invoke "core.clj" 6126]
[clojure.core$load invokeStatic "core.clj" 6125]
[clojure.core$load doInvoke "core.clj" 6109]
[clojure.lang.RestFn invoke "RestFn.java" 408]
[clojure.core$load_one invokeStatic "core.clj" 5908]
[clojure.core$load_one invoke "core.clj" 5903]
[clojure.core$load_lib$fn__6780 invoke "core.clj" 5948]
[clojure.core$load_lib invokeStatic "core.clj" 5947]
[clojure.core$load_lib doInvoke "core.clj" 5928]
[clojure.lang.RestFn applyTo "RestFn.java" 142]
[clojure.core$apply invokeStatic "core.clj" 667]
[clojure.core$load_libs invokeStatic "core.clj" 5985]
[clojure.core$load_libs doInvoke "core.clj" 5969]
[clojure.lang.RestFn applyTo "RestFn.java" 137]
[clojure.core$apply invokeStatic "core.clj" 667]
[clojure.core$require invokeStatic "core.clj" 6007]
[clojure.core$require doInvoke "core.clj" 6007]
[clojure.lang.RestFn invoke "RestFn.java" 408]
[user$eval6201 invokeStatic "form-init11065104546226567643.clj" 1]
[user$eval6201 invoke "form-init11065104546226567643.clj" 1]
[clojure.lang.Compiler eval "Compiler.java" 7177]
[clojure.lang.Compiler eval "Compiler.java" 7132]
[clojure.core$eval invokeStatic "core.clj" 3214]
[clojure.core$eval invoke "core.clj" 3210]
[nrepl.middleware.interruptible_eval$evaluate$fn__921 invoke "interruptible_eval.clj" 91]
[clojure.main$repl$read_eval_print__9086$fn__9089 invoke "main.clj" 437]
[clojure.main$repl$read_eval_print__9086 invoke "main.clj" 437]
[clojure.main$repl$fn__9095 invoke "main.clj" 458]
[clojure.main$repl invokeStatic "main.clj" 458]
[clojure.main$repl doInvoke "main.clj" 368]
[clojure.lang.RestFn invoke "RestFn.java" 1523]
[nrepl.middleware.interruptible_eval$evaluate invokeStatic "interruptible_eval.clj" 84]
[nrepl.middleware.interruptible_eval$evaluate invoke "interruptible_eval.clj" 56]
[nrepl.middleware.interruptible_eval$interruptible_eval$fn__947$fn__951 invoke "interruptible_eval.clj" 155]
[clojure.lang.AFn run "AFn.java" 22]
[nrepl.middleware.session$session_exec$main_loop__1048$fn__1052 invoke "session.clj" 190]
[nrepl.middleware.session$session_exec$main_loop__1048 invoke "session.clj" 189]
[clojure.lang.AFn run "AFn.java" 22]
[java.lang.Thread run "Thread.java" 830]]}

@blueberry
Copy link
Member

blueberry commented Aug 21, 2020

@genovese Does it work when you remove the bytedeco dependency and just use the system-wide libraries? In case you wonder where to find these libraries, you can even rename the bytedeco jar to zip, unzip it, and then use the dylib files from there. You don't need to fully install MKL, you just need to add it to the DYLD_LIBRARY_PATH (or whatever is current for your particular version of macOS). Note: you'd need to configure SIP and whatever developer blocking mechanisms Apple activated in addition to it.

@genovese
Copy link

I did not yet install the library system wide because I expected the bytedeco package to handle it. I will now, however, and will report back. Thanks.

@blueberry
Copy link
Member

blueberry commented Aug 22, 2020

Bytedeco is under the same macOS restrictions as any other means of adding native libraries. Maybe the problem is that you didn't disable Apple's restrictions (SIP, etc.).

Can you please try just bytedeco with SIP enabled, before installing MKL system-wide?

@jasonwaack
Copy link

jasonwaack commented Aug 23, 2020

I already have full install of MKL working. As an experiment as suggested by blueberry I ...

  1. Uninstalled MKL (running installer and choosing uninstall)
  2. Removed simlink to libiomp5.dylib
  3. Commented out export of INTEL_COMPILERS_AND_LIBS and source of mklvars.sh in my .zshrc
  4. Confirmed I had SIP disabled
  5. Added bytedeco jar to my project.clj and executed lein deps
  6. Loaded file with requiring core and native from uncomplicate.neanderthal in repl
    And I got original error before I tried to install MKL...
Execution error (UnsatisfiedLinkError) at java.lang.ClassLoader$NativeLibrary/load0 (ClassLoader.java:-2).
/private/var/folders/bd/6lgx256d14g5xw99n44dbvh80000gn/T/libneanderthal-mkl-0.33.03764098863930245147.jnilib: dlopen(/private/var/folders/bd/6lgx256d14g5xw99n44dbvh80000gn/T/libneanderthal-mkl-0.33.03764098863930245147.jnilib, 1): Library not loaded: @rpath/libmkl_intel_lp64.dylib
  Referenced from: /private/var/folders/bd/6lgx256d14g5xw99n44dbvh80000gn/T/libneanderthal-mkl-0.33.03764098863930245147.jnilib
  Reason: image not found

Just out of curiosity I also tried executing (clojure.lang.RT/loadClassForName "org.bytedeco.mkl.global.mkl_rt") in the repl and the command succeeded. When I then try to load same file requiring neanderthal above I get ...

Syntax error compiling at (project.clj:1:1).
Unable to resolve symbol: defproject in this context

I will go back to full install of MKL that worked using steps I mentioned earlier in this thread.

NOTE: I am using Java 11 and have uncommented line :jvm-opts ^:replace [#_"--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED"] in my projectclj.

@blueberry
Copy link
Member

blueberry commented Aug 23, 2020

I've pushed new MKL loading implementation to github. Can you please clone the neanderthal repo, do only lein install, and change dependency to "0.36.0-SNAPSHOT" in your project? Does it load bytedeco now?

(Of course, both ways work on my machines, but I need someone of you to try whether it works on yours too).

@genovese
Copy link

genovese commented Aug 23, 2020 via email

@jasonwaack
Copy link

blueberry do we need dependency uncomplicate:commons:jar:0.12.0-SNAPSHOT for uncomplicate:neanderthal:jar:0.36.0-SNAPSHOT?

@blueberry
Copy link
Member

Yes.

@jasonwaack
Copy link

I lein installed both uncomplicate:commons:jar:0.12.0-SNAPSHOT and uncomplicate:neanderthal:jar:0.36.0-SNAPSHOT and added [uncomplicate/neanderthal "0.36.0-SNAPSHOT"]. Still got error in repl when loading hello-world.native

Execution error (UnsatisfiedLinkError) at java.lang.ClassLoader$NativeLibrary/load (ClassLoader.java:-2).
/private/var/folders/bd/6lgx256d14g5xw99n44dbvh80000gn/T/libneanderthal-mkl-0.33.02848579040618247445.jnilib: dlopen(/private/var/folders/bd/6lgx256d14g5xw99n44dbvh80000gn/T/libneanderthal-mkl-0.33.02848579040618247445.jnilib, 1): Library not loaded: @rpath/libmkl_intel_lp64.dylib
  Referenced from: /private/var/folders/bd/6lgx256d14g5xw99n44dbvh80000gn/T/libneanderthal-mkl-0.33.02848579040618247445.jnilib
  Reason: image not found

I will try again later when I have time.

@blueberry
Copy link
Member

If you can, please see the mkl.clj file, an the way I load the namespace. If possible, try to experiment with various ways of blocking the namespace loading until mkl_rt has been loaded. For example,
you could try to call some simple method in the MKL.java class, and, conditionally on its result, load the mkl namespace.

@Hendekagon
Copy link

Hendekagon commented Sep 7, 2020

I can confirm that a fresh install of MKL on a different laptop works if the contents of both /opt/intel/mkl/lib and /opt/intel/lib are linked to /usr/local/lib, and JAVA_HOME is set correctly. You don't need anything in addition to /usr/local/lib on your PATH or in java.library.path. However, if using IntelliJ ensure your project's JDK is set to the same one as your JAVA_HOME and confirm it works in a normal REPL in terminal with clj first.

@lccambiaghi
Copy link

Unfortunately I have to disagree with the above. I couldn't get Neanderthal to work with MKL on my MacOS with none of the suggestions in this issue. I always get either image not found or Symbol not found: _mkl_serv_core_register_cleanup as posted by other users.

@danieltanfh95
Copy link

Unfortunately I have to disagree with the above. I couldn't get Neanderthal to work with MKL on my MacOS with none of the suggestions in this issue. I always get either image not found or Symbol not found: _mkl_serv_core_register_cleanup as posted by other users.

same here

@blueberry
Copy link
Member

Are you sure you've disabled SIP?

@lccambiaghi
Copy link

Ah no that I haven't tried to be honest.

Do I understand correctly that Neanderthal as of today cannot be used without turning off SIP on MacOS?

@blueberry
Copy link
Member

I don't know since I don't have a recent mac. Some users say it can, some say it can't. It probably can, but you have to know macOS and Apple's countless roadblocks well enough to control them.

@lccambiaghi
Copy link

I see. Is there a Neanderthal docker image available? Sorry if off topic but maybe it is relevant for Mac users who can't get MKL to behave..

@Hendekagon
Copy link

(I'm using Mojave and do not have SIP disabled)

@Hendekagon
Copy link

Hendekagon commented Sep 27, 2020

I also have Neanderthal working on another machine with Catalina without SIP disabled - same setup as above - can only make it work using JDK 10.0.2

I don't think SIP is a factor here

@blueberry
Copy link
Member

Oh, wait a minute here. I am definitely positive that SIP is absolutely central to this issue.

The only thing that is debatable is whether there is a way to configure macos path without SIP. In your case it worked, but only with Java 10.0.2 (which is strange).

IMHO, for newbies, it is best to disable SIP and make MKL work first, and then, perhaps, explore how to make this configurable while re-enabling SIP.

(On my machine, I had to disable SIP, but I also need to use MKL in the compilation step, while the users don't need to compile anything, just make MKL dylibs visible to the JVM)

@lccambiaghi
Copy link

For whoever did not manage to install Neanderthal on MacOS, I used this docker image: https://github.com/ComputeSoftware/java-neanderthal-docker/blob/master/tools-deps/Dockerfile , to which I added:

RUN echo '{:deps { uncomplicate/neanderthal {:mvn/version "0.36.0"} nrepl/nrepl {:mvn/version "0.6.0"} org.clojure/tools.deps.alpha {:mvn/version "0.6.496"} cider/cider-nrepl {:mvn/version "0.25.3"}} :aliases {:nREPL {:extra-deps {nrepl/nrepl {:mvn/version "0.6.0"}}}}}' > deps.edn &&\
    echo '{:bind "0.0.0.0" :port 8888}' > .nrepl.edn &&\
    clj -Sforce < /dev/null >&0

CMD ["clj", "-R:nREPL", "-m", "nrepl.cmdline", "--middleware", "[cider.nrepl/cider-middleware]"]

Build the image, run it with port forwarding 8888 and connect with CIDER. Half an hour well spent.

@SolbiatiAlessandro
Copy link

Mac Catalina 10.15.6 having the same problem

user=> (use '[uncomplicate.neanderthal.native])
dyld: lazy symbol binding failed: Symbol not found: _mkl_serv_core_register_cleanup
  Referenced from: /opt/intel/compilers_and_libraries_2020.3.279/mac/mkl/lib/libmkl_intel_thread.dylib
  Expected in: flat namespace

Disabling SIP is not an option for me because of a bug with Mac recovery mode

On the other hand I don't think is SIP, cause I actually tried the hello-world example from this repo and in that case the import worked

user=> (use '[uncomplicate.neanderthal.native])
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
nil
user=> (def b (dge 3 2 [1 3 5 7 9 11]))
#'user/b

I examined the repo where the error repro but I couldn't get what's the difference. I am really happy @lccambiaghi made a docker image! That really saves my day

SolbiatiAlessandro added a commit to SolbiatiAlessandro/steamengine that referenced this issue Oct 19, 2020
@lccambiaghi
Copy link

@dragandj would you be interested in a PR with the Dockerfile and instructions on how to build, run and connect to be added to the README?

@blueberry
Copy link
Member

@lccambiaghi Sure, if you (or someone else) would accept the responsibility to keep it up to date with latest version of Neanderthal and answer potential questions, since I don't use docker nor have appropriate macOS to test it.

@lccambiaghi
Copy link

I added #101 .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests