Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow specifying custom location for MKL #38

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

alyst
Copy link

@alyst alyst commented Mar 12, 2020

Allows specifying custom paths that would be used for MKL injection into sysimg.jl.
The paths default to the current ["@v#.#", "@"], but could be explicitly specified when calling enable_mkl_startup(), or via the JULIA_MKL_LOAD_PATH environment variable (so that it would be picked up when building sysimg upon Pkg.add(MKL)).

The motivation
Now that building Julia in ArchLinux is broken (due to http-parser/libgit2 incompatibilities, see JuliaLang/julia#34785), this PR allows using the official Julia binaries to build MKL-enabled ArchLinux package.
I install MKL into /usr/share/julia/site using this AUR script snippet:

...

build() {
  # rebuild sysimage with MKL support
  SITE_DEPOT_PATH=${srcdir}/julia-${pkgver}/share/julia/site
  rm -rf ${SITE_DEPOT_PATH}
  mkdir -p ${SITE_DEPOT_PATH}
  JULIA_LOAD_PATH="${SITE_DEPOT_PATH}:@stdlib" \
  JULIA_DEPOT_PATH=${SITE_DEPOT_PATH} \
  JULIA_MKL_LOAD_PATH="${SITE_DEPOT_PATH}" \
  ${srcdir}/julia-${pkgver}/bin/julia \
    -H ${srcdir}/julia-${pkgver}/bin \
    --startup-file=no \
    --project=${SITE_DEPOT_PATH} \
    ${srcdir}/build_sysimg_with_mkl.jl
}

package() {
  mkdir -p ${pkgdir}/usr/share/licenses/julia
  cp -r julia-${pkgver}/{bin,etc,include,lib,share} ${pkgdir}/usr/
  rm -rf ${pkgdir}/usr/share/julia/site/{artifacts,clones,logs,registries}
  sed "s!${srcdir}/julia-${pkgver}!/usr!" -i ${pkgdir}/usr/share/julia/base/sysimg.jl 
  install -Dm644 julia-${pkgver}/LICENSE.md \
      ${pkgdir}/usr/share/licenses/julia/LICENSE.md
}

build_sysimg_with_mkl.jl is just:

using Pkg
Pkg.add(PackageSpec(url="https://github.com/alyst/MKL.jl", rev="custom_pkg_location"))

@alyst
Copy link
Author

alyst commented Mar 12, 2020

Actually, it would be nicer to use JULIA_MKL_LOAD_PATH="@stdlib/../../site" -- no need to fix sysimg.jl afterwards (though I don't know whether that is relevant at all).
But at the moment Base.load_path_expand() treats anything after "@" that is not literally @stdlib as a name of some environment. Would it make sense fixing Base.load_path_expand()?

@KristofferC
Copy link
Contributor

I still don't really understand why this is needed. Could you elaborate on what is special with ArchLinux that requires this?

@alyst
Copy link
Author

alyst commented Mar 13, 2020

I still don't really understand why this is needed. Could you elaborate on what is special with ArchLinux that requires this?

Until recently, the official ArchLinux Julia package was working + I was able to build AUR package with MKL support (because e.g. TensorDecompositions.jl is 10x faster with it).
ATM the ArchLinux-official package and AUR ones are broken and, IIUC, the recommendation from Julia devs is to install JuliaLang-official Julia binaries in the home folder and patch users's sysimg.jl to use MKL.jl from user's depot. BTW, is that right?
I don't want to do it this way, because I would like to keep the version of Julia (and few other software packages like R) on my laptop, on the server and on the Docker/Charliecloud image in sync, and I would like to have a simple, reliable and automated way of doing that.
So far I was pretty happy with ArchLinux and its AUR system for managing custom software I use, so I would like to keep using it, instead of inventing some other approaches.

What I want is that Julia is installed into /usr, as well as MKL.jl (e.g. into share/julia/site), and this is done by the OS (ArchLinux) package manager.
Above I've posted an extract from AUR PKGBUILD that automates this process, so in the end I have a proper ArchLinux package with official Julia binaries and MKL support out-of-the-box.
(Well, now I realize that it's not over yet, and I have to solve the problem with artifacts installed somewhere else)

I hope now it's a bit more clear.

The alternative would be to fix the http-parser/libgit2 issue, so that it would be possible to build Julia (with MKL enabled at compile time) on ArchLinux with its current http-parser-2.9.3 package.

@aminya
Copy link
Contributor

aminya commented Mar 13, 2020

I would like to have a simple, reliable and automated way of doing that.

I recommend other Linux distributions (e.g. Ubuntu, Mint, Fedora, etc.) then.

@ViralBShah
Copy link
Contributor

Also related: #84

@ViralBShah ViralBShah marked this pull request as draft May 14, 2024 14:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants