Cannot locate artifact 'libknet8' for aarch64-linux-gnu-libgfortran5-cxx11-libstdcxx29-julia_version+1.7.1 in Docker Container on Apple Silicon

[manuEbg]

Hello,
I am trying to get Knet working in a docker container on my apple silicon Mac.
however the following problem occurred

(@v1.7) pkg> add Knet
  Installing known registries into `~/.julia`
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
   Installed IrrationalConstants ─ v0.1.1
   Installed JpegTurbo_jll ─────── v2.1.0+0
   Installed Adapt ─────────────── v3.3.2
   Installed ColorTypes ────────── v0.11.0
   Installed Preferences ───────── v1.2.3
   Installed OffsetArrays ──────── v1.10.8
   Installed JLD2 ──────────────── v0.4.17
   Installed ImageCore ─────────── v0.9.3
   Installed Zstd_jll ──────────── v1.5.0+0
   Installed Libtiff_jll ───────── v4.3.0+0
   Installed FixedPointNumbers ─── v0.8.4
   Installed TensorCore ────────── v0.1.1
   Installed ImageMagick ───────── v1.2.1
   Installed GPUCompiler ───────── v0.13.10
   Installed SpecialFunctions ──── v1.8.1
   Installed NNlib ─────────────── v0.7.31
   Installed TimerOutputs ──────── v0.5.13
   Installed AbstractFFTs ──────── v1.0.1
   Installed LLVMExtra_jll ─────── v0.0.13+0
   Installed NaNMath ───────────── v0.3.6
   Installed ColorVectorSpace ──── v0.9.8
   Installed OrderedCollections ── v1.4.1
   Installed AutoGrad ──────────── v1.2.4
   Installed JLLWrappers ───────── v1.3.0
   Installed TranscodingStreams ── v0.9.6
   Installed ChainRulesCore ────── v1.11.3
   Installed CEnum ─────────────── v0.4.1
   Installed BFloat16s ─────────── v0.2.0
   Installed Reexport ──────────── v1.2.2
   Installed Random123 ─────────── v1.4.2
   Installed Graphics ──────────── v1.1.1
   Installed GPUArrays ─────────── v8.1.3
   Installed FileIO ────────────── v1.12.0
   Installed RandomNumbers ─────── v1.5.3
   Installed ImageMagick_jll ───── v6.9.12+0
   Installed Requires ──────────── v1.2.0
   Installed Colors ────────────── v0.12.8
   Installed MappedArrays ──────── v0.4.1
   Installed StackViews ────────── v0.1.1
   Installed DataStructures ────── v0.18.11
   Installed PaddedViews ───────── v0.5.11
   Installed OpenSpecFun_jll ───── v0.5.5+0
   Installed Compat ────────────── v3.41.0
   Installed ExprTools ─────────── v0.1.6
   Installed MacroTools ────────── v0.5.9
   Installed InverseFunctions ──── v0.1.2
   Installed LogExpFunctions ───── v0.3.6
   Installed libpng_jll ────────── v1.6.38+0
   Installed MosaicViews ───────── v0.3.3
   Installed ChangesOfVariables ── v0.1.2
   Installed DocStringExtensions ─ v0.8.6
   Installed LLVM ──────────────── v4.7.0
   Installed Knet ──────────────── v1.4.9
   Installed CUDA ──────────────── v3.6.2
  Downloaded artifact: JpegTurbo
  Downloaded artifact: Zstd
  Downloaded artifact: Libtiff
  Downloaded artifact: LLVMExtra
  Downloaded artifact: ImageMagick
  Downloaded artifact: OpenSpecFun
  Downloaded artifact: libpng
    Updating `~/.julia/environments/v1.7/Project.toml`
  [1902f260] + Knet v1.4.9
    Updating `~/.julia/environments/v1.7/Manifest.toml`
  [621f4979] + AbstractFFTs v1.0.1
  [79e6a3ab] + Adapt v3.3.2
  [6710c13c] + AutoGrad v1.2.4
  [ab4f0b2a] + BFloat16s v0.2.0
  [fa961155] + CEnum v0.4.1
  [052768ef] + CUDA v3.6.2
  [d360d2e6] + ChainRulesCore v1.11.3
  [9e997f8a] + ChangesOfVariables v0.1.2
  [3da002f7] + ColorTypes v0.11.0
  [c3611d14] + ColorVectorSpace v0.9.8
  [5ae59095] + Colors v0.12.8
  [34da2185] + Compat v3.41.0
  [864edb3b] + DataStructures v0.18.11
  [ffbed154] + DocStringExtensions v0.8.6
  [e2ba6199] + ExprTools v0.1.6
  [5789e2e9] + FileIO v1.12.0
  [53c48c17] + FixedPointNumbers v0.8.4
  [0c68f7d7] + GPUArrays v8.1.3
  [61eb1bfa] + GPUCompiler v0.13.10
  [a2bd30eb] + Graphics v1.1.1
  [a09fc81d] + ImageCore v0.9.3
  [6218d12a] + ImageMagick v1.2.1
  [3587e190] + InverseFunctions v0.1.2
  [92d709cd] + IrrationalConstants v0.1.1
  [033835bb] + JLD2 v0.4.17
  [692b3bcd] + JLLWrappers v1.3.0
  [1902f260] + Knet v1.4.9
  [929cbde3] + LLVM v4.7.0
  [2ab3a3ac] + LogExpFunctions v0.3.6
  [1914dd2f] + MacroTools v0.5.9
  [dbb5928d] + MappedArrays v0.4.1
  [e94cdb99] + MosaicViews v0.3.3
  [872c559c] + NNlib v0.7.31
  [77ba4419] + NaNMath v0.3.6
  [6fe1bfb0] + OffsetArrays v1.10.8
  [bac558e1] + OrderedCollections v1.4.1
  [5432bcbf] + PaddedViews v0.5.11
  [21216c6a] + Preferences v1.2.3
  [74087812] + Random123 v1.4.2
  [e6cf234a] + RandomNumbers v1.5.3
  [189a3867] + Reexport v1.2.2
  [ae029012] + Requires v1.2.0
  [276daf66] + SpecialFunctions v1.8.1
  [cae243ae] + StackViews v0.1.1
  [62fd8b95] + TensorCore v0.1.1
  [a759f4b9] + TimerOutputs v0.5.13
  [3bb67fe8] + TranscodingStreams v0.9.6
  [c73af94c] + ImageMagick_jll v6.9.12+0
  [aacddb02] + JpegTurbo_jll v2.1.0+0
  [dad2f222] + LLVMExtra_jll v0.0.13+0
  [89763e89] + Libtiff_jll v4.3.0+0
  [efe28fd5] + OpenSpecFun_jll v0.5.5+0
  [3161d3a3] + Zstd_jll v1.5.0+0
  [b53b4c65] + libpng_jll v1.6.38+0
  [0dad84c5] + ArgTools
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [8bb1440f] + DelimitedFiles
  [8ba89e20] + Distributed
  [f43a241f] + Downloads
  [b77e0a4c] + InteractiveUtils
  [4af54fe1] + LazyArtifacts
  [b27032c2] + LibCURL
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [ca575930] + NetworkOptions
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [1a1011a3] + SharedArrays
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics
  [fa267f1f] + TOML
  [a4e569a6] + Tar
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [e66e0078] + CompilerSupportLibraries_jll
  [deac9b47] + LibCURL_jll
  [29816b5a] + LibSSH2_jll
  [c8ffd9c3] + MbedTLS_jll
  [14a3606d] + MozillaCACerts_jll
  [4536629a] + OpenBLAS_jll
  [05823500] + OpenLibm_jll
  [83775a58] + Zlib_jll
  [8e850b90] + libblastrampoline_jll
  [8e850ede] + nghttp2_jll
  [3f19e933] + p7zip_jll
    Building Random123 → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/0e8b146557ad1c6deb1367655e052276690e71a3/build.log`
Precompiling project...
  ✗ Knet
  60 dependencies successfully precompiled in 101 seconds
  1 dependency errored. To see a full report either run `import Pkg; Pkg.precompile()` or load the package

(@v1.7) pkg> precompile
Precompiling project...
  ✗ Knet
  0 dependencies successfully precompiled in 3 seconds (60 already precompiled)

ERROR: The following 1 direct dependency failed to precompile:

Knet [1902f260-5fb4-5aff-8c31-6271790ab950]

Failed to precompile Knet [1902f260-5fb4-5aff-8c31-6271790ab950] to /root/.julia/compiled/v1.7/Knet/jl_nZfCMi.
ERROR: LoadError: Cannot locate artifact 'libknet8' for aarch64-linux-gnu-libgfortran5-cxx11-libstdcxx29-julia_version+1.7.1 in '/root/.julia/packages/Knet/RCkV0/Artifacts.toml'
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:33
  [2] artifact_slash_lookup(name::String, artifact_dict::Dict{String, Any}, artifacts_toml::String, platform::Base.BinaryPlatforms.Platform)
    @ Artifacts /usr/local/julia/share/julia/stdlib/v1.7/Artifacts/src/Artifacts.jl:608
  [3] var"@artifact_str"(__source__::LineNumberNode, __module__::Module, name::Any, platform::Nothing)
    @ Artifacts /usr/local/julia/share/julia/stdlib/v1.7/Artifacts/src/Artifacts.jl:678
  [4] var"@artifact_str"(__source__::LineNumberNode, __module__::Module, name::Any)
    @ Artifacts /usr/local/julia/share/julia/stdlib/v1.7/Artifacts/src/Artifacts.jl:641
  [5] include(mod::Module, _path::String)
    @ Base ./Base.jl:418
  [6] include(x::String)
    @ Knet ~/.julia/packages/Knet/RCkV0/src/Knet.jl:1
  [7] top-level scope
    @ ~/.julia/packages/Knet/RCkV0/src/Knet.jl:17
  [8] include
    @ ./Base.jl:418 [inlined]
  [9] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
    @ Base ./loading.jl:1318
 [10] top-level scope
    @ none:1
 [11] eval
    @ ./boot.jl:373 [inlined]
 [12] eval(x::Expr)
    @ Base.MainInclude ./client.jl:453
 [13] top-level scope
    @ none:1
in expression starting at /root/.julia/packages/Knet/RCkV0/src/libknet8/LibKnet8.jl:7
in expression starting at /root/.julia/packages/Knet/RCkV0/src/libknet8/LibKnet8.jl:3
in expression starting at /root/.julia/packages/Knet/RCkV0/src/Knet.jl:1

(@v1.7) pkg> 

I am using the official Julia Docker image with Julia v1.7.1
I am not using CUDA
On My Ubuntu machine it runs perfectly fine.

I am not sure if this is related to Knet, Docker or Apple...
But I hope someone could help me find a Solution:)

[denizyuret]

Dear @manuEbg: The libknet8 library is only needed if you want to use the GPU. As far as I know this is not possible with the Macs at the moment as CUDA.jl no longer supports them. Do you just want a clean install for cpu use only?

[manuEbg]

Hey @denizyuret,
thank you very much for you're reply!
Yes I only want to get started with julia and Knet and wanna do some basic Learning Stuff inside a clean docker-container.
No need for GPU-Acceleration at this point:)
So is there a way to install knet without GPU-support?

[denizyuret]

@maleadt: we seem to have a problem in Knet where the installation gets stuck on a new architecture with no gpu support because it cannot locate the libknet8.so artifact. In these cases we want the installation to proceed with no libknet8. Thinking that you may have had a similar problem in CUDA.jl and its artifacts I wanted to pick your brain -- do you have any suggestions?

[maleadt]

Wrap the (function that calls the) artifact macro in try/catch? https://github.com/JuliaGPU/CUDA.jl/blob/e1ba6121e4d32a5d90b6f1849926e9b4f0062700/deps/bindeps.jl#L95-L100

[denizyuret]

@manuEbg can you try:

const libknet8 = Libdl.find_library(["libknet8"], try [artifact"libknet8"]; catch ex; []; end)                                      

[manuEbg]

@manuEbg can you try:

const libknet8 = Libdl.find_library(["libknet8"], try [artifact"libknet8"]; catch ex; []; end)                                      

Hey @denizyuret , I am very sorry, but I absolutely lost track of this one.

I just noticed you implemented the suggested solution with 553a5b6 .

However I still get

ERROR: The following 1 direct dependency failed to precompile:

Knet [1902f260-5fb4-5aff-8c31-6271790ab950]

Failed to precompile Knet [1902f260-5fb4-5aff-8c31-6271790ab950] to /home/vscode/.julia/compiled/v1.7/Knet/jl_z37ZHU.
ERROR: LoadError: Cannot locate artifact 'libknet8' for aarch64-linux-gnu-libgfortran4-cxx11-libstdcxx29-julia_version+1.7.3 in '/home/vscode/.julia/packages/Knet/YIFWC/Artifacts.toml'
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:33
  [2] artifact_slash_lookup(name::String, artifact_dict::Dict{String, Any}, artifacts_toml::String, platform::Base.BinaryPlatforms.Platform)
    @ Artifacts /usr/local/julia/share/julia/stdlib/v1.7/Artifacts/src/Artifacts.jl:608
  [3] var"@artifact_str"(__source__::LineNumberNode, __module__::Module, name::Any, platform::Nothing)
    @ Artifacts /usr/local/julia/share/julia/stdlib/v1.7/Artifacts/src/Artifacts.jl:678
  [4] var"@artifact_str"(__source__::LineNumberNode, __module__::Module, name::Any)
    @ Artifacts /usr/local/julia/share/julia/stdlib/v1.7/Artifacts/src/Artifacts.jl:641
  [5] include(mod::Module, _path::String)
    @ Base ./Base.jl:418
  [6] include(x::String)
    @ Knet ~/.julia/packages/Knet/YIFWC/src/Knet.jl:1
  [7] top-level scope
    @ ~/.julia/packages/Knet/YIFWC/src/Knet.jl:17
  [8] include
    @ ./Base.jl:418 [inlined]
  [9] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
    @ Base ./loading.jl:1318
 [10] top-level scope
    @ none:1
 [11] eval
    @ ./boot.jl:373 [inlined]
 [12] eval(x::Expr)
    @ Base.MainInclude ./client.jl:453
 [13] top-level scope
    @ none:1
in expression starting at /home/vscode/.julia/packages/Knet/YIFWC/src/libknet8/LibKnet8.jl:7
in expression starting at /home/vscode/.julia/packages/Knet/YIFWC/src/libknet8/LibKnet8.jl:3
in expression starting at /home/vscode/.julia/packages/Knet/YIFWC/src/Knet.jl:1

[manuEbg]

This also falls into the same error.
It does not really make sense to me.

try 
     tempLibVar = Libdl.find_library(["libknet8"],[artifact"libknet8"])
catch ex 
     print("I catched the Exception");
end

[manuEbg]

I did some more test.

try
    artifact"notExistent"
catch ex
    print("Got it")
end

try
    platform = Base.BinaryPlatforms.HostPlatform()
    rand(4,4) * rand(5,5)
    @artifact_str("notExistent", platform)
catch ex
    print("Got it: $ex")
end

Both of the above result in :

ERROR: LoadError: Cannot locate '(Julia)Artifacts.toml' file when attempting to use artifact 'notExistent' in 'Main'
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:33
  [2] var"@artifact_str"(__source__::LineNumberNode, __module__::Module, name::Any, platform::Any)
    @ Artifacts /usr/local/julia/share/julia/stdlib/v1.7/Artifacts/src/Artifacts.jl:647
  [3] eval
    @ ./boot.jl:373 [inlined]
  [4] include_string(mapexpr::typeof(REPL.softscope), mod::Module, code::String, filename::String)
    @ Base ./loading.jl:1196
  [5] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Base ./essentials.jl:716
  [6] invokelatest(::Any, ::Any, ::Vararg{Any})
    @ Base ./essentials.jl:714
  [7] inlineeval(m::Module, code::String, code_line::Int64, code_column::Int64, file::String; softscope::Bool)
    @ VSCodeServer ~/.vscode-server/extensions/julialang.language-julia-1.6.17/scripts/packages/VSCodeServer/src/eval.jl:211
  [8] (::VSCodeServer.var"#65#69"{Bool, Bool, Module, String, Int64, Int64, String, VSCodeServer.ReplRunCodeRequestParams})()
    @ VSCodeServer ~/.vscode-server/extensions/julialang.language-julia-1.6.17/scripts/packages/VSCodeServer/src/eval.jl:155
  [9] withpath(f::VSCodeServer.var"#65#69"{Bool, Bool, Module, String, Int64, Int64, String, VSCodeServer.ReplRunCodeRequestParams}, path::String)
    @ VSCodeServer ~/.vscode-server/extensions/julialang.language-julia-1.6.17/scripts/packages/VSCodeServer/src/repl.jl:184
 [10] (::VSCodeServer.var"#64#68"{Bool, Bool, Bool, Module, String, Int64, Int64, String, VSCodeServer.ReplRunCodeRequestParams})()
    @ VSCodeServer ~/.vscode-server/extensions/julialang.language-julia-1.6.17/scripts/packages/VSCodeServer/src/eval.jl:153
 [11] hideprompt(f::VSCodeServer.var"#64#68"{Bool, Bool, Bool, Module, String, Int64, Int64, String, VSCodeServer.ReplRunCodeRequestParams})
    @ VSCodeServer ~/.vscode-server/extensions/julialang.language-julia-1.6.17/scripts/packages/VSCodeServer/src/repl.jl:36
 [12] (::VSCodeServer.var"#63#67"{Bool, Bool, Bool, Module, String, Int64, Int64, String, VSCodeServer.ReplRunCodeRequestParams})()
    @ VSCodeServer ~/.vscode-server/extensions/julialang.language-julia-1.6.17/scripts/packages/VSCodeServer/src/eval.jl:124
 [13] with_logstate(f::Function, logstate::Any)
    @ Base.CoreLogging ./logging.jl:511
 [14] with_logger
    @ ./logging.jl:623 [inlined]
 [15] (::VSCodeServer.var"#62#66"{VSCodeServer.ReplRunCodeRequestParams})()
    @ VSCodeServer ~/.vscode-server/extensions/julialang.language-julia-1.6.17/scripts/packages/VSCodeServer/src/eval.jl:201
 [16] #invokelatest#2
    @ ./essentials.jl:716 [inlined]
 [17] invokelatest(::Any)
    @ Base ./essentials.jl:714
 [18] macro expansion
    @ ~/.vscode-server/extensions/julialang.language-julia-1.6.17/scripts/packages/VSCodeServer/src/eval.jl:34 [inlined]
 [19] (::VSCodeServer.var"#60#61")()
    @ VSCodeServer ./task.jl:429

However this one:

try
    platform = Base.BinaryPlatforms.HostPlatform()
    rand(4,4) * rand(5,5)
    # @artifact_str("notExistent", platform)
catch ex
    print("Got it: $ex")
end

results in:

Got it: DimensionMismatch("A has dimensions (4,4) but B has dimensions (5,5)")

I think this behaviour is very strange

[dpo]

Same issue on FreeBSD:

LoadError: Cannot locate artifact 'libknet8' for x86_64-unknown-freebsd-libgfortran5-cxx11-julia_version+1.7.3 in '/.julia/packages/Knet/YIFWC/Artifacts.toml'

https://cirrus-ci.com/task/6610845332406272?logs=test#L201

All seems well on the GitHub Actions VMs (Ubuntu, macOS and Windows64), even though I imagine they don't have a GPU either.