[NCCL] 2.19.4 build from source

[simonbyrne]

It was suggested in #7491 (comment) to try building this from source, instead of repackaging the binaries.

[simonbyrne]

catastrophic error: error while writing generated C file: No space left on device

Any ideas?

[giordano]

Run df -hT after the failure: #7779 (comment)

[simonbyrne]

How do I run something after the failure?

[giordano]

command_that_fails || df -hT

like in

Yggdrasil/X/XLA/build_tarballs.jl

Line 27 in 4c71b69

bazel build --test_output=all --spawn_strategy=local --verbose_failures //xla/... || df -hT

[simonbyrne]

The non-x86_64 builds are failing because it is trying to run fmt: I added coreutils as a BuildDependency, but that installs the target arch version. Is there a way to get the x86_64 version?

[giordano]

HostBuildDependency: https://docs.binarybuilder.org/stable/build_tips/#Dependencies-for-the-target-system-vs-host-system

[simonbyrne]

https://buildkite.com/julialang/yggdrasil/builds/7094#018c5efb-c3a9-463c-9cc5-a1c47ab2f2e7/6-7609

[simonbyrne]

In the cross-compiled targets (aarch64 and powerpc64le) I get

[20:04:32] bash: line 1: /workspace/destdir/cuda/bin/nvcc: cannot execute binary file: Exec format error

Are we picking up the wrong nvcc?

[simonbyrne]

As for the space issue:
https://buildkite.com/julialang/yggdrasil/builds/7099#018c5f9f-b77a-41ed-8340-0095d6fbe0c7/6-2661

[20:06:02] ./prims_ll128.h(222): catastrophic error: error while writing generated C file: No space left on device
[20:06:02] 
[20:06:02] 1 catastrophic error detected in the compilation of "/workspace/destdir/obj/device/gensrc/reduce_scatter_sum_u8.cu".
[20:06:02] Compilation terminated.

and
https://buildkite.com/julialang/yggdrasil/builds/7099#018c5f9f-b77a-41ed-8340-0095d6fbe0c7/6-7520

[20:11:56]  ---> df -hT
[20:11:56] Filesystem           Type            Size      Used Available Use% Mounted on
[20:11:56] overlay              overlay         1.0G    184.0K   1023.8M   0% /
[20:11:56] df: /opt/x86_64-linux-musl/GCCBootstrap-4.8.5: No such file or directory
[20:11:56] df: /opt/x86_64-linux-musl/PlatformSupport-2023.6.10: No such file or directory
[20:11:56] df: /opt/x86_64-linux-musl/LLVMBootstrap-16.0.6: No such file or directory
[20:11:56] df: /opt/x86_64-linux-gnu/GCCBootstrap-4.8.5: No such file or directory
[20:11:56] df: /opt/x86_64-linux-gnu/PlatformSupport-2023.6.10: No such file or directory
[20:11:56] overlay              overlay         1.0G    184.0K   1023.8M   0% /opt/x86_64-linux-gnu
[20:11:56] overlay              overlay         1.0G    184.0K   1023.8M   0% /opt/x86_64-linux-musl
[20:11:56] udev                 devtmpfs      251.8G         0    251.8G   0% /dev/null
[20:11:56] udev                 devtmpfs      251.8G         0    251.8G   0% /dev/tty
[20:11:56] udev                 devtmpfs      251.8G         0    251.8G   0% /dev/urandom
[20:11:56] tank/root            zfs           228.6G     24.8G    203.8G  11% /etc/resolv.conf
[20:11:56] /dev/nvme0n1p1       ext4            3.4T      2.3T    938.6G  72% /root/.ccache
[20:11:56] tank/root            zfs           228.6G     24.8G    203.8G  11% /opt/toolchains
[20:11:56] tank/root            zfs           228.6G     24.8G    203.8G  11% /opt/bin
[20:11:56] /dev/nvme0n1p1       ext4            3.4T      2.3T    938.6G  72% /meta
[20:11:56] /dev/nvme0n1p1       ext4            3.4T      2.3T    938.6G  72% /workspace

so it doesn't look full?

[simonbyrne]

Okay, changing TMPDIR seemed to fix it.

Now I get https://buildkite.com/julialang/yggdrasil/builds/7100#018c6005-3d85-477f-9cbc-ca9413d67490/6-7523:

/opt/x86_64-linux-gnu/bin/../lib/gcc/x86_64-linux-gnu/4.8.5/../../../../x86_64-linux-gnu/bin/ld: cannot find -lcudart_static

[simonbyrne]

Okay I think this is now working: @maleadt @vchuravy can you take a look?

Some questions:

• Can nvcc cross compile? I don't see any other examples where we do.
• Should I link against the cudart_static (the default) or cudart (what I've done above)?

[simonbyrne]

Also, should we delete the static library from the artifact? It's pretty big (200mb)

[maleadt]

• Can nvcc cross compile? I don't see any other examples where we do.

No. You can try using clang for that, but I'm not sure it'll manage to compile all of NCCL's CUDA code.

• Should I link against the cudart_static (the default) or cudart (what I've done above)?

The dynamic one is fine, and keeps the generated binaries smaller.
It does of course mean that you need to depend on the CUDA_Runtime_jll, but that should be fine here, I guess.

[simonbyrne]

Okay, I think this is ready. I'll leave the cross compilation part for now.

[simonbyrne]

Can we merge this?

LICENSE has been re-added

[simonbyrne]

Hmm, it looks like it isn't loading correctly on Linux when no CUDA is installed (JuliaRegistries/General#97147 (comment))

(nccl) pkg> add NCCL_jll#main
...

julia> using NCCL_jll
ERROR: InitError: could not load library "/home/spjbyrne/.julia/artifacts/c99fd5bde3173b2e3f53a67a1383b5bb611c8267/lib/libnccl.so"
libcudart.so.12: cannot open shared object file: No such file or directory
Stacktrace:
  [1] dlopen(s::String, flags::UInt32; throw_error::Bool)
    @ Base.Libc.Libdl ./libdl.jl:117
  [2] dlopen(s::String, flags::UInt32)
    @ Base.Libc.Libdl ./libdl.jl:116
  [3] macro expansion
    @ ~/.julia/packages/JLLWrappers/pG9bm/src/products/library_generators.jl:63 [inlined]
  [4] __init__()
    @ NCCL_jll ~/.julia/packages/NCCL_jll/qlpSf/src/wrappers/x86_64-linux-gnu-cuda+12.3.jl:10
  [5] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1115
  [6] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any})
    @ Base ./loading.jl:1061
  [7] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128)
    @ Base ./loading.jl:1506
  [8] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1783
  [9] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1660
 [10] macro expansion
    @ ./loading.jl:1648 [inlined]
 [11] macro expansion
    @ ./lock.jl:267 [inlined]
 [12] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1611
during initialization of module NCCL_jll