Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Segmentation Fault on Loading CUDA #2453

Closed
loganpknudsen opened this issue Jul 22, 2024 · 12 comments
Closed

Segmentation Fault on Loading CUDA #2453

loganpknudsen opened this issue Jul 22, 2024 · 12 comments
Labels
bug Something isn't working

Comments

@loganpknudsen
Copy link

I am trying to load CUDA in my Julia script as follows:
using CUDA
and when I do so I get the following error message relating to a segmentation fault:


[60718] signal (11.1): Segmentation fault
in expression starting at /glade/u/home/knudsenl/.julia/packages/CUDA_Runtime_jll/YgJCI/.pkg/platform_augmentation.jl:210
Allocations: 2907 (Pool: 2898; Big: 9); GC: 0
ERROR: LoadError: Failed to precompile CUDA_Runtime_jll [76a88914-d11a-5bdc-97e0-2f5a05c973a2] to "/glade/u/home/knudsenl/.julia/compiled/v1.10/CUDA_Runtime_jll/jl_XyISs8".
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
    @ Base ./loading.jl:2468
  [3] compilecache
    @ ./loading.jl:2340 [inlined]
  [4] (::Base.var"#968#969"{Base.PkgId})()
    @ Base ./loading.jl:1974
  [5] mkpidlock(f::Base.var"#968#969"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
    @ FileWatching.Pidfile /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:93
  [6] #mkpidlock#6
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:88 [inlined]
  [7] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
    @ FileWatching.Pidfile /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:111
  [8] #invokelatest#2
    @ ./essentials.jl:894 [inlined]
  [9] invokelatest
    @ ./essentials.jl:889 [inlined]
 [10] maybe_cachefile_lock(f::Base.var"#968#969"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
    @ Base ./loading.jl:2983
 [11] maybe_cachefile_lock
    @ ./loading.jl:2980 [inlined]
 [12] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1970
 [13] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1812
 [14] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [15] invoke_in_world
    @ ./essentials.jl:923 [inlined]
[16] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1803
 [17] macro expansion
    @ ./loading.jl:1790 [inlined]
 [18] macro expansion
    @ ./lock.jl:267 [inlined]
 [19] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1753
 [20] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [21] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [22] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1746
 [23] include
    @ ./Base.jl:495 [inlined]
 [24] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
    @ Base ./loading.jl:2222
[25] top-level scope
    @ stdin:3
in expression starting at /glade/u/home/knudsenl/.julia/packages/CUDA/Tl08O/src/CUDA.jl:1
in expression starting at stdin:3
ERROR: LoadError: Failed to precompile CUDA [052768ef-5323-5732-b1bb-66c8b64840ba] to "/glade/u/home/knudsenl/.julia/compiled/v1.10/CUDA/jl_meOBVB".
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
    @ Base ./loading.jl:2468
  [3] compilecache
    @ ./loading.jl:2340 [inlined]
  [4] (::Base.var"#968#969"{Base.PkgId})()
    @ Base ./loading.jl:1974
  [5] mkpidlock(f::Base.var"#968#969"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
    @ FileWatching.Pidfile /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:93
  [6] #mkpidlock#6
  @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:88 [inlined]
  [7] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
    @ FileWatching.Pidfile /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:111
  [8] #invokelatest#2
    @ ./essentials.jl:894 [inlined]
  [9] invokelatest
    @ ./essentials.jl:889 [inlined]
 [10] maybe_cachefile_lock(f::Base.var"#968#969"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
    @ Base ./loading.jl:2983
 [11] maybe_cachefile_lock
    @ ./loading.jl:2980 [inlined]
 [12] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1970
 [13] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1812
 [14] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [15] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [16] _require_prelocked(uuidkey::Base.PkgId, env::String)
 @ Base ./loading.jl:1803
 [17] macro expansion
    @ ./loading.jl:1790 [inlined]
 [18] macro expansion
    @ ./lock.jl:267 [inlined]
 [19] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1753
 [20] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [21] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [22] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1746
in expression starting at /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/testcode.jl:1

my versioninfo() for what I am running is:

Julia Version 1.10.2
Commit bd47eca2c8 (2024-03-01 10:14 UTC)
Build Info:

    Note: This is an unofficial build, please report bugs to the project
    responsible for this build and not to the Julia project unless you can
    reproduce the issue using official builds available at https://julialang.org/downloads

Platform Info:
  OS: Linux (x86_64-suse-linux)
  CPU: 72 × Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, cascadelake)
Threads: 1 default, 0 interactive, 1 GC (on 72 virtual cores)
Environment:
  LD_LIBRARY_PATH = /glade/u/apps/casper/23.10/spack/opt/spack/openmpi/4.1.6/oneapi/2023.2.1/dgcv/lib:/glade/u/apps/common/23.08/spack/opt/spack/cuda/12.2.1/lib64:/glade/u/apps/common/23.08/spack/opt/spack/cuda/12.2.1/nvvm/lib64:/glade/u/apps/common/23.08/spack/opt/spack/cuda/12.2.1/extras/CUPTI/lib64:/glade/u/apps/common/23.08/spack/opt/spack/cuda/12.2.1/extras/Debugger/lib64:/glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/lib:/glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/lib/x64:/glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/lib/oclfpga/host/linux64/lib:/glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/compiler/lib/intel64_lin:/glade/u/apps/casper/23.10/spack/opt/spack/hdf5/1.12.2/oneapi/2023.2.1/6vf2/lib

and the status check is:

Status `~/.julia/environments/v1.10/Project.toml`
  [c7e460c6] ArgParse v1.2.0
⌃ [052768ef] CUDA v5.4.2
  [9e8cae18] Oceananigans v0.91.4
  [de0858da] Printf
  [9a3f8284] Random
Info Packages marked with ⌃ have new versions available and may be upgradable.

Any help would be appreciated! I have been working with @tomchor to fix this issue.
@loganpknudsen loganpknudsen added the bug Something isn't working label Jul 22, 2024
@loganpknudsen
Copy link
Author

loganpknudsen commented Jul 23, 2024 •
edited
Loading

A quick update, following this issue I have run

git clone https://github.com/JuliaBinaryWrappers/CUDA_Runtime_jll.jl
cd CUDA_Runtime_jll.jl
git checkout CUDA_Runtime-v0.14.1+0
JULIA_DEBUG=all julia --project -e 'using CUDA_Runtime_jll'
JULIA_DEBUG=all julia --project -e 'using Pkg; Pkg.instantiate'
JULIA_DEBUG=all julia --project -e 'using CUDA_Runtime_jll; @show Base.BinaryPlatforms.triplet(CUDA_Runtime_jll.host_platform)'

I get the following output:

┌ Debug: Ignoring cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/Preferences/pWSk8_BvCUb.ji for Preferences [21216c6a-2e73-6563-6e65-726566657250] (fafbfcfd-cc3e-9703-001f-6626af1aa549) since it is does not provide desired build_id (fafbfcfd-2d97-b1c6-0017-56458bd4c481)
└ @ Base loading.jl:3058
┌ Debug: Ignoring cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/JLLWrappers/7Zgw7_BvCUb.ji for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] (fafbfcfd-eaa4-3a3e-001f-6626df4f7b2b) since it is does not provide desired build_id (fafbfcfd-0a46-ca48-0017-564611ab88b0)
└ @ Base loading.jl:3058
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/LazyArtifacts/MRP8l_veGKK.so for LazyArtifacts [4af54fe1-eca0-43a8-85a7-787d91b784e3]
└ @ Base loading.jl:1057
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/Preferences/pWSk8_veGKK.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:1057
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/JLLWrappers/7Zgw7_veGKK.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:1057
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/CUDA_Driver_jll/QJyk7_veGKK.so for CUDA_Driver_jll [4ee394cb-3365-5eb0-8335-949819d2adfc]
└ @ Base loading.jl:1057
┌ Debug: System CUDA driver found at libcuda.so.1, detected as version 12.3.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:130
┌ Debug: Forward-compatible CUDA driver found at /glade/u/home/knudsenl/.julia/artifacts/a86b67fd924e2a8c72d376d301a34b2364281978/lib/libcuda.so; known to be version 12.5.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:184
┌ Debug: Could not use forward compatibility package (error 100)
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:192
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/CUDA_Runtime_jll/Hs50y_5HSzE.so for CUDA_Runtime_jll [76a88914-d11a-5bdc-97e0-2f5a05c973a2]
└ @ Base loading.jl:1057
Base.BinaryPlatforms.triplet(CUDA_Runtime_jll.host_platform) = "x86_64-linux-gnu-libgfortran5-cxx11-libstdcxx30-cuda+12.5-julia_version+1.10.2-cuda_local+false"

Does this help with the issue? How do I fix the forward compatability?

@maleadt
Copy link
Member

maleadt commented Jul 23, 2024

What happens if you run with JULIA_CUDA_USE_COMPAT=false?

@loganpknudsen
Copy link
Author

So I ran the following:

JULIA_DEBUG=all JULIA_CUDA_USE_COMPAT=false julia --project -e 'using CUDA_Runtime_jll; @show Base.BinaryPlatforms.triplet(CUDA_Runtime_jll.host_platform)'

and the output was:

┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/LazyArtifacts/MRP8l_veGKK.so for LazyArtifacts [4af54fe1-eca0-43a8-85a7-787d91b784e3]
└ @ Base loading.jl:1057
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/Preferences/pWSk8_veGKK.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:1057
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/JLLWrappers/7Zgw7_veGKK.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:1057
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/CUDA_Driver_jll/QJyk7_veGKK.so for CUDA_Driver_jll [4ee394cb-3365-5eb0-8335-949819d2adfc]
└ @ Base loading.jl:1057
┌ Debug: System CUDA driver found at libcuda.so.1, detected as version 12.3.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:130
┌ Debug: CUDA compat preference: false
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:142
┌ Debug: User disallows using forward-compatible driver.
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:144
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/CUDA_Runtime_jll/Hs50y_5HSzE.so for CUDA_Runtime_jll [76a88914-d11a-5bdc-97e0-2f5a05c973a2]
└ @ Base loading.jl:1057
Base.BinaryPlatforms.triplet(CUDA_Runtime_jll.host_platform) = "x86_64-linux-gnu-libgfortran5-cxx11-libstdcxx30-cuda+12.5-julia_version+1.10.2-cuda_local+false"

@tomchor tomchor mentioned this issue Jul 23, 2024
Closed
@maleadt
Copy link
Member

maleadt commented Jul 23, 2024

I meant to try and precompile CUDA.jl with that env var set.

@loganpknudsen
Copy link
Author

My bad, I got the following after running JULIA_DEBUG=all julia --project="." -e 'using CUDA_Runtime_jll; @show Base.BinaryPlatforms.triplet(CUDA_Runtime_jll.host_platform)':

┌ Debug: Skipping mtime check for file /glade/derecho/scratch/csgteam/temp/spack/casper/23.10/builds/spack-stage-julia-1.10.2-apod2ebbzv2cxtnlmfltk45mbfdgyeq3/spack-src/usr/share/julia/stdlib/v1.10/LazyArtifacts/src/LazyArtifacts.jl used by /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/compiled/v1.10/LazyArtifacts/MRP8l_VacWF.ji, since it is a stdlib
└ @ Base loading.jl:3129
┌ Debug: Loading object cache file /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/compiled/v1.10/LazyArtifacts/MRP8l_VacWF.so for LazyArtifacts [4af54fe1-eca0-43a8-85a7-787d91b784e3]
└ @ Base loading.jl:1057
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/Preferences/pWSk8_veGKK.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:1057
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/JLLWrappers/7Zgw7_veGKK.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:1057
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/CUDA_Driver_jll/QJyk7_veGKK.so for CUDA_Driver_jll [4ee394cb-3365-5eb0-8335-949819d2adfc]
└ @ Base loading.jl:1057
┌ Debug: System CUDA driver found at libcuda.so.1, detected as version 12.3.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:130
┌ Debug: Forward-compatible CUDA driver found at /glade/u/home/knudsenl/.julia/artifacts/a86b67fd924e2a8c72d376d301a34b2364281978/lib/libcuda.so; known to be version 12.5.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:184
┌ Debug: Could not use forward compatibility package (error 100)
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:192
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/CUDA_Runtime_jll/Hs50y_5HSzE.so for CUDA_Runtime_jll [76a88914-d11a-5bdc-97e0-2f5a05c973a2]
└ @ Base loading.jl:1057
Base.BinaryPlatforms.triplet(CUDA_Runtime_jll.host_platform) = "x86_64-linux-gnu-libgfortran5-cxx11-libstdcxx30-cuda+12.5-julia_version+1.10.2-cuda_local+false"

I believe this is what you meant correct?

@loganpknudsen
Copy link
Author

I don't know if this helps. but aomething else I just tried was deleting the Project.toml and Manifest.toml file as well as ~/.julia, and after running the following:

using Pkg
Pkg.activate(".")
Pkg.add("CUDA")
Pkg.instantiate()
using CUDA

I got:

[57577] signal (11.1): Segmentation fault
in expression starting at /glade/u/home/knudsenl/.julia/packages/CUDA_Runtime_jll/YgJCI/.pkg/platform_augmentation.jl:210
Allocations: 2907 (Pool: 2898; Big: 9); GC: 0
  Activating new project at `/glade/derecho/scratch/knudsenl/BottomBoundaryLayer`
  Installing known registries into `~/.julia`
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
   Installed GPUArraysCore ─────────────── v0.1.6
   Installed Crayons ───────────────────── v4.1.1
   Installed TableTraits ───────────────── v1.0.1
   Installed ColorTypes ────────────────── v0.11.5
   Installed Adapt ─────────────────────── v4.0.4
   Installed Scratch ───────────────────── v1.2.1
   Installed Preferences ───────────────── v1.4.3
   Installed PrettyTables ──────────────── v2.3.2
   Installed CUDA_Driver_jll ───────────── v0.9.1+1
   Installed SentinelArrays ────────────── v1.4.5
   Installed DataAPI ───────────────────── v1.16.0
   Installed PooledArrays ──────────────── v1.4.3
   Installed FixedPointNumbers ─────────── v0.8.5
   Installed Tables ────────────────────── v1.12.0
   Installed LLVMLoopInfo ──────────────── v1.0.0
   Installed GPUCompiler ───────────────── v0.26.7
   Installed StaticArraysCore ──────────── v1.4.3
   Installed TimerOutputs ──────────────── v0.5.24
   Installed JLLWrappers ───────────────── v1.5.0
   Installed InlineStrings ─────────────── v1.4.2
   Installed AbstractFFTs ──────────────── v1.5.0
   Installed IteratorInterfaceExtensions ─ v1.0.0
   Installed StaticArrays ──────────────── v1.9.7
   Installed PrecompileTools ───────────── v1.2.1
   Installed DataValueInterfaces ───────── v1.0.0
   Installed CUDA_Runtime_Discovery ────── v0.3.4
   Installed NVTX_jll ──────────────────── v3.1.0+2
   Installed LLVMExtra_jll ─────────────── v0.0.30+0
   Installed LaTeXStrings ──────────────── v1.3.1
   Installed OrderedCollections ────────── v1.6.3
   Installed NVTX ──────────────────────── v0.3.4
   Installed UnsafeAtomicsLLVM ─────────── v0.1.5
   Installed CEnum ─────────────────────── v0.5.0
   Installed InvertedIndices ───────────── v1.3.0
   Installed Reexport ──────────────────── v1.2.2
   Installed JuliaNVTXCallbacks_jll ────── v0.2.1+0
   Installed BFloat16s ─────────────────── v0.5.0
   Installed MacroTools ────────────────── v0.5.13
   Installed DataStructures ────────────── v0.18.20
   Installed Colors ────────────────────── v0.12.11
   Installed KernelAbstractions ────────── v0.9.22
   Installed RandomNumbers ─────────────── v1.5.3
   Installed Missings ──────────────────── v1.2.0
   Installed Compat ────────────────────── v4.15.0
   Installed StringManipulation ────────── v0.3.4
   Installed SortingAlgorithms ─────────── v1.2.1
   Installed UnsafeAtomics ─────────────── v0.2.1
   Installed Atomix ────────────────────── v0.1.0
   Installed LLVM ──────────────────────── v8.0.0
   Installed CUDA ──────────────────────── v5.4.3
    Updating `/glade/derecho/scratch/knudsenl/BottomBoundaryLayer/Project.toml`
  [052768ef] + CUDA v5.4.3
    Updating `/glade/derecho/scratch/knudsenl/BottomBoundaryLayer/Manifest.toml`
ERROR: LoadError: failed process: Process(`/glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/bin/julia -C native -J/glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/lib/julia/sys.so -g1 -O0 --color=no --history-file=no --startup-file=no --project=/glade/derecho/scratch/knudsenl/BottomBoundaryLayer/Project.toml --eval 'append!(empty!(Base.DEPOT_PATH), ["/glade/u/home/knudsenl/.julia", "/glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/local/share/julia", "/glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia"])
append!(empty!(Base.DL_LOAD_PATH), String[])

cd("/glade/u/home/knudsenl/.julia/packages/CUDA_Runtime_jll/YgJCI/.pkg")
include("/glade/u/home/knudsenl/.julia/packages/CUDA_Runtime_jll/YgJCI/.pkg/select_artifacts.jl")
' -t1 --startup-file=no x86_64-linux-gnu-libgfortran5-cxx11-libstdcxx30-julia_version+1.10.2`, ProcessSignaled(11)) [0]

Stacktrace:
  [1] pipeline_error
    @ ./process.jl:565 [inlined]
  [2] read(cmd::Cmd)
    @ Base ./process.jl:449
  [3] collect_artifacts(pkg_root::String; platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:720
  [4] collect_artifacts
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:706 [inlined]
  [5] check_artifacts_downloaded(pkg_root::String; platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:764
  [6] check_artifacts_downloaded
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:763 [inlined]
  [7] is_package_downloaded(manifest_file::String, pkg::Pkg.Types.PackageSpec; platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2195
  [8] is_package_downloaded
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2190 [inlined]
  [9] print_status(env::Pkg.Types.EnvCache, old_env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, header::Symbol, uuids::Vector{Base.UUID}, names::Vector{String}; manifest::Bool, diff::Bool, ignore_indent::Bool, outdated::Bool, extensions::Bool, io::IOStream, mode::Pkg.Types.PackageMode, hidden_upgrades_info::Bool, show_usagetips::Bool)
    @ Pkg.Operations /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2308
 [10] print_status
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2240 [inlined]
 [11] status(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}; header::Symbol, mode::Pkg.Types.PackageMode, git_diff::Bool, env_diff::Pkg.Types.EnvCache, ignore_indent::Bool, io::IOStream, outdated::Bool, extensions::Bool, hidden_upgrades_info::Bool, show_usagetips::Bool)
    @ Pkg.Operations /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2474
 [12] status (repeats 2 times)
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2442 [inlined]
 [13] #show_update#167
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2438 [inlined]
 [14] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, new_git::Set{Base.UUID}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:1397
 [15] add
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:1376 [inlined]
 [16] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform, kwargs::@Kwargs{io::IOStream})
    @ Pkg.API /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/API.jl:278
 [17] add(pkgs::Vector{Pkg.Types.PackageSpec}; io::IOStream, kwargs::@Kwargs{})
    @ Pkg.API /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/API.jl:159
 [18] add(pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.API /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/API.jl:148
 [19] add
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/API.jl:147 [inlined]
 [20] add(pkg::String)
    @ Pkg.API /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/API.jl:146
 [21] top-level scope
    @ /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/testcode.jl:3
in expression starting at /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/testcode.jl:3

@maleadt
Copy link
Member

maleadt commented Jul 24, 2024

You're posting a lot of things here, please take a step back. If I understand your issue correctly:

  • precompiling CUDA.jl v5.4.2 segfaults during precompilation of CUDA_Runtime_jll v0.14.1
  • precompiling CUDA_Runtime_jll v0.14.1 in isolation doesn't segfault, as you didn't show a segfault in Segmentation Fault on Loading CUDA #2453 (comment)
  • the impact of JULIA_CUDA_USE_COMPAT=false isn't clear to me because you seem to have applied it to a non-segfaulting operation. Can you try again on something that segfaults without it, and report if that helps?

@loganpknudsen
Copy link
Author

It appears you have a correct understanding of the issue. I went back and tried rerunning the previous steps on a GPU to see if that would prompt the segmentation fault, and after running(note that I did this on a new environment):

git clone https://github.com/JuliaBinaryWrappers/CUDA_Runtime_jll.jl
cd CUDA_Runtime_jll.jl
git checkout CUDA_Runtime-v0.14.1+0
JULIA_DEBUG=all julia --project -e 'using Pkg; Pkg.activate(".")'
JULIA_DEBUG=all julia --project -e 'using Pkg; Pkg.add("CUDA")'

I got the following output from trying to add CUDA, and it appears that I am having an issue(both on GPU and CPU) of CUDA_Runtime_jll failing to precompile causing CUDA to undergo a segmentation fault:

  Installing known registries into `~/.julia`
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
┌ Debug: tiered_resolve: trying PRESERVE_ALL
└ @ Pkg.Operations /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:1331
   Installed GPUArraysCore ─────────────── v0.1.6
   Installed Crayons ───────────────────── v4.1.1
   Installed Adapt ─────────────────────── v4.0.4
   Installed TableTraits ───────────────── v1.0.1
   Installed Scratch ───────────────────── v1.2.1
   Installed ColorTypes ────────────────── v0.11.5
   Installed Preferences ───────────────── v1.4.3
   Installed PrettyTables ──────────────── v2.3.2
   Installed CUDA_Driver_jll ───────────── v0.9.1+1
   Installed DataAPI ───────────────────── v1.16.0
   Installed SentinelArrays ────────────── v1.4.5
   Installed PooledArrays ──────────────── v1.4.3
   Installed FixedPointNumbers ─────────── v0.8.5
   Installed GPUCompiler ───────────────── v0.26.7
   Installed LLVMLoopInfo ──────────────── v1.0.0
   Installed Tables ────────────────────── v1.12.0
   Installed InlineStrings ─────────────── v1.4.2
   Installed TimerOutputs ──────────────── v0.5.24
   Installed StaticArraysCore ──────────── v1.4.3
   Installed IteratorInterfaceExtensions ─ v1.0.0
   Installed JLLWrappers ───────────────── v1.5.0
   Installed AbstractFFTs ──────────────── v1.5.0
   Installed StaticArrays ──────────────── v1.9.7
   Installed PrecompileTools ───────────── v1.2.1
   Installed NVTX_jll ──────────────────── v3.1.0+2
   Installed DataValueInterfaces ───────── v1.0.0
   Installed CUDA_Runtime_Discovery ────── v0.3.4
   Installed LaTeXStrings ──────────────── v1.3.1
   Installed LLVMExtra_jll ─────────────── v0.0.30+0
   Installed UnsafeAtomicsLLVM ─────────── v0.1.5
   Installed OrderedCollections ────────── v1.6.3
   Installed NVTX ──────────────────────── v0.3.4
   Installed JuliaNVTXCallbacks_jll ────── v0.2.1+0
   Installed InvertedIndices ───────────── v1.3.0
   Installed CEnum ─────────────────────── v0.5.0
   Installed BFloat16s ─────────────────── v0.5.0
   Installed Reexport ──────────────────── v1.2.2
   Installed Random123 ─────────────────── v1.7.0
   Installed GPUArrays ─────────────────── v10.3.0
   Installed DataFrames ────────────────── v1.6.1
   Installed Requires ──────────────────── v1.3.0
   Installed ExprTools ─────────────────── v0.1.10
   Installed Colors ────────────────────── v0.12.11
   Installed MacroTools ────────────────── v0.5.13
   Installed DataStructures ────────────── v0.18.20
   Installed RandomNumbers ─────────────── v1.5.3
   Installed KernelAbstractions ────────── v0.9.22
   Installed CUDA ──────────────────────── v5.4.3
   Installed StringManipulation ────────── v0.3.4
   Installed Compat ────────────────────── v4.15.0
   Installed Missings ──────────────────── v1.2.0
   Installed UnsafeAtomics ─────────────── v0.2.1
   Installed SortingAlgorithms ─────────── v1.2.1
   Installed Atomix ────────────────────── v0.1.0
   Installed LLVM ──────────────────────── v8.0.0
┌ Debug: Downloading artifact from Pkg server
│   name = "CUDA_Driver"
│   artifacts_toml = "/glade/u/home/knudsenl/.julia/packages/CUDA_Driver_jll/PZjqb/Artifacts.toml"
│   platform = Linux x86_64 {cxxstring_abi=cxx11, julia_version=1.10.2, libc=glibc, libgfortran_version=5.0.0, libstdcxx_version=3.4.30}
│   url = "https://pkg.julialang.org/artifact/a86b67fd924e2a8c72d376d301a34b2364281978"
└ @ Pkg.Artifacts /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Artifacts.jl:421
  Downloaded artifact: CUDA_Driver
┌ Debug: Downloading artifact from Pkg server
│   name = "LLVMExtra"
│   artifacts_toml = "/glade/u/home/knudsenl/.julia/packages/LLVMExtra_jll/gZzRL/Artifacts.toml"
│   platform = Linux x86_64 {cxxstring_abi=cxx11, julia_version=1.10.2, libc=glibc, libgfortran_version=5.0.0, libstdcxx_version=3.4.30}
│   url = "https://pkg.julialang.org/artifact/9ba4abb9d5e99d0e3e3b0eed78625b5c90957bfc"
└ @ Pkg.Artifacts /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Artifacts.jl:421
  Downloaded artifact: LLVMExtra
┌ Debug: Downloading artifact from Pkg server
│   name = "NVTX"
│   artifacts_toml = "/glade/u/home/knudsenl/.julia/packages/NVTX_jll/W9h82/Artifacts.toml"
│   platform = Linux x86_64 {cxxstring_abi=cxx11, julia_version=1.10.2, libc=glibc, libgfortran_version=5.0.0, libstdcxx_version=3.4.30}
│   url = "https://pkg.julialang.org/artifact/0b6483311b60bc85d320f64bb66fd2e310916fb1"
└ @ Pkg.Artifacts /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Artifacts.jl:421
  Downloaded artifact: NVTX
┌ Debug: Downloading artifact from Pkg server
│   name = "JuliaNVTXCallbacks"
│   artifacts_toml = "/glade/u/home/knudsenl/.julia/packages/JuliaNVTXCallbacks_jll/AIP88/Artifacts.toml"
│   platform = Linux x86_64 {cxxstring_abi=cxx11, julia_version=1.10.2, libc=glibc, libgfortran_version=5.0.0, libstdcxx_version=3.4.30}
│   url = "https://pkg.julialang.org/artifact/ff17ab283d1f7456123711f21d8b1d7e852f8024"
└ @ Pkg.Artifacts /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Artifacts.jl:421
  Downloaded artifact: JuliaNVTXCallbacks
┌ Debug: Adding include dependency on /usr/lib64/libcuda.so.1
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:78
┌ Debug: CUDA_Driver_jll not available; not selecting an artifact
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:147
┌ Debug: Failed to query CUDA driver version
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:245
┌ Debug: Adding include dependency on /usr/lib64/libcuda.so.1
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:78
┌ Debug: CUDA_Driver_jll not available; not selecting an artifact
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:147
┌ Debug: Failed to query CUDA driver version
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:245
    Updating `/glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/Project.toml`
  [052768ef] + CUDA v5.4.3
  [692b3bcd] + JLLWrappers v1.5.0
  [4ee394cb] + CUDA_Driver_jll v0.9.1+1
  [fa267f1f] ~ TOML ⇒ v1.0.3
    Updating `/glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/Manifest.toml`
  [621f4979] + AbstractFFTs v1.5.0
  [79e6a3ab] + Adapt v4.0.4
  [a9b6321e] + Atomix v0.1.0
  [ab4f0b2a] + BFloat16s v0.5.0
  [fa961155] + CEnum v0.5.0
  [052768ef] + CUDA v5.4.3
  [1af6417a] + CUDA_Runtime_Discovery v0.3.4
  [3da002f7] + ColorTypes v0.11.5
  [5ae59095] + Colors v0.12.11
  [34da2185] + Compat v4.15.0
  [a8cc5b0e] + Crayons v4.1.1
  [9a962f9c] + DataAPI v1.16.0
  [a93c6f00] + DataFrames v1.6.1
  [864edb3b] + DataStructures v0.18.20
  [e2d170a0] + DataValueInterfaces v1.0.0
  [e2ba6199] + ExprTools v0.1.10
  [53c48c17] + FixedPointNumbers v0.8.5
  [0c68f7d7] + GPUArrays v10.3.0
  [46192b85] + GPUArraysCore v0.1.6
  [61eb1bfa] + GPUCompiler v0.26.7
  [842dd82b] + InlineStrings v1.4.2
  [41ab1584] + InvertedIndices v1.3.0
  [82899510] + IteratorInterfaceExtensions v1.0.0
  [692b3bcd] + JLLWrappers v1.5.0
  [63c18a36] + KernelAbstractions v0.9.22
  [929cbde3] + LLVM v8.0.0
  [8b046642] + LLVMLoopInfo v1.0.0
  [b964fa9f] + LaTeXStrings v1.3.1
  [1914dd2f] + MacroTools v0.5.13
  [e1d29d7a] + Missings v1.2.0
  [5da4648a] + NVTX v0.3.4
  [bac558e1] + OrderedCollections v1.6.3
  [2dfb63ee] + PooledArrays v1.4.3
  [aea7be01] + PrecompileTools v1.2.1
  [21216c6a] + Preferences v1.4.3
  [08abe8d2] + PrettyTables v2.3.2
  [74087812] + Random123 v1.7.0
  [e6cf234a] + RandomNumbers v1.5.3
  [189a3867] + Reexport v1.2.2
  [ae029012] + Requires v1.3.0
  [6c6a2e73] + Scratch v1.2.1
  [91c51154] + SentinelArrays v1.4.5
  [a2af1166] + SortingAlgorithms v1.2.1
  [90137ffa] + StaticArrays v1.9.7
  [1e83bf80] + StaticArraysCore v1.4.3
  [892a3eda] + StringManipulation v0.3.4
  [3783bdb8] + TableTraits v1.0.1
  [bd369af6] + Tables v1.12.0
  [a759f4b9] + TimerOutputs v0.5.24
  [013be700] + UnsafeAtomics v0.2.1
  [d80eeb9a] + UnsafeAtomicsLLVM v0.1.5
  [4ee394cb] + CUDA_Driver_jll v0.9.1+1
  [9c1d0b0a] + JuliaNVTXCallbacks_jll v0.2.1+0
  [dad2f222] + LLVMExtra_jll v0.0.30+0
  [e98f9f5b] + NVTX_jll v3.1.0+2
  [0dad84c5] + ArgTools v1.1.1
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching
  [9fa8497b] + Future
  [b77e0a4c] + InteractiveUtils
  [4af54fe1] + LazyArtifacts
  [b27032c2] + LibCURL v0.6.4
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [ca575930] + NetworkOptions v1.2.0
  [44cfe95a] + Pkg v1.10.0
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays v1.10.0
  [10745b16] + Statistics v1.10.0
  [fa267f1f] + TOML v1.0.3
  [a4e569a6] + Tar v1.10.0
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [e66e0078] + CompilerSupportLibraries_jll v1.1.0+0
  [deac9b47] + LibCURL_jll v8.4.0+0
  [e37daf67] + LibGit2_jll v1.6.4+0
  [29816b5a] + LibSSH2_jll v1.11.0+1
  [c8ffd9c3] + MbedTLS_jll v2.28.2+1
  [14a3606d] + MozillaCACerts_jll v2023.1.10
  [4536629a] + OpenBLAS_jll v0.3.23+4
  [bea87d4a] + SuiteSparse_jll v7.2.1+1
  [83775a58] + Zlib_jll v1.2.13+1
  [8e850b90] + libblastrampoline_jll v5.8.0+1
  [8e850ede] + nghttp2_jll v1.52.0+1
  [3f19e933] + p7zip_jll v17.4.0+2
┌ Debug: Adding include dependency on /usr/lib64/libcuda.so.1
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:78
┌ Debug: Precompiling CUDA_Driver_jll [4ee394cb-3365-5eb0-8335-949819d2adfc]
└ @ Base loading.jl:2353
┌ Debug: Precompiling JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:2353
┌ Debug: Precompiling Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:2353
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/Preferences/pWSk8_a3zg0.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:1057
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/Preferences/pWSk8_a3zg0.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:1057
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/JLLWrappers/7Zgw7_a3zg0.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:1057
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/Preferences/pWSk8_a3zg0.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:1057
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/JLLWrappers/7Zgw7_a3zg0.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:1057
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/CUDA_Driver_jll/QJyk7_a3zg0.so for CUDA_Driver_jll [4ee394cb-3365-5eb0-8335-949819d2adfc]
└ @ Base loading.jl:1057
┌ Debug: System CUDA driver found at libcuda.so.1, detected as version 12.3.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:130
┌ Debug: Forward-compatible CUDA driver found at /glade/u/home/knudsenl/.julia/artifacts/a86b67fd924e2a8c72d376d301a34b2364281978/lib/libcuda.so; known to be version 12.5.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:184
┌ Debug: Could not use forward compatibility package (error 803)
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:192

[7090] signal (11.1): Segmentation fault
in expression starting at /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:144
Allocations: 605840 (Pool: 605079; Big: 761); GC: 1
ERROR: failed process: Process(`/glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/bin/julia -C native -J/glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/lib/julia/sys.so -g1 -O0 --color=no --history-file=no --startup-file=no --project=/glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/Project.toml --eval 'append!(empty!(Base.DEPOT_PATH), ["/glade/u/home/knudsenl/.julia", "/glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/local/share/julia", "/glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia"])
append!(empty!(Base.DL_LOAD_PATH), String[])

cd("/glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg")
include("/glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/select_artifacts.jl")
' -t1 --startup-file=no x86_64-linux-gnu-libgfortran5-cxx11-libstdcxx30-julia_version+1.10.2`, ProcessSignaled(11)) [0]

Stacktrace:
  [1] pipeline_error
    @ ./process.jl:565 [inlined]
  [2] read(cmd::Cmd)
    @ Base ./process.jl:449
  [3] collect_artifacts(pkg_root::String; platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:720
  [4] collect_artifacts
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:706 [inlined]
  [5] check_artifacts_downloaded(pkg_root::String; platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:764
  [6] check_artifacts_downloaded
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:763 [inlined]
  [7] is_package_downloaded(manifest_file::String, pkg::Pkg.Types.PackageSpec; platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2195
  [8] is_package_downloaded
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2190 [inlined]
  [9] #10
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:132 [inlined]
 [10] _all
    @ ./reduce.jl:1288 [inlined]
 [11] all
    @ ./reducedim.jl:1023 [inlined]
 [12] is_instantiated(env::Pkg.Types.EnvCache; platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:132
 [13] is_instantiated
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:116 [inlined]
 [14] instantiate(ctx::Pkg.Types.Context; manifest::Nothing, update_registry::Bool, verbose::Bool, platform::Base.BinaryPlatforms.Platform, allow_build::Bool, allow_autoprecomp::Bool, kwargs::@Kwargs{})
    @ Pkg.API /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/API.jl:1820
 [15] precompile(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; internal_call::Bool, strict::Bool, warn_loaded::Bool, already_instantiated::Bool, timing::Bool, _from_loading::Bool, kwargs::@Kwargs{})
    @ Pkg.API /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/API.jl:1082
 [16] precompile
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/API.jl:1078 [inlined]
 [17] #_auto_precompile#6
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Pkg.jl:805 [inlined]
 [18] _auto_precompile (repeats 2 times)
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Pkg.jl:803 [inlined]
 [19] add(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::@Kwargs{})
    @ Pkg.API /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/API.jl:160
 [20] add(pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.API /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/API.jl:148
 [21] add
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/API.jl:147 [inlined]
 [22] add(pkg::String)
    @ Pkg.API /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/API.jl:146
 [23] top-level scope
    @ none:1

Does this help make the issue clearer?

@maleadt
Copy link
Member

maleadt commented Jul 25, 2024

That confirms my understanding, however, can you try that again now with JULIA_CUDA_USE_COMPAT=false set in your environment? If that works, you can keep it around, or set it permanently as a preference:

CUDA.jl/LocalPreferences.toml

Lines 19 to 23 in beccab1

[CUDA_Driver_jll]
# whether to attempt to load a forwards-compatibile userspace driver.
# only turn this off if you experience issues, e.g., when using a local
# toolkit that's much older than the available forwards compatible driver.
#compat = "true"

@loganpknudsen
Copy link
Author

Sorry for the late response, I just ran that and the output was the following(it seemed to work!):

  Activating project at `/glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl`
knudsenl@casper-login2:/glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl> JULIA_DEBUG=all JULIA_CUDA_USE_COMPAT=false julia --project -e 'using Pkg; Pkg.add("CUDA")'
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
┌ Debug: tiered_resolve: trying PRESERVE_ALL
└ @ Pkg.Operations /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.9.2/gcc/7.5.0/mjea/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1328
┌ Debug: Adding include dependency on /usr/lib64/libcuda.so.1
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:78
┌ Debug: CUDA_Driver_jll not available; not selecting an artifact
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:147
┌ Debug: Failed to query CUDA driver version
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:245
┌ Debug: Adding include dependency on /usr/lib64/libcuda.so.1
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:78
┌ Debug: CUDA_Driver_jll not available; not selecting an artifact
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:147
┌ Debug: Failed to query CUDA driver version
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:245
    Updating `/glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/Project.toml`
  [052768ef] + CUDA v5.4.3
  [692b3bcd] + JLLWrappers v1.5.0
  [4ee394cb] + CUDA_Driver_jll v0.9.1+1
  [fa267f1f] ~ TOML ⇒ v1.0.3
    Updating `/glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/Manifest.toml`
  [621f4979] + AbstractFFTs v1.5.0
  [79e6a3ab] + Adapt v4.0.4
  [a9b6321e] + Atomix v0.1.0
  [ab4f0b2a] + BFloat16s v0.5.0
  [fa961155] + CEnum v0.5.0
  [052768ef] + CUDA v5.4.3
  [1af6417a] + CUDA_Runtime_Discovery v0.3.4
  [3da002f7] + ColorTypes v0.11.5
  [5ae59095] + Colors v0.12.11
  [34da2185] + Compat v4.15.0
  [a8cc5b0e] + Crayons v4.1.1
  [9a962f9c] + DataAPI v1.16.0
  [a93c6f00] + DataFrames v1.6.1
  [864edb3b] + DataStructures v0.18.20
  [e2d170a0] + DataValueInterfaces v1.0.0
  [e2ba6199] + ExprTools v0.1.10
  [53c48c17] + FixedPointNumbers v0.8.5
⌅ [0c68f7d7] + GPUArrays v10.2.3
  [46192b85] + GPUArraysCore v0.1.6
  [61eb1bfa] + GPUCompiler v0.26.7
  [842dd82b] + InlineStrings v1.4.2
  [41ab1584] + InvertedIndices v1.3.0
  [82899510] + IteratorInterfaceExtensions v1.0.0
  [692b3bcd] + JLLWrappers v1.5.0
  [63c18a36] + KernelAbstractions v0.9.22
  [929cbde3] + LLVM v8.0.0
  [8b046642] + LLVMLoopInfo v1.0.0
  [b964fa9f] + LaTeXStrings v1.3.1
  [1914dd2f] + MacroTools v0.5.13
  [e1d29d7a] + Missings v1.2.0
  [5da4648a] + NVTX v0.3.4
  [bac558e1] + OrderedCollections v1.6.3
  [2dfb63ee] + PooledArrays v1.4.3
  [aea7be01] + PrecompileTools v1.2.1
  [21216c6a] + Preferences v1.4.3
  [08abe8d2] + PrettyTables v2.3.2
  [74087812] + Random123 v1.7.0
  [e6cf234a] + RandomNumbers v1.5.3
  [189a3867] + Reexport v1.2.2
  [ae029012] + Requires v1.3.0
  [6c6a2e73] + Scratch v1.2.1
  [91c51154] + SentinelArrays v1.4.5
  [a2af1166] + SortingAlgorithms v1.2.1
  [90137ffa] + StaticArrays v1.9.7
  [1e83bf80] + StaticArraysCore v1.4.3
  [892a3eda] + StringManipulation v0.3.4
  [3783bdb8] + TableTraits v1.0.1
  [bd369af6] + Tables v1.12.0
  [a759f4b9] + TimerOutputs v0.5.24
  [013be700] + UnsafeAtomics v0.2.1
  [d80eeb9a] + UnsafeAtomicsLLVM v0.1.5
  [4ee394cb] + CUDA_Driver_jll v0.9.1+1
  [9c1d0b0a] + JuliaNVTXCallbacks_jll v0.2.1+0
  [dad2f222] + LLVMExtra_jll v0.0.30+0
  [e98f9f5b] + NVTX_jll v3.1.0+2
  [0dad84c5] + ArgTools v1.1.1
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching
  [9fa8497b] + Future
  [b77e0a4c] + InteractiveUtils
  [4af54fe1] + LazyArtifacts
  [b27032c2] + LibCURL v0.6.3
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [ca575930] + NetworkOptions v1.2.0
  [44cfe95a] + Pkg v1.9.2
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics v1.9.0
  [fa267f1f] + TOML v1.0.3
  [a4e569a6] + Tar v1.10.0
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [e66e0078] + CompilerSupportLibraries_jll v1.0.5+0
  [deac9b47] + LibCURL_jll v7.84.0+0
  [29816b5a] + LibSSH2_jll v1.10.2+0
  [c8ffd9c3] + MbedTLS_jll v2.28.2+0
  [14a3606d] + MozillaCACerts_jll v2022.10.11
  [4536629a] + OpenBLAS_jll v0.3.21+4
  [bea87d4a] + SuiteSparse_jll v5.10.1+6
  [83775a58] + Zlib_jll v1.2.13+0
  [8e850b90] + libblastrampoline_jll v5.8.0+0
  [8e850ede] + nghttp2_jll v1.48.0+0
  [3f19e933] + p7zip_jll v17.4.0+0
        Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. To see why use `status --outdated -m`
┌ Debug: Adding include dependency on /usr/lib64/libcuda.so.1
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:78
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.9/Preferences/pWSk8_VphDT.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:1051
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.9/JLLWrappers/7Zgw7_VphDT.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:1051
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.9/CUDA_Driver_jll/QJyk7_VphDT.so for CUDA_Driver_jll [4ee394cb-3365-5eb0-8335-949819d2adfc]
└ @ Base loading.jl:1051
┌ Debug: System CUDA driver found at libcuda.so.1, detected as version 12.3.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:130
┌ Debug: CUDA compat preference: false
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:142
┌ Debug: User disallows using forward-compatible driver.
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:144
┌ Debug: Using CUDA_Driver_jll for driver discovery
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:152
┌ Debug: Found CUDA driver at 'libcuda.so.1'
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:179
┌ Debug: CUDA driver version: 12.3.0
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:248
┌ Debug: Selected CUDA toolkit: 12.5.0
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:279
┌ Warning: Circular dependency detected. Precompilation will be skipped for:
│   CUDA_Runtime_jll [76a88914-d11a-5bdc-97e0-2f5a05c973a2]
│   CUDA [052768ef-5323-5732-b1bb-66c8b64840ba]
└ @ Pkg.API /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.9.2/gcc/7.5.0/mjea/share/julia/stdlib/v1.9/Pkg/src/API.jl:1234
┌ Debug: Rejecting cache file /glade/u/home/knudsenl/.julia/compiled/v1.9/MbedTLS_jll/u5NEn_X8D0f.ji for  [top-level] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2
│   cache file:      use_pkgimages = false, debug_level = 1, check_bounds = 0, inline = true, opt_level = 0
└ @ Base loading.jl:2730
┌ Debug: Rejecting cache file /glade/u/home/knudsenl/.julia/compiled/v1.9/Zlib_jll/xjq3Q_X8D0f.ji for  [top-level] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2
│   cache file:      use_pkgimages = false, debug_level = 1, check_bounds = 0, inline = true, opt_level = 0
└ @ Base loading.jl:2730
┌ Debug: Rejecting cache file /glade/u/home/knudsenl/.julia/compiled/v1.9/InlineStrings/prmY3_X8D0f.ji for  [top-level] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2
│   cache file:      use_pkgimages = false, debug_level = 1, check_bounds = 0, inline = true, opt_level = 0
└ @ Base loading.jl:2730
┌ Debug: Rejecting cache file /glade/u/home/knudsenl/.julia/compiled/v1.9/Statistics/ERcPL_X8D0f.ji for  [top-level] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2
│   cache file:      use_pkgimages = false, debug_level = 1, check_bounds = 0, inline = true, opt_level = 0
└ @ Base loading.jl:2730
┌ Debug: Rejecting cache file /glade/u/home/knudsenl/.julia/compiled/v1.9/SuiteSparse_jll/ME9At_X8D0f.ji for  [top-level] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2
│   cache file:      use_pkgimages = false, debug_level = 1, check_bounds = 0, inline = true, opt_level = 0
└ @ Base loading.jl:2730
┌ Debug: Skipping mtime check for file /glade/derecho/scratch/csgteam/temp/spack/casper/23.10/builds/spack-stage-julia-1.9.2-mjeadxih745lj3s24lbol2ou7lpwqtse/spack-src/usr/share/julia/stdlib/v1.9/MbedTLS_jll/src/MbedTLS_jll.jl used by /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.9.2/gcc/7.5.0/mjea/share/julia/compiled/v1.9/MbedTLS_jll/u5NEn_1a2YF.ji, since it is a stdlib
└ @ Base loading.jl:2835
┌ Debug: Skipping mtime check for file /glade/derecho/scratch/csgteam/temp/spack/casper/23.10/builds/spack-stage-julia-1.9.2-mjeadxih745lj3s24lbol2ou7lpwqtse/spack-src/usr/share/julia/stdlib/v1.9/Zlib_jll/src/Zlib_jll.jl used by /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.9.2/gcc/7.5.0/mjea/share/julia/compiled/v1.9/Zlib_jll/xjq3Q_1a2YF.ji, since it is a stdlib
└ @ Base loading.jl:2835
┌ Debug: Skipping mtime check for file /glade/derecho/scratch/csgteam/temp/spack/casper/23.10/builds/spack-stage-julia-1.9.2-mjeadxih745lj3s24lbol2ou7lpwqtse/spack-src/usr/share/julia/stdlib/v1.9/Statistics/src/Statistics.jl used by /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.9.2/gcc/7.5.0/mjea/share/julia/compiled/v1.9/Statistics/ERcPL_1a2YF.ji, since it is a stdlib
└ @ Base loading.jl:2835
┌ Debug: Skipping mtime check for file /glade/derecho/scratch/csgteam/temp/spack/casper/23.10/builds/spack-stage-julia-1.9.2-mjeadxih745lj3s24lbol2ou7lpwqtse/spack-src/usr/share/julia/stdlib/v1.9/SuiteSparse_jll/src/SuiteSparse_jll.jl used by /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.9.2/gcc/7.5.0/mjea/share/julia/compiled/v1.9/SuiteSparse_jll/ME9At_1a2YF.ji, since it is a stdlib
└ @ Base loading.jl:2835
┌ Debug: Rejecting cache file /glade/u/home/knudsenl/.julia/compiled/v1.9/Statistics/ERcPL_X8D0f.ji for Statistics [10745b16-79ce-11e8-11f9-7d13ad32a3b2] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2
│   cache file:      use_pkgimages = false, debug_level = 1, check_bounds = 0, inline = true, opt_level = 0
└ @ Base loading.jl:2730
┌ Debug: Rejecting cache file /glade/u/home/knudsenl/.julia/compiled/v1.9/LibSSH2_jll/K6mup_X8D0f.ji for  [top-level] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2
│   cache file:      use_pkgimages = false, debug_level = 1, check_bounds = 0, inline = true, opt_level = 0
└ @ Base loading.jl:2730
┌ Debug: Rejecting cache file /glade/u/home/knudsenl/.julia/compiled/v1.9/Statistics/ERcPL_X8D0f.ji for Statistics [10745b16-79ce-11e8-11f9-7d13ad32a3b2] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2
│   cache file:      use_pkgimages = false, debug_level = 1, check_bounds = 0, inline = true, opt_level = 0
└ @ Base loading.jl:2730
┌ Debug: Rejecting cache file /glade/u/home/knudsenl/.julia/compiled/v1.9/Statistics/ERcPL_X8D0f.ji for Statistics [10745b16-79ce-11e8-11f9-7d13ad32a3b2] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2
│   cache file:      use_pkgimages = false, debug_level = 1, check_bounds = 0, inline = true, opt_level = 0
└ @ Base loading.jl:2730
┌ Debug: Skipping mtime check for file /glade/derecho/scratch/csgteam/temp/spack/casper/23.10/builds/spack-stage-julia-1.9.2-mjeadxih745lj3s24lbol2ou7lpwqtse/spack-src/usr/share/julia/stdlib/v1.9/Statistics/src/Statistics.jl used by /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.9.2/gcc/7.5.0/mjea/share/julia/compiled/v1.9/Statistics/ERcPL_1a2YF.ji, since it is a stdlib
└ @ Base loading.jl:2835
┌ Debug: Skipping mtime check for file /glade/derecho/scratch/csgteam/temp/spack/casper/23.10/builds/spack-stage-julia-1.9.2-mjeadxih745lj3s24lbol2ou7lpwqtse/spack-src/usr/share/julia/stdlib/v1.9/LibSSH2_jll/src/LibSSH2_jll.jl used by /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.9.2/gcc/7.5.0/mjea/share/julia/compiled/v1.9/LibSSH2_jll/K6mup_1a2YF.ji, since it is a stdlib
└ @ Base loading.jl:2835
┌ Debug: Skipping mtime check for file /glade/derecho/scratch/csgteam/temp/spack/casper/23.10/builds/spack-stage-julia-1.9.2-mjeadxih745lj3s24lbol2ou7lpwqtse/spack-src/usr/share/julia/stdlib/v1.9/Statistics/src/Statistics.jl used by /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.9.2/gcc/7.5.0/mjea/share/julia/compiled/v1.9/Statistics/ERcPL_1a2YF.ji, since it is a stdlib
└ @ Base loading.jl:2835
┌ Debug: Skipping mtime check for file /glade/derecho/scratch/csgteam/temp/spack/casper/23.10/builds/spack-stage-julia-1.9.2-mjeadxih745lj3s24lbol2ou7lpwqtse/spack-src/usr/share/julia/stdlib/v1.9/Statistics/src/Statistics.jl used by /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.9.2/gcc/7.5.0/mjea/share/julia/compiled/v1.9/Statistics/ERcPL_1a2YF.ji, since it is a stdlib
└ @ Base loading.jl:2835
┌ Debug: Rejecting cache file /glade/u/home/knudsenl/.julia/compiled/v1.9/MbedTLS_jll/u5NEn_X8D0f.ji for MbedTLS_jll [c8ffd9c3-330d-5841-b78e-0817d7145fa1] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2
│   cache file:      use_pkgimages = false, debug_level = 1, check_bounds = 0, inline = true, opt_level = 0
└ @ Base loading.jl:2730
┌ Debug: Skipping mtime check for file /glade/derecho/scratch/csgteam/temp/spack/casper/23.10/builds/spack-stage-julia-1.9.2-mjeadxih745lj3s24lbol2ou7lpwqtse/spack-src/usr/share/julia/stdlib/v1.9/MbedTLS_jll/src/MbedTLS_jll.jl used by /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.9.2/gcc/7.5.0/mjea/share/julia/compiled/v1.9/MbedTLS_jll/u5NEn_1a2YF.ji, since it is a stdlib
└ @ Base loading.jl:2835

@loganpknudsen
Copy link
Author

Thank you to everyone who took the time to respond to this issue! I am still unclear on what exactly was driving the error I was getting(I am open to continuing to work on this issue but feel free to close it), but I was able to fix it by creating a new environment in a clean directory and running the following modules on Casper for the UCAR HPC resources:

module --force purge
module --ignore-cache load ncarenv/23.10 gcc ncarcompilers netcdf
module --ignore-cache load cuda
module --ignore-cache load julia/1.10.2

and then running my script. Thanks again for all the help!

@maleadt
Copy link
Member

maleadt commented Jul 30, 2024

Great, thanks for following up. I think we can close this in favor of #2446 then, which should resolve the issue by more safely handling the forwards-compatible library.

@maleadt maleadt closed this as completed Jul 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@maleadt @loganpknudsen