Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Segmentation Fault when trying to run a script #3655

Closed
loganpknudsen opened this issue Jul 16, 2024 · 22 comments
Closed

Segmentation Fault when trying to run a script #3655

loganpknudsen opened this issue Jul 16, 2024 · 22 comments

Comments

@loganpknudsen
Copy link

loganpknudsen commented Jul 16, 2024 •
edited
Loading

I am trying to run a script and when opening packages as below:

### Load in Packages
# using Pkg
# Pkg.instantiate()
using Oceananigans
using Oceananigans.AbstractOperations: @at, ∂x, ∂y, ∂z
using Oceananigans.AbstractOperations: @at, Average
using Oceananigans.Grids: Center, Face
using Oceananigans.Units
using Random
using Printf
using ArgParse
using CUDA: has_cuda_gpu
using CUDA 
using Oceanostics

I get the following error:

[32609] signal (11.1): Segmentation fault
in expression starting at /glade/u/home/knudsenl/.julia/packages/Oceanostics/cbCj1/src/Oceanostics.jl:35
Allocations: 575575 (Pool: 574672; Big: 903); GC: 1
┌ Warning: You are using a non-official build of Julia. This may cause issues with CUDA.jl.
│ Please consider using an official build from https://julialang.org/downloads/.
└ @ CUDA ~/.julia/packages/CUDA/Tl08O/src/initialization.jl:180
┌ Warning: CUDA runtime library `libcublasLt.so.12` was loaded from a system path, `/glade/u/apps/common/23.08/spack/opt/spack/cuda/12.2.1/lib64/libcublasLt.so.12`.
│
│ This may cause errors. Ensure that you have not set the LD_LIBRARY_PATH
│ environment variable, or that it does not contain paths to CUDA libraries.
│
│ In any other case, please file an issue.
└ @ CUDA ~/.julia/packages/CUDA/Tl08O/src/initialization.jl:219
┌ Warning: CUDA runtime library `libnvJitLink.so.12` was loaded from a system path, `/glade/u/apps/common/23.08/spack/opt/spack/cuda/12.2.1/lib64/libnvJitLink.so.12`.
│
│ This may cause errors. Ensure that you have not set the LD_LIBRARY_PATH
"BBL_O_10.out" 47L, 2643B                                     1,0-1         Top

How do I fix this? Thank you for any help
Edit: I have been working with @tomchor on trying to debug this issue
@vchuravy
Copy link
Collaborator

What is your versioninfo()? How did you install Julia?

┌ Warning: You are using a non-official build of Julia. This may cause issues with CUDA.jl.
│ Please consider using an official build from https://julialang.org/downloads/.
└ @ CUDA ~/.julia/packages/CUDA/Tl08O/src/initialization.jl:180

I would recommend using juliaup to manage your Julia installation.

@loganpknudsen
Copy link
Author

I checked and it is showing that I am running version 1.9.2. I just installed juliaup and made sure Oceanostics was updated. I got the following error

[36147] signal (11.1): Segmentation fault
in expression starting at /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/BBL_with_oscillations_code_10_percent.jl:4
Allocations: 609090 (Pool: 608187; Big: 903); GC: 1
/var/spool/pbs/mom_priv/jobs/1726131.casper-pbs.SC: line 31: 36147 Segmentation fault      (core dumped) julia --project BBL_with_oscillations_code_10_percent.jl /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/

@loganpknudsen
Copy link
Author

So I reduced my code to be just

using Oceananigans

and the code gave the following error

[20610] signal (11.1): Segmentation fault
in expression starting at /glade/u/home/knudsenl/.julia/packages/CUDA_Runtime_jll/YgJCI/.pkg/platform_augmentation.jl:283
Allocations: 2907 (Pool: 2898; Big: 9); GC: 0
ERROR: LoadError: Failed to precompile CUDA_Runtime_jll [76a88914-d11a-5bdc-97e0-2f5a05c973a2] to "/glade/u/home/knudsenl/.julia/compiled/v1.10/CUDA_Runtime_jll/jl_FIXMhf".
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
    @ Base ./loading.jl:2468
  [3] compilecache
    @ ./loading.jl:2340 [inlined]
  [4] (::Base.var"#968#969"{Base.PkgId})()
    @ Base ./loading.jl:1974
  [5] mkpidlock(f::Base.var"#968#969"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
    @ FileWatching.Pidfile /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:93
  [6] #mkpidlock#6    
@ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:88 [inlined]
  [7] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
    @ FileWatching.Pidfile /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:111
  [8] #invokelatest#2
    @ ./essentials.jl:894 [inlined]
  [9] invokelatest
    @ ./essentials.jl:889 [inlined]
 [10] maybe_cachefile_lock(f::Base.var"#968#969"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
    @ Base ./loading.jl:2983
 [11] maybe_cachefile_lock
    @ ./loading.jl:2980 [inlined]
 [12] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1970
 [13] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1812
 [14] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [15] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [16] _require_prelocked(uuidkey::Base.PkgId, env::String)
 @ Base ./loading.jl:1803
 [17] macro expansion
    @ ./loading.jl:1790 [inlined]
 [18] macro expansion
    @ ./lock.jl:267 [inlined]
 [19] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1753
 [20] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [21] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [22] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1746
 [23] include
    @ ./Base.jl:495 [inlined]
 [24] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
    @ Base ./loading.jl:2222
 [25] top-level scope
    @ stdin:3
in expression starting at /glade/u/home/knudsenl/.julia/packages/CUDA/Tl08O/src/CUDA.jl:1
in expression starting at stdin:3
ERROR: LoadError: Failed to precompile CUDA [052768ef-5323-5732-b1bb-66c8b64840ba] to "/glade/u/home/knudsenl/.julia/compiled/v1.10/CUDA/jl_zRopeZ".
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
    @ Base ./loading.jl:2468
  [3] compilecache
    @ ./loading.jl:2340 [inlined]
  [4] (::Base.var"#968#969"{Base.PkgId})()
    @ Base ./loading.jl:1974
  [5] mkpidlock(f::Base.var"#968#969"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
    @ FileWatching.Pidfile /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:93
  [6] #mkpidlock#6
@ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:88 [inlined]
  [7] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
    @ FileWatching.Pidfile /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:111
  [8] #invokelatest#2
    @ ./essentials.jl:894 [inlined]
  [9] invokelatest
    @ ./essentials.jl:889 [inlined]
 [10] maybe_cachefile_lock(f::Base.var"#968#969"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
    @ Base ./loading.jl:2983
 [11] maybe_cachefile_lock
    @ ./loading.jl:2980 [inlined]
 [12] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1970
 [13] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1812
 [14] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [15] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [16] _require_prelocked(uuidkey::Base.PkgId, env::String)
 @ Base ./loading.jl:1803
 [17] macro expansion
    @ ./loading.jl:1790 [inlined]
 [18] macro expansion
    @ ./lock.jl:267 [inlined]
 [19] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1753
 [20] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [21] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [22] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1746
 [23] include
    @ ./Base.jl:495 [inlined]
 [24] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
    @ Base ./loading.jl:2222
 [25] top-level scope
    @ stdin:3
in expression starting at /glade/u/home/knudsenl/.julia/packages/Oceananigans/M82LU/src/Oceananigans.jl:1
in expression starting at stdin:3
ERROR: LoadError: Failed to precompile Oceananigans [9e8cae18-63c1-5223-a75c-80ca9d6e9a09] to "/glade/u/home/knudsenl/.julia/compiled/v1.10/Oceananigans/jl_k7YOZN".
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
    @ Base ./loading.jl:2468
  [3] compilecache
    @ ./loading.jl:2340 [inlined]
  [4] (::Base.var"#968#969"{Base.PkgId})()
    @ Base ./loading.jl:1974
  [5] mkpidlock(f::Base.var"#968#969"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
    @ FileWatching.Pidfile /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:93
  [6] #mkpidlock#6
@ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:88 [inlined]
  [7] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
    @ FileWatching.Pidfile /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:111
  [8] #invokelatest#2
    @ ./essentials.jl:894 [inlined]
  [9] invokelatest
    @ ./essentials.jl:889 [inlined]
 [10] maybe_cachefile_lock(f::Base.var"#968#969"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
    @ Base ./loading.jl:2983
 [11] maybe_cachefile_lock
    @ ./loading.jl:2980 [inlined]
 [12] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1970
 [13] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1812
 [14] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [15] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [16] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1803
 [17] macro expansion
    @ ./loading.jl:1790 [inlined]
 [18] macro expansion
    @ ./lock.jl:267 [inlined]
 [19] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1753
 [20] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [21] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [22] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1746
in expression starting at /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/testcode.jl:1

@loganpknudsen
Copy link
Author

loganpknudsen commented Jul 17, 2024 •
edited
Loading

Update: I have been able to reduce the error to be a procompiling error in oceananigans.jl:

ERROR: LoadError: Failed to precompile Oceananigans [9e8cae18-63c1-5223-a75c-80ca9d6e9a09] to "/glade/u/home/knudsenl/.julia/compiled/v1.9/Oceananigans/jl_AMNEzH".
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
   @ Base ./loading.jl:2300
 [3] compilecache
   @ ./loading.jl:2167 [inlined]
 [4] _require(pkg::Base.PkgId, env::String)
   @ Base ./loading.jl:1805
 [5] _require_prelocked(uuidkey::Base.PkgId, env::String)
   @ Base ./loading.jl:1660
 [6] macro expansion
   @ ./loading.jl:1648 [inlined]
 [7] macro expansion
   @ ./lock.jl:267 [inlined]
 [8] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1611
in expression starting at /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/testcode.jl:1

I have been trying to make sure that everything is up to date, and I am running on Julia version 1.9.2. My code is just

using Oceananingans

as I have been trying to get the library to load properly. Does anyone have any experience with this error or does it make things any clearer?

@vchuravy
Copy link
Collaborator

vchuravy commented Jul 17, 2024 via email

@loganpknudsen
Copy link
Author

Sorry I didn't realize I didn't get the full error message:

[52922] signal (11.1): Segmentation fault
in expression starting at /glade/u/home/knudsenl/.julia/packages/Oceananigans/M82LU/src/Oceananigans.jl:129
Allocations: 618273 (Pool: 617476; Big: 797); GC: 1
ERROR: LoadError: Failed to precompile Oceananigans [9e8cae18-63c1-5223-a75c-80ca9d6e9a09] to "/glade/u/home/knudsenl/.julia/compiled/v1.9/Oceananigans/jl_AMNEzH".
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
   @ Base ./loading.jl:2300
 [3] compilecache
   @ ./loading.jl:2167 [inlined]
 [4] _require(pkg::Base.PkgId, env::String)
   @ Base ./loading.jl:1805
 [5] _require_prelocked(uuidkey::Base.PkgId, env::String)
   @ Base ./loading.jl:1660
 [6] macro expansion
   @ ./loading.jl:1648 [inlined]
 [7] macro expansion
   @ ./lock.jl:267 [inlined]
 [8] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1611
in expression starting at /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/testcode.jl:1

The versioninfo gave me the following as output:

Julia Version 1.9.2
Commit e4ee485e90 (2023-07-05 09:39 UTC)
Platform Info:
  OS: Linux (x86_64-suse-linux)
      "openSUSE Leap 15.4"
  uname: Linux 5.14.21-150400.24.46-default #1 SMP PREEMPT_DYNAMIC Thu Feb 9 08:38:18 UTC 2023 (2d95137) x86_64 x86_64
  CPU: Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz: 
                 speed         user         nice          sys         idle          irq
       #1-72  2600 MHz  922553194 s     727867 s  363212152 s  5414020342 s          0 s
  Memory: 370.290340423584 GB (301775.56640625 MB free)
  Uptime: 9.37897786e6 sec
  Load Avg:  13.61  25.96  27.94
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, cascadelake)
  Threads: 1 on 72 virtual cores
Environment:
  LD_LIBRARY_PATH = /glade/u/apps/casper/23.10/spack/opt/spack/openmpi/4.1.6/oneapi/2023.2.1/dgcv/lib:/glade/u/apps/common/23.08/spack/opt/spack/cuda/12.2.1/lib64:/glade/u/apps/common/23.08/spack/opt/spack/cuda/12.2.1/nvvm/lib64:/glade/u/apps/common/23.08/spack/opt/spack/cuda/12.2.1/extras/CUPTI/lib64:/glade/u/apps/common/23.08/spack/opt/spack/cuda/12.2.1/extras/Debugger/lib64:/glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/lib:/glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/lib/x64:/glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/lib/oclfpga/host/linux64/lib:/glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/compiler/lib/intel64_lin:/glade/u/apps/casper/23.10/spack/opt/spack/hdf5/1.12.2/oneapi/2023.2.1/6vf2/lib
  NCAR_LDFLAGS_OPENMPI = /glade/u/apps/casper/23.10/spack/opt/spack/openmpi/4.1.6/oneapi/2023.2.1/dgcv/lib
  __LMOD_REF_COUNT_PATH = /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.9.2/gcc/7.5.0/mjea/bin:2;/glade/u/apps/casper/23.10/spack/opt/spack/netcdf/4.9.2/oneapi/2023.2.1/gfig/bin:2;/glade/u/apps/casper/23.10/spack/opt/spack/hdf5/1.12.2/oneapi/2023.2.1/6vf2/bin:2;/glade/u/apps/casper/23.10/spack/opt/spack/ncarcompilers/1.0.0/oneapi/2023.2.1/mai6/bin/mpi:1;/glade/u/apps/casper/23.10/spack/opt/spack/openmpi/4.1.6/oneapi/2023.2.1/dgcv/bin:2;/glade/u/apps/casper/23.10/spack/opt/spack/ucx/1.14.1/gcc/7.5.0/vf2h/bin:2;/glade/u/apps/common/23.08/spack/opt/spack/cuda/12.2.1/bin:2;/glade/u/apps/casper/23.10/spack/opt/spack/ncarcompilers/1.0.0/oneapi/2023.2.1/mai6/bin:2;/glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/lib/oclfpga/bin:1;/glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/bin/intel64:1;/glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/bin:1;/glade/u/apps/casper/23.10/opt/wrappers/bin:1;/glade/u/apps/casper/23.10/opt/utils/bin:1;/glade/u/home/knudsenl/.local/bin:1;/usr/lib/mit/bin:1;/opt/pbs/bin:1;/glade/u/apps/casper/23.10/opt/view/bin:1;/usr/local/bin:3;/usr/bin:3;/sbin:2;/bin:3;/glade/u/apps/casper/23.10/spack/opt/spack/llvm/14.0.6/gcc/7.5.0/arf7/bin:1
  __LMOD_REF_COUNT_NCAR_LDFLAGS_CUDA64 = /glade/u/apps/common/23.08/spack/opt/spack/cuda/12.2.1/lib64:1
  NCAR_LDFLAGS_UCX = /glade/u/apps/casper/23.10/spack/opt/spack/ucx/1.14.1/gcc/7.5.0/vf2h/lib
  __LMOD_REF_COUNT_ACL_BOARD_VENDOR_PATH = /opt/Intel/OpenCLFPGA/oneAPI/Boards:1
  ACL_BOARD_VENDOR_PATH = /opt/Intel/OpenCLFPGA/oneAPI/Boards
  __LMOD_REF_COUNT_NLSPATH = /glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/compiler/lib/intel64_lin/locale/%l_%t/%N:1
  JAVA_HOME = /usr/lib64/jvm/java-11-openjdk-11
  INSTALLPATH_ROOT = /glade/u/apps/casper/default/spack/opt/spack
  __LMOD_REF_COUNT_LD_LIBRARY_PATH = /glade/u/apps/casper/23.10/spack/opt/spack/openmpi/4.1.6/oneapi/2023.2.1/dgcv/lib:1;/glade/u/apps/common/23.08/spack/opt/spack/cuda/12.2.1/lib64:1;/glade/u/apps/common/23.08/spack/opt/spack/cuda/12.2.1/nvvm/lib64:1;/glade/u/apps/common/23.08/spack/opt/spack/cuda/12.2.1/extras/CUPTI/lib64:1;/glade/u/apps/common/23.08/spack/opt/spack/cuda/12.2.1/extras/Debugger/lib64:1;/glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/lib:1;/glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/lib/x64:1;/glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/lib/oclfpga/host/linux64/lib:1;/glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/compiler/lib/intel64_lin:1;/glade/u/apps/casper/23.10/spack/opt/spack/hdf5/1.12.2/oneapi/2023.2.1/6vf2/lib:1
  __LMOD_REF_COUNT_PKG_CONFIG_PATH = /glade/u/apps/casper/23.10/spack/opt/spack/netcdf/4.9.2/oneapi/2023.2.1/gfig/lib/pkgconfig:2;/glade/u/apps/casper/23.10/spack/opt/spack/hdf5/1.12.2/oneapi/2023.2.1/6vf2/lib/pkgconfig:2;/glade/u/apps/casper/23.10/spack/opt/spack/openmpi/4.1.6/oneapi/2023.2.1/dgcv/lib/pkgconfig:2;/glade/u/apps/casper/23.10/spack/opt/spack/ucx/1.14.1/gcc/7.5.0/vf2h/lib/pkgconfig:2;/glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/lib/pkgconfig:1;/glade/u/apps/casper/23.10/opt/view/lib64/pkgconfig:1;/glade/u/apps/casper/23.10/opt/view/lib/pkgconfig:1
  __LMOD_REF_COUNT_DIAGUTIL_PATH = /glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/sys_check/sys_check.sh:1
  NCAR_DEFAULT_MANPATH = /usr/local/share/man:/usr/share/man
  __LMOD_REF_COUNT_MODULEPATH = /glade/u/apps/casper/modules/environment:1;/glade/work/knudsenl/spack-downstreams/casper/modules/23.10/Core:1;/glade/u/apps/casper/modules/23.10/Core:1;/glade/work/knudsenl/spack-downstreams/casper/modules/23.10/oneapi/2023.2.1:1;/glade/u/apps/casper/modules/23.10/oneapi/2023.2.1:1;/glade/work/knudsenl/spack-downstreams/casper/modules/23.10/openmpi/4.1.6/oneapi/2023.2.1:1;/glade/u/apps/casper/modules/23.10/openmpi/4.1.6/oneapi/2023.2.1:1
  __LMOD_REF_COUNT_NCAR_LDFLAGS_INTEL_ONEAPI_COMPILERS = /glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/lib:1
  HOME = /glade/u/home/knudsenl
  CMAKE_PREFIX_PATH = /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.9.2/gcc/7.5.0/mjea:/glade/u/apps/casper/23.10/spack/opt/spack/netcdf/4.9.2/oneapi/2023.2.1/gfig:/glade/u/apps/casper/23.10/spack/opt/spack/hdf5/1.12.2/oneapi/2023.2.1/6vf2:/glade/u/apps/casper/23.10/spack/opt/spack/openmpi/4.1.6/oneapi/2023.2.1/dgcv:/glade/u/apps/casper/23.10/spack/opt/spack/ucx/1.14.1/gcc/7.5.0/vf2h:/glade/u/apps/common/23.08/spack/opt/spack/cuda/12.2.1:/glade/u/apps/casper/23.10/spack/opt/spack/ncarcompilers/1.0.0/oneapi/2023.2.1/mai6:/glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/IntelDPCPP:/glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1
  NCAR_LDFLAGS_0_COMMON64 = /glade/u/apps/casper/23.10/opt/view/lib64
  CUDA_HOME = /glade/u/apps/common/23.08/spack/opt/spack/cuda/12.2.1
  XNLSPATH = /usr/share/X11/nls
  CPATH = /glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/lib/oclfpga/include
  NCAR_DEFAULT_PATH = /usr/local/bin:/usr/bin:/sbin:/bin
  NCAR_LDFLAGS_NETCDF = /glade/u/apps/casper/23.10/spack/opt/spack/netcdf/4.9.2/oneapi/2023.2.1/gfig/lib
  __LMOD_REF_COUNT_INFOPATH = :1;/glade/u/apps/casper/23.10/opt/view/share/info:1;/usr/local/share/info:2;/usr/share/info:2
  SDK_HOME = /usr/lib64/jvm/java-11-openjdk-11
  NVHPC_CUDA_HOME = /glade/u/apps/common/23.08/spack/opt/spack/cuda/12.2.1
  NLSPATH = /glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/compiler/lib/intel64_lin/locale/%l_%t/%N
  __LMOD_REF_COUNT_NCAR_LDFLAGS_JULIA = /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.9.2/gcc/7.5.0/mjea/lib:1
  JDK_HOME = /usr/lib64/jvm/java-11-openjdk-11
  __LMOD_REF_COUNT_LIBRARY_PATH = /glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/compiler/lib/intel64_lin:1;/glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/lib:1
  LIBRARY_PATH = /glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/compiler/lib/intel64_lin:/glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/lib
  __LMOD_REF_COUNT_CMAKE_PREFIX_PATH = /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.9.2/gcc/7.5.0/mjea:1;/glade/u/apps/casper/23.10/spack/opt/spack/netcdf/4.9.2/oneapi/2023.2.1/gfig:1;/glade/u/apps/casper/23.10/spack/opt/spack/hdf5/1.12.2/oneapi/2023.2.1/6vf2:1;/glade/u/apps/casper/23.10/spack/opt/spack/openmpi/4.1.6/oneapi/2023.2.1/dgcv:1;/glade/u/apps/casper/23.10/spack/opt/spack/ucx/1.14.1/gcc/7.5.0/vf2h:1;/glade/u/apps/common/23.08/spack/opt/spack/cuda/12.2.1:1;/glade/u/apps/casper/23.10/spack/opt/spack/ncarcompilers/1.0.0/oneapi/2023.2.1/mai6:1;/glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/IntelDPCPP:1;/glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1:1
  __LMOD_REF_COUNT_NCAR_LDFLAGS_OPENMPI = /glade/u/apps/casper/23.10/spack/opt/spack/openmpi/4.1.6/oneapi/2023.2.1/dgcv/lib:1
  __LMOD_REF_COUNT_NCAR_LDFLAGS_NETCDF = /glade/u/apps/casper/23.10/spack/opt/spack/netcdf/4.9.2/oneapi/2023.2.1/gfig/lib:1
  DIAGUTIL_PATH = /glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/sys_check/sys_check.sh
  __LMOD_REF_COUNT_ACLOCAL_PATH = /usr/share/aclocal:1;/glade/u/apps/casper/23.10/opt/view/share/aclocal:1
  __LMOD_REF_COUNT_MANPATH = /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.9.2/gcc/7.5.0/mjea/share/man:2;/glade/u/apps/casper/23.10/spack/opt/spack/netcdf/4.9.2/oneapi/2023.2.1/gfig/share/man:2;/glade/u/apps/casper/23.10/spack/opt/spack/openmpi/4.1.6/oneapi/2023.2.1/dgcv/share/man:2;/glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/documentation/en/man/common:1;/usr/local/man:1;/opt/pbs/share/man:1;/glade/u/apps/casper/23.10/opt/view/man:1;/glade/u/apps/casper/23.10/opt/view/share/man:1;/usr/local/share/man:3;/usr/share/man:3
  __LMOD_REF_COUNT_NCAR_LDFLAGS_HDF5 = /glade/u/apps/casper/23.10/spack/opt/spack/hdf5/1.12.2/oneapi/2023.2.1/6vf2/lib:1
  __LMOD_REF_COUNT_NCAR_LDFLAGS_UCX = /glade/u/apps/casper/23.10/spack/opt/spack/ucx/1.14.1/gcc/7.5.0/vf2h/lib:1
  TERM = xterm-256color
  NCAR_LDFLAGS_0_COMMON = /glade/u/apps/casper/23.10/opt/view/lib
  NCAR_LDFLAGS_CUDA64 = /glade/u/apps/common/23.08/spack/opt/spack/cuda/12.2.1/lib64
  __LMOD_REF_COUNT_CPATH = /glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/lib/oclfpga/include:1
  CUDATOOLKIT_HOME = /glade/u/apps/common/23.08/spack/opt/spack/cuda/12.2.1
  ACLOCAL_PATH = /usr/share/aclocal:/glade/u/apps/casper/23.10/opt/view/share/aclocal
  MANPATH = /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.9.2/gcc/7.5.0/mjea/share/man:/glade/u/apps/casper/23.10/spack/opt/spack/netcdf/4.9.2/oneapi/2023.2.1/gfig/share/man:/glade/u/apps/casper/23.10/spack/opt/spack/openmpi/4.1.6/oneapi/2023.2.1/dgcv/share/man:/glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/documentation/en/man/common:/usr/local/man:/opt/pbs/share/man:/glade/u/apps/casper/23.10/opt/view/man:/glade/u/apps/casper/23.10/opt/view/share/man:/usr/local/share/man:/usr/share/man
  INTEL_PATH = /glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1
  MODULEPATH = /glade/u/apps/casper/modules/environment:/glade/work/knudsenl/spack-downstreams/casper/modules/23.10/Core:/glade/u/apps/casper/modules/23.10/Core:/glade/work/knudsenl/spack-downstreams/casper/modules/23.10/oneapi/2023.2.1:/glade/u/apps/casper/modules/23.10/oneapi/2023.2.1:/glade/work/knudsenl/spack-downstreams/casper/modules/23.10/openmpi/4.1.6/oneapi/2023.2.1:/glade/u/apps/casper/modules/23.10/openmpi/4.1.6/oneapi/2023.2.1
  NCAR_DEFAULT_INFOPATH = /usr/local/share/info:/usr/share/info
  MODULEPATH_ROOT = /glade/u/apps/casper/modules
  LMOD_PACKAGE_PATH = /glade/work/csgteam/spack-deployments/casper/23.10/envs/public/util
  JRE_HOME = /usr/lib64/jvm/java-11-openjdk-11
  PATH = /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.9.2/gcc/7.5.0/mjea/bin:/glade/u/apps/casper/23.10/spack/opt/spack/netcdf/4.9.2/oneapi/2023.2.1/gfig/bin:/glade/u/apps/casper/23.10/spack/opt/spack/hdf5/1.12.2/oneapi/2023.2.1/6vf2/bin:/glade/u/apps/casper/23.10/spack/opt/spack/ncarcompilers/1.0.0/oneapi/2023.2.1/mai6/bin/mpi:/glade/u/apps/casper/23.10/spack/opt/spack/openmpi/4.1.6/oneapi/2023.2.1/dgcv/bin:/glade/u/apps/casper/23.10/spack/opt/spack/ucx/1.14.1/gcc/7.5.0/vf2h/bin:/glade/u/apps/common/23.08/spack/opt/spack/cuda/12.2.1/bin:/glade/u/apps/casper/23.10/spack/opt/spack/ncarcompilers/1.0.0/oneapi/2023.2.1/mai6/bin:/glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/lib/oclfpga/bin:/glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/bin/intel64:/glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/bin:/glade/u/apps/casper/23.10/opt/wrappers/bin:/glade/u/apps/casper/23.10/opt/utils/bin:/glade/u/home/knudsenl/.local/bin:/usr/lib/mit/bin:/opt/pbs/bin:/glade/u/apps/casper/23.10/opt/view/bin:/usr/local/bin:/usr/bin:/sbin:/bin:/glade/u/apps/casper/23.10/spack/opt/spack/llvm/14.0.6/gcc/7.5.0/arf7/bin
  NCAR_LDFLAGS_INTEL_ONEAPI_COMPILERS = /glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/lib
  MODULESHOME = /glade/u/apps/casper/23.10/spack/opt/spack/lmod/8.7.24/gcc/7.5.0/m4jx/lmod/lmod
  PKG_CONFIG_PATH = /glade/u/apps/casper/23.10/spack/opt/spack/netcdf/4.9.2/oneapi/2023.2.1/gfig/lib/pkgconfig:/glade/u/apps/casper/23.10/spack/opt/spack/hdf5/1.12.2/oneapi/2023.2.1/6vf2/lib/pkgconfig:/glade/u/apps/casper/23.10/spack/opt/spack/openmpi/4.1.6/oneapi/2023.2.1/dgcv/lib/pkgconfig:/glade/u/apps/casper/23.10/spack/opt/spack/ucx/1.14.1/gcc/7.5.0/vf2h/lib/pkgconfig:/glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/lib/pkgconfig:/glade/u/apps/casper/23.10/opt/view/lib64/pkgconfig:/glade/u/apps/casper/23.10/opt/view/lib/pkgconfig
  INFOPATH = :/glade/u/apps/casper/23.10/opt/view/share/info:/usr/local/share/info:/usr/share/info
  NCAR_LDFLAGS_JULIA = /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.9.2/gcc/7.5.0/mjea/lib
  NCAR_LDFLAGS_HDF5 = /glade/u/apps/casper/23.10/spack/opt/spack/hdf5/1.12.2/oneapi/2023.2.1/6vf2/lib
  NCAR_WRAPPER_MPI_PATH = /glade/u/apps/casper/23.10/spack/opt/spack/ncarcompilers/1.0.0/oneapi/2023.2.1/mai6/bin/mpi
  __LMOD_STACK_LMOD_PACKAGE_PATH = L2dsYWRlL3dvcmsvY3NndGVhbS9zcGFjay1kZXBsb3ltZW50cy9jYXNwZXIvMjMuMTAvZW52cy9wdWJsaWMvdXRpbA==:L2dsYWRlL3dvcmsvY3NndGVhbS9zcGFjay1kZXBsb3ltZW50cy9jYXNwZXIvMjMuMTAvZW52cy9wdWJsaWMvdXRpbA==

and the status check message gave me:

Status `~/.julia/environments/v1.9/Manifest.toml`
  [621f4979] AbstractFFTs v1.5.0
  [79e6a3ab] Adapt v4.0.4
⌅ [4fba245c] ArrayInterface v7.7.1
  [a9b6321e] Atomix v0.1.0
  [ab4f0b2a] BFloat16s v0.5.0
  [fa961155] CEnum v0.5.0
  [179af706] CFTime v0.1.3
  [052768ef] CUDA v5.4.3
  [1af6417a] CUDA_Runtime_Discovery v0.3.4
  [d360d2e6] ChainRulesCore v1.24.0
  [3da002f7] ColorTypes v0.11.5
  [5ae59095] Colors v0.12.11
  [1fbeeb36] CommonDataModel v0.3.6
  [34da2185] Compat v4.15.0
  [187b0558] ConstructionBase v1.5.5
  [a8cc5b0e] Crayons v4.1.1
  [7445602f] CubedSphere v0.2.5
  [9a962f9c] DataAPI v1.16.0
  [a93c6f00] DataFrames v1.6.1
  [864edb3b] DataStructures v0.18.20
  [e2d170a0] DataValueInterfaces v1.0.0
⌅ [3c3547ce] DiskArrays v0.3.23
  [b4f34e82] Distances v0.10.11
  [ffbed154] DocStringExtensions v0.9.3
  [b305315f] Elliptic v1.0.1
  [e2ba6199] ExprTools v0.1.10
  [7a1cc6ca] FFTW v1.8.0
  [5789e2e9] FileIO v1.16.3
  [53c48c17] FixedPointNumbers v0.8.5
  [0c68f7d7] GPUArrays v10.2.3
  [46192b85] GPUArraysCore v0.1.6
  [61eb1bfa] GPUCompiler v0.26.7
  [c27321d9] Glob v1.3.1
  [615f187c] IfElse v0.1.1
  [40713840] IncompleteLU v0.2.1
  [842dd82b] InlineStrings v1.4.2
  [41ab1584] InvertedIndices v1.3.0
  [92d709cd] IrrationalConstants v0.2.2
  [42fd0dbc] IterativeSolvers v0.9.4
  [82899510] IteratorInterfaceExtensions v1.0.0
  [033835bb] JLD2 v0.4.50
  [692b3bcd] JLLWrappers v1.5.0
  [0f8b85d8] JSON3 v1.14.0
  [63c18a36] KernelAbstractions v0.9.22
  [929cbde3] LLVM v8.0.0
  [8b046642] LLVMLoopInfo v1.0.0
  [8ac3fa9e] LRUCache v1.6.1
  [b964fa9f] LaTeXStrings v1.3.1
  [2ab3a3ac] LogExpFunctions v0.3.28
⌃ [da04e1cc] MPI v0.20.16
  [3da0fdf6] MPIPreferences v0.1.11
  [1914dd2f] MacroTools v0.5.13
  [e1d29d7a] Missings v1.2.0
  [85f8d34a] NCDatasets v0.14.4
  [5da4648a] NVTX v0.3.4
  [9e8cae18] Oceananigans v0.91.4
  [6fe1bfb0] OffsetArrays v1.14.1
  [bac558e1] OrderedCollections v1.6.3
  [65ce6f38] PackageExtensionCompat v1.0.2
  [69de0a69] Parsers v2.8.1
  [0e08944d] PencilArrays v0.19.5
  [4a48f351] PencilFFTs v0.15.1
  [eebad327] PkgVersion v0.3.3
  [2dfb63ee] PooledArrays v1.4.3
  [aea7be01] PrecompileTools v1.2.1
  [21216c6a] Preferences v1.4.3
  [08abe8d2] PrettyTables v2.3.2
  [49802e3a] ProgressBars v1.5.1
  [94ee1d12] Quaternions v0.7.6
  [74087812] Random123 v1.7.0
  [e6cf234a] RandomNumbers v1.5.3
  [c1ae055f] RealDot v0.1.0
  [3cdcf5f2] RecipesBase v1.3.4
  [189a3867] Reexport v1.2.2
  [ae029012] Requires v1.3.0
  [6038ab10] Rotations v1.7.1
  [6c6a2e73] Scratch v1.2.1
  [d496a93d] SeawaterPolynomials v0.3.4
  [91c51154] SentinelArrays v1.4.5
  [a2af1166] SortingAlgorithms v1.2.1
  [276daf66] SpecialFunctions v2.4.0
⌅ [aedffcd0] Static v0.8.9
  [0d7ed370] StaticArrayInterface v1.5.1
  [90137ffa] StaticArrays v1.9.7
  [1e83bf80] StaticArraysCore v1.4.3
  [15972242] StaticPermutations v0.3.0
  [82ae8749] StatsAPI v1.7.0
  [5e0ebb24] Strided v2.1.0
  [4db3bf67] StridedViews v0.3.1
  [892a3eda] StringManipulation v0.3.4
  [09ab397b] StructArrays v0.6.18
  [856f2bd8] StructTypes v1.10.0
  [3783bdb8] TableTraits v1.0.1
  [bd369af6] Tables v1.12.0
⌅ [6aa5eb33] TaylorSeries v0.16.0
  [a759f4b9] TimerOutputs v0.5.24
  [3bb67fe8] TranscodingStreams v0.11.0
  [24ddb15e] TransmuteDims v0.1.16
  [9d95972d] TupleTools v1.5.0
  [013be700] UnsafeAtomics v0.2.1
  [d80eeb9a] UnsafeAtomicsLLVM v0.1.5
  [81def892] VersionParsing v1.3.0
  [0b7ba130] Blosc_jll v1.21.5+0
  [6e34b625] Bzip2_jll v1.0.8+1
  [4ee394cb] CUDA_Driver_jll v0.9.1+1
  [76a88914] CUDA_Runtime_jll v0.14.1+0
  [f5851436] FFTW_jll v3.3.10+0
  [0951126a] GnuTLS_jll v3.8.4+0
⌃ [0234f1f7] HDF5_jll v1.14.2+1
  [e33a78d0] Hwloc_jll v2.11.0+0
  [1d5cc7b8] IntelOpenMP_jll v2024.2.0+0
  [9c1d0b0a] JuliaNVTXCallbacks_jll v0.2.1+0
  [dad2f222] LLVMExtra_jll v0.0.30+0
  [1d63c593] LLVMOpenMP_jll v15.0.7+0
  [94ce4f54] Libiconv_jll v1.17.0+0
  [5ced341a] Lz4_jll v1.9.4+0
  [856f044c] MKL_jll v2024.2.0+0
  [7cb0a576] MPICH_jll v4.2.1+1
  [f1f71cc9] MPItrampoline_jll v5.4.0+0
  [9237b28f] MicrosoftMPI_jll v10.1.4+2
  [e98f9f5b] NVTX_jll v3.1.0+2
⌃ [7243133f] NetCDF_jll v400.902.209+0
⌅ [4c82536e] Nettle_jll v3.7.2+0
  [fe0851c0] OpenMPI_jll v5.0.3+0
  [458c3c95] OpenSSL_jll v3.0.14+0
  [efe28fd5] OpenSpecFun_jll v0.5.5+0
  [c2071276] P11Kit_jll v0.24.1+0
  [02c8fc9c] XML2_jll v2.13.1+0
  [ffd25f8a] XZ_jll v5.4.6+0
  [3161d3a3] Zstd_jll v1.5.6+0
  [477f73a3] libaec_jll v1.1.2+0
  [337d8026] libzip_jll v1.10.1+0
  [1317d2d5] oneTBB_jll v2021.12.0+0
  [0dad84c5] ArgTools v1.1.1
  [56f22d72] Artifacts
  [2a0f44e3] Base64
  [ade2ca70] Dates
  [8ba89e20] Distributed
  [f43a241f] Downloads v1.6.0
  [7b1f6079] FileWatching
  [9fa8497b] Future
  [b77e0a4c] InteractiveUtils
  [4af54fe1] LazyArtifacts
  [b27032c2] LibCURL v0.6.3
  [76f85450] LibGit2
  [8f399da3] Libdl
  [37e2e46d] LinearAlgebra
  [56ddb016] Logging
  [d6f4376e] Markdown
  [a63ad114] Mmap
  [ca575930] NetworkOptions v1.2.0
  [44cfe95a] Pkg v1.9.2
  [de0858da] Printf
  [3fa0cd96] REPL
  [9a3f8284] Random
  [ea8e919c] SHA v0.7.0
  [9e88b42a] Serialization
  [6462fe0b] Sockets
  [2f01184e] SparseArrays
  [10745b16] Statistics v1.9.0
  [4607b0f0] SuiteSparse
  [fa267f1f] TOML v1.0.3
  [a4e569a6] Tar v1.10.0
  [8dfed614] Test
  [cf7118a7] UUIDs
  [4ec0a83e] Unicode
  [e66e0078] CompilerSupportLibraries_jll v1.0.5+0
  [781609d7] GMP_jll v6.2.1+2
  [deac9b47] LibCURL_jll v7.84.0+0
  [29816b5a] LibSSH2_jll v1.10.2+0
  [c8ffd9c3] MbedTLS_jll v2.28.2+0
  [14a3606d] MozillaCACerts_jll v2022.10.11
  [4536629a] OpenBLAS_jll v0.3.21+4
  [05823500] OpenLibm_jll v0.8.1+0
  [bea87d4a] SuiteSparse_jll v5.10.1+6
  [83775a58] Zlib_jll v1.2.13+0
  [8e850b90] libblastrampoline_jll v5.8.0+0
  [8e850ede] nghttp2_jll v1.48.0+0
  [3f19e933] p7zip_jll v17.4.0+0
Info Packages marked with ⌃ and ⌅ have new versions available, but those with ⌅ are restricted by compatibility constraints from upgrading. To see why use `status --outdated -m`

@loganpknudsen
Copy link
Author

I don't know if this helps to make the issue clearer, but I just re-ran the code for just

using Oceananigans

and it executed without error. I ran it again to see if it was just a fluke and I got the following error:


The following have been reloaded with a version change:
  1) cuda/12.2.1 => cuda/11.8.0


[59837] signal (11.1): Segmentation fault
in expression starting at /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/testcode.jl:1
Allocations: 605144 (Pool: 604223; Big: 921); GC: 1
/var/spool/pbs/mom_priv/jobs/1741845.casper-pbs.SC: line 31: 59837 Segmentation fault      (core dumped) julia --project testcode.jl /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/

Could it be an issue with the computer loading Oceananigans.jl with a bunch of extra functions?

@loganpknudsen
Copy link
Author

More information, I tried changing Oceananigans.jl builds by running in Julia

] add https://github.com/climate-machine/Oceananigans.jl.git

and I ran my code and it loaded fine the first time, but again it had a segmentation fault on the rerun. Similarly, when I executed

] add Oceananigans

it ran the first time then got a segmentation fault on the second run again. Could this not be an issue with Oceananigans.jl but with how I am running my code on the GPU?

@glwagner
Copy link
Member

Just to clarify things a bit for you @loganpknudsen --- your error says

ERROR: LoadError: Failed to precompile Oceananigans

this means that the segmentation fault occurs during precompilation of Oceananigans, which occurs before any code you have written executes. That's why the error comes from "line 1" of your script (is that where you write using Oceananigans?)

Another clue is the text

[52922] signal (11.1): Segmentation fault
in expression starting at /glade/u/home/knudsenl/.julia/packages/Oceananigans/M82LU/src/Oceananigans.jl:129

That says the error comes from line 129 in the file src/Oceananigans.jl. Going to that line on main branch we find:

Oceananigans.jl/src/Oceananigans.jl

Line 129 in d6e63e5

using CUDA

so there's something wrong with your CUDA / how it's loaded.

The best course of action is probably to update to julia 1.10 first of all rather than using julia 1.9.2.

Next, see if you can simply write using CUDA rather than trying to run your whole script. If you can get that to work (better yet if you can use some of CUDA.jl on a GPU on derecho then move on to simply writing using Oceananigans and trying to build a grid on the GPU. If that succeeds move on to your script.

@loganpknudsen
Copy link
Author

Thank you for your help! I have installed the updated version of Julia and when running it appears to be Segfaulting when trying to add CUDA. I will open an issue with them if I cannot figure out how to solve it. Thank you again

@tomchor
Copy link
Collaborator

tomchor commented Jul 23, 2024

Thank you for your help! I have installed the updated version of Julia and when running it appears to be Segfaulting when trying to add CUDA. I will open an issue with them if I cannot figure out how to solve it. Thank you again

@logan can you link the CUDA issue here please?

Also, if you haven't tried, erase everything on your $JULIA_DEPOT_PATH (really to be safe you should make a backup of everything there first) and then run the same script you've been running with using Pkg; Pkg.instantiate() on the first line. This should "re-install" all the packages from scratch.

For context $JULIA_DEPOT_PATH by default is ~/.julia, I think, and that's where julia stores the package files it downloads. The hypothesis here is that at some point some package download/compilation went wrong and you have some broken code there.

@tomchor
Copy link
Collaborator

tomchor commented Jul 23, 2024 •
edited
Loading

Okay, here's the CUDA issue: JuliaGPU/CUDA.jl#2453

@loganpknudsen
Copy link
Author

So I removed ~/.julia and ran the instantiate before and I got the following output

[45606] signal (11.1): Segmentation fault
in expression starting at /glade/u/home/knudsenl/.julia/packages/CUDA/Tl08O/src/CUDA.jl:25
Allocations: 2907 (Pool: 2898; Big: 9); GC: 0
  Installing known registries into `~/.julia`
┌ Warning: The active manifest file has dependencies that were resolved with a different julia version (1.9.2). Unexpected behavior may occur.
└ @ /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/Manifest.toml:0
   Installed GPUArraysCore ─────────────── v0.1.6
   Installed Crayons ───────────────────── v4.1.1
   Installed Adapt ─────────────────────── v4.0.4
   Installed TableTraits ───────────────── v1.0.1
   Installed ColorTypes ────────────────── v0.11.5
   Installed Scratch ───────────────────── v1.2.1
   Installed Preferences ───────────────── v1.4.3
   Installed PrettyTables ──────────────── v2.3.2
   Installed SentinelArrays ────────────── v1.4.5
   Installed CUDA_Driver_jll ───────────── v0.9.1+1
   Installed DataAPI ───────────────────── v1.16.0
   Installed PooledArrays ──────────────── v1.4.3
   Installed Tables ────────────────────── v1.12.0
   Installed FixedPointNumbers ─────────── v0.8.5
   Installed GPUCompiler ───────────────── v0.26.7
   Installed LLVMLoopInfo ──────────────── v1.0.0
   Installed IteratorInterfaceExtensions ─ v1.0.0
   Installed StaticArraysCore ──────────── v1.4.3
   Installed JLLWrappers ───────────────── v1.5.0
   Installed InlineStrings ─────────────── v1.4.2
   Installed StaticArrays ──────────────── v1.9.7
   Installed TimerOutputs ──────────────── v0.5.24
   Installed AbstractFFTs ──────────────── v1.5.0
   Installed PrecompileTools ───────────── v1.2.1
   Installed DataValueInterfaces ───────── v1.0.0
   Installed OrderedCollections ────────── v1.6.3
   Installed NVTX_jll ──────────────────── v3.1.0+2
   Installed CUDA_Runtime_Discovery ────── v0.3.4
   Installed LLVMExtra_jll ─────────────── v0.0.30+0
   Installed UnsafeAtomicsLLVM ─────────── v0.1.5
   Installed LaTeXStrings ──────────────── v1.3.1
   Installed NVTX ──────────────────────── v0.3.4
   Installed CEnum ─────────────────────── v0.5.0
   Installed Random123 ─────────────────── v1.7.0
   Installed JuliaNVTXCallbacks_jll ────── v0.2.1+0
   Installed InvertedIndices ───────────── v1.3.0
   Installed BFloat16s ─────────────────── v0.5.0
   Installed Reexport ──────────────────── v1.2.2
   Installed CUDA_Runtime_jll ──────────── v0.14.1+0
   Installed GPUArrays ─────────────────── v10.2.3
   Installed RandomNumbers ─────────────── v1.5.3
   Installed DataFrames ────────────────── v1.6.1
   Installed DataStructures ────────────── v0.18.20
   Installed Compat ────────────────────── v4.15.0
   Installed Requires ──────────────────── v1.3.0
   Installed ExprTools ─────────────────── v0.1.10
   Installed MacroTools ────────────────── v0.5.13
   Installed Colors ────────────────────── v0.12.11
   Installed KernelAbstractions ────────── v0.9.22
   Installed CUDA ──────────────────────── v5.4.3
   Installed Missings ──────────────────── v1.2.0
   Installed StringManipulation ────────── v0.3.4
   Installed UnsafeAtomics ─────────────── v0.2.1
   Installed SortingAlgorithms ─────────── v1.2.1
   Installed Atomix ────────────────────── v0.1.0
   Installed LLVM ──────────────────────── v8.0.0
Precompiling project...
  ✓ LLVMLoopInfo
  ✓ DataValueInterfaces
  ✓ Reexport
  ✓ IteratorInterfaceExtensions
  ✓ LaTeXStrings
  ✓ InvertedIndices
  ✓ ExprTools
  ✓ DataAPI
  ✓ Requires
  ✓ CompilerSupportLibraries_jll
  ✓ OrderedCollections
  ✓ Compat
  ✓ AbstractFFTs
  ✓ InlineStrings
  ✓ Scratch
  ✓ CEnum
  ✓ StaticArraysCore
  ✓ TableTraits
  ✓ Missings
  ✓ PooledArrays
  ✓ BFloat16s
  ✓ Preferences

  ✓ Adapt
  ✓ Statistics
  ✓ SentinelArrays
  ✓ Compat → CompatLinearAlgebraExt
  ✓ Crayons
  ✓ UnsafeAtomics
  ✓ CUDA_Runtime_Discovery
  ✓ PrecompileTools
  ✓ JLLWrappers
  ✓ Tables
  ✓ RandomNumbers
  ✓ Atomix
  ✓ TimerOutputs
  ✓ AbstractFFTs → AbstractFFTsTestExt
  ✓ NVTX_jll
  ✓ JuliaNVTXCallbacks_jll
  ✓ MacroTools
  ✓ LLVMExtra_jll
  ✓ CUDA_Driver_jll
  ✓ Random123
✓ DataStructures
  ✓ StringManipulation
  ✓ FixedPointNumbers
  ✓ SortingAlgorithms
  ✗ CUDA_Runtime_jll
  ✓ ColorTypes
  ✓ LLVM
  ✓ LLVM → BFloat16sExt
  ✓ StaticArrays
  ✓ Adapt → AdaptStaticArraysExt
  ✓ StaticArrays → StaticArraysStatisticsExt
  ✓ UnsafeAtomicsLLVM
  ✓ Colors
  ✓ GPUArraysCore
  ✓ NVTX

  ✓ GPUArrays
  ✓ KernelAbstractions
  ✓ PrettyTables
  ✓ GPUCompiler
  ✓ DataFrames
  ✗ CUDA
  61 dependencies successfully precompiled in 190 seconds. 5 already precompiled.

The following 1 direct dependency failed to precompile:

CUDA [052768ef-5323-5732-b1bb-66c8b64840ba]

Failed to precompile CUDA [052768ef-5323-5732-b1bb-66c8b64840ba] to "/glade/u/home/knudsenl/.julia/compiled/v1.10/CUDA/jl_UQIv2i".
[45592] signal (11.1): Segmentation fault
in expression starting at /glade/u/home/knudsenl/.julia/packages/CUDA/Tl08O/src/CUDA.jl:25
Allocations: 2907 (Pool: 2898; Big: 9); GC: ERROR: LoadError: Failed to precompile CUDA [052768ef-5323-5732-b1bb-66c8b64840ba] to "/glade/u/home/knudsenl/.julia/compiled/v1.10/CUDA/jl_CUC33l".
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
    @ Base ./loading.jl:2468
  [3] compilecache
    @ ./loading.jl:2340 [inlined]
  [4] (::Base.var"#968#969"{Base.PkgId})()
    @ Base ./loading.jl:1974
  [5] mkpidlock(f::Base.var"#968#969"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
    @ FileWatching.Pidfile /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:93
  [6] #mkpidlock#6
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:88 [inlined]
  [7] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
    @ FileWatching.Pidfile /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:111
[8] #invokelatest#2
    @ ./essentials.jl:894 [inlined]
  [9] invokelatest
    @ ./essentials.jl:889 [inlined]
 [10] maybe_cachefile_lock(f::Base.var"#968#969"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
    @ Base ./loading.jl:2983
 [11] maybe_cachefile_lock
    @ ./loading.jl:2980 [inlined]
 [12] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1970
 [13] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1812
 [14] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [15] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [16] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1803
 [17] macro expansion
    @ ./loading.jl:1790 [inlined]
 [18] macro expansion
    @ ./lock.jl:267 [inlined]
 [19] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1753
 [20] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [21] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [22] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1746
in expression starting at /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/testcode.jl:3

@loganpknudsen
Copy link
Author

Something else I tried was deleting the Project.toml and Manifest.toml file as well as ~/.julia, and after running the following:

using Pkg
Pkg.activate(".")
Pkg.add("CUDA")
Pkg.instantiate()
using CUDA

I got:

[57577] signal (11.1): Segmentation fault
in expression starting at /glade/u/home/knudsenl/.julia/packages/CUDA_Runtime_jll/YgJCI/.pkg/platform_augmentation.jl:210
Allocations: 2907 (Pool: 2898; Big: 9); GC: 0
  Activating new project at `/glade/derecho/scratch/knudsenl/BottomBoundaryLayer`
  Installing known registries into `~/.julia`
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
   Installed GPUArraysCore ─────────────── v0.1.6
   Installed Crayons ───────────────────── v4.1.1
   Installed TableTraits ───────────────── v1.0.1
   Installed ColorTypes ────────────────── v0.11.5
   Installed Adapt ─────────────────────── v4.0.4
   Installed Scratch ───────────────────── v1.2.1
   Installed Preferences ───────────────── v1.4.3
   Installed PrettyTables ──────────────── v2.3.2
   Installed CUDA_Driver_jll ───────────── v0.9.1+1
   Installed SentinelArrays ────────────── v1.4.5
   Installed DataAPI ───────────────────── v1.16.0
   Installed PooledArrays ──────────────── v1.4.3
   Installed FixedPointNumbers ─────────── v0.8.5
   Installed Tables ────────────────────── v1.12.0
   Installed LLVMLoopInfo ──────────────── v1.0.0
   Installed GPUCompiler ───────────────── v0.26.7
   Installed StaticArraysCore ──────────── v1.4.3
   Installed TimerOutputs ──────────────── v0.5.24
   Installed JLLWrappers ───────────────── v1.5.0
   Installed InlineStrings ─────────────── v1.4.2
   Installed AbstractFFTs ──────────────── v1.5.0
   Installed IteratorInterfaceExtensions ─ v1.0.0
   Installed StaticArrays ──────────────── v1.9.7
   Installed PrecompileTools ───────────── v1.2.1
   Installed DataValueInterfaces ───────── v1.0.0
   Installed CUDA_Runtime_Discovery ────── v0.3.4
   Installed NVTX_jll ──────────────────── v3.1.0+2
   Installed LLVMExtra_jll ─────────────── v0.0.30+0
   Installed LaTeXStrings ──────────────── v1.3.1
   Installed OrderedCollections ────────── v1.6.3
   Installed NVTX ──────────────────────── v0.3.4
   Installed UnsafeAtomicsLLVM ─────────── v0.1.5
   Installed CEnum ─────────────────────── v0.5.0
   Installed InvertedIndices ───────────── v1.3.0
   Installed Reexport ──────────────────── v1.2.2
   Installed JuliaNVTXCallbacks_jll ────── v0.2.1+0
   Installed BFloat16s ─────────────────── v0.5.0
   Installed MacroTools ────────────────── v0.5.13
   Installed DataStructures ────────────── v0.18.20
   Installed Colors ────────────────────── v0.12.11
   Installed KernelAbstractions ────────── v0.9.22
   Installed RandomNumbers ─────────────── v1.5.3
   Installed Missings ──────────────────── v1.2.0
   Installed Compat ────────────────────── v4.15.0
   Installed StringManipulation ────────── v0.3.4
   Installed SortingAlgorithms ─────────── v1.2.1
   Installed UnsafeAtomics ─────────────── v0.2.1
   Installed Atomix ────────────────────── v0.1.0
   Installed LLVM ──────────────────────── v8.0.0
   Installed CUDA ──────────────────────── v5.4.3
    Updating `/glade/derecho/scratch/knudsenl/BottomBoundaryLayer/Project.toml`
  [052768ef] + CUDA v5.4.3
    Updating `/glade/derecho/scratch/knudsenl/BottomBoundaryLayer/Manifest.toml`
ERROR: LoadError: failed process: Process(`/glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/bin/julia -C native -J/glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/lib/julia/sys.so -g1 -O0 --color=no --history-file=no --startup-file=no --project=/glade/derecho/scratch/knudsenl/BottomBoundaryLayer/Project.toml --eval 'append!(empty!(Base.DEPOT_PATH), ["/glade/u/home/knudsenl/.julia", "/glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/local/share/julia", "/glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia"])
append!(empty!(Base.DL_LOAD_PATH), String[])

cd("/glade/u/home/knudsenl/.julia/packages/CUDA_Runtime_jll/YgJCI/.pkg")
include("/glade/u/home/knudsenl/.julia/packages/CUDA_Runtime_jll/YgJCI/.pkg/select_artifacts.jl")
' -t1 --startup-file=no x86_64-linux-gnu-libgfortran5-cxx11-libstdcxx30-julia_version+1.10.2`, ProcessSignaled(11)) [0]

Stacktrace:
  [1] pipeline_error
    @ ./process.jl:565 [inlined]
  [2] read(cmd::Cmd)
    @ Base ./process.jl:449
  [3] collect_artifacts(pkg_root::String; platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:720
  [4] collect_artifacts
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:706 [inlined]
  [5] check_artifacts_downloaded(pkg_root::String; platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:764
  [6] check_artifacts_downloaded
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:763 [inlined]
  [7] is_package_downloaded(manifest_file::String, pkg::Pkg.Types.PackageSpec; platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2195
  [8] is_package_downloaded
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2190 [inlined]
  [9] print_status(env::Pkg.Types.EnvCache, old_env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, header::Symbol, uuids::Vector{Base.UUID}, names::Vector{String}; manifest::Bool, diff::Bool, ignore_indent::Bool, outdated::Bool, extensions::Bool, io::IOStream, mode::Pkg.Types.PackageMode, hidden_upgrades_info::Bool, show_usagetips::Bool)
    @ Pkg.Operations /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2308
 [10] print_status
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2240 [inlined]
 [11] status(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}; header::Symbol, mode::Pkg.Types.PackageMode, git_diff::Bool, env_diff::Pkg.Types.EnvCache, ignore_indent::Bool, io::IOStream, outdated::Bool, extensions::Bool, hidden_upgrades_info::Bool, show_usagetips::Bool)
    @ Pkg.Operations /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2474
 [12] status (repeats 2 times)
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2442 [inlined]
 [13] #show_update#167
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2438 [inlined]
 [14] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, new_git::Set{Base.UUID}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:1397
 [15] add
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:1376 [inlined]
 [16] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform, kwargs::@Kwargs{io::IOStream})
    @ Pkg.API /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/API.jl:278
 [17] add(pkgs::Vector{Pkg.Types.PackageSpec}; io::IOStream, kwargs::@Kwargs{})
    @ Pkg.API /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/API.jl:159
 [18] add(pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.API /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/API.jl:148
 [19] add
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/API.jl:147 [inlined]
 [20] add(pkg::String)
    @ Pkg.API /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/API.jl:146
 [21] top-level scope
    @ /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/testcode.jl:3
in expression starting at /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/testcode.jl:3

@loganpknudsen
Copy link
Author

Thank you to everyone who took the time to respond to this issue! I am still unclear on what exactly was driving the error I was getting, but I was able to fix it by creating a new environment in a clean directory and running the following modules on Casper for the UCAR HPC resources:

module --force purge
module --ignore-cache load ncarenv/23.10 gcc ncarcompilers netcdf
module --ignore-cache load cuda
module --ignore-cache load julia/1.10.2

and then running my script. Thanks again for all the help!

@tomchor
Copy link
Collaborator

tomchor commented Jul 29, 2024 •
edited
Loading

Thanks for posting the solution. What does the ignore-cache flag do? And how did you get to it?

@loganpknudsen
Copy link
Author

I am not certain what the ignore-cache flag does, I borrowed the module loading sequence. I am pretty it ignores previously loaded modules with the same or similar name.

@glwagner
Copy link
Member

What was the failing workflow / how was it different than this?

@loganpknudsen
Copy link
Author

The module loading sequence I was running was before that didn't work were

module --force purge
module load ncarenv/23.10 gcc/12.2.0
module load ncarcompilers/1.0.0
module load netcdf/4.9.2 openmpi/4.1.6 
module load julia/1.9.2
module load cuda/12.2.1

if needed I can see if I can find the versions of the modules that ended up working. There are no other major difference in how I load in the packages or Julia. Let me know if you have any questions

@glwagner
Copy link
Member

Okay interesting!

I'm not sure why you would need gcc. Have you tried omitting that?

Also, did you try installing julia yourself? There could be some benefit in using the system Julia via module load julia/1.9.2, but this would mostly have to do with compilation speed (not execution speed) I think. If you have found discrepancy between the system Julia vs your own Julia install that'd be interesting to hear about...

You could also let julia handle netcdf --- again, I'm not sure if this would have performance implications or not, but it might provide faster way to get up and running if you are simply trying to run scripts. Julia's package manager should be able to automatically detect and install netcdf appropriate for the system, I think.

@loganpknudsen
Copy link
Author

I have been running on the system Julia for these, I attempted to do my own install but I do not think I ended up doing much with it. I believe gcc is a dependency for the subsequent modules as for example when I run:

module --force purge
module load ncarenv/23.10 
module load ncarcompilers/1.0.0

the last line returns

Lmod has detected the following error:  These module(s) or
extension(s) exist but cannot be loaded as requested: "ncarcompilers/1.0.0"
   Try: "module spider ncarcompilers/1.0.0" to see how to load the module(s).

which does not happen if I include gcc. I was also able to run my code without netcdf!

@glwagner
Copy link
Member

Ok, good to know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@vchuravy @tomchor @glwagner @loganpknudsen