Obscure GPU error when time stepping a model with Julia 1.5 and Oceananigans v0.33.0 #828

ali-ramadhan · 2020-08-03T14:17:33Z

ERROR: InvalidIRError: compiling kernel gpu_calculate_pressure_right_hand_side!(Cassette.Context{nametype(CUDACtx),KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.StaticSize{(16, 16, 16)},KernelAbstractions.NDIteration.DynamicCheck,Nothing,Nothing,KernelAbstractions.NDIteration.NDRange{3,KernelAbstractions.NDIteration.StaticSize{(1, 1, 16)},KernelAbstractions.NDIteration.StaticSize{(16, 16, 1)},Nothing,Nothing}},Nothing,KernelAbstractions.var"##PassType#253",Nothing,Cassette.DisableHooks}, typeof(Oceananigans.Solvers.gpu_calculate_pressure_right_hand_side!), CUDA.CuDeviceArray{Complex{Float64},3,CUDA.AS.Global}, Oceananigans.Solvers.HorizontallyPeriodic, GPU, RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}}, Int64, NamedTuple{(:u, :v, :w),Tuple{OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}},OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}},OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}}}}) resulted in invalid LLVM IR                                                                                                                                                         
Reason: unsupported call to the Julia runtime (call to jl_f_tuple)

Might have something to do with the new tuple/named tuple syntax but I thought Julia 1.4 -> 1.5 wasn't supposed to introduce any breaking changes? https://julialang.org/blog/2020/08/julia-1.5-highlights/#implicit_keyword_argument_values

Julia 1.4.2

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.4.2 (2020-05-23)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using Oceananigans; grid = RegularCartesianGrid(size=(16, 16, 16), extent=(1, 1, 1)); model = IncompressibleModel(architecture=GPU(), grid=grid); time_step!(model, 1)
[ Info: Precompiling Oceananigans [9e8cae18-63c1-5223-a75c-80ca9d6e9a09]
CUDA-enabled GPU(s) detected:
CUDA.CuDevice(0)
CUDA.CuDevice(1)
CUDA.CuDevice(2)
CUDA.CuDevice(3)
┌ Warning: Performing scalar operations on GPU arrays: This is very slow, consider disallowing these operations with `allowscalar(false)`
└ @ GPUArrays ~/.julia/packages/GPUArrays/4W5rW/src/host/indexing.jl:43

Julia 1.5.0

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.5.0 (2020-08-01)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using Oceananigans; grid = RegularCartesianGrid(size=(16, 16, 16), extent=(1, 1, 1)); model = IncompressibleModel(architecture=GPU(), grid=grid); time_step!(model, 1)
┌ Warning: Performing scalar operations on GPU arrays: This is very slow, consider disallowing these operations with `allowscalar(false)`
└ @ GPUArrays ~/.julia/packages/GPUArrays/4W5rW/src/host/indexing.jl:43
ERROR: InvalidIRError: compiling kernel gpu_calculate_pressure_right_hand_side!(Cassette.Context{nametype(CUDACtx),KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.StaticSize{(16, 16, 16)},KernelAbstractions.NDIteration.DynamicCheck,Nothing,Nothing,KernelAbstractions.NDIteration.NDRange{3,KernelAbstractions.NDIteration.StaticSize{(1, 1, 16)},KernelAbstractions.NDIteration.StaticSize{(16, 16, 1)},Nothing,Nothing}},Nothing,KernelAbstractions.var"##PassType#253",Nothing,Cassette.DisableHooks}, typeof(Oceananigans.Solvers.gpu_calculate_pressure_right_hand_side!), CUDA.CuDeviceArray{Complex{Float64},3,CUDA.AS.Global}, Oceananigans.Solvers.HorizontallyPeriodic, GPU, RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}}, Int64, NamedTuple{(:u, :v, :w),Tuple{OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}},OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}},OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}}}}) resulted in invalid LLVM IR                                                                                                                                                         
Reason: unsupported call to the Julia runtime (call to jl_f_tuple)
Stacktrace:
 [1] overdub at /home/alir/.julia/packages/Cassette/158rp/src/overdub.jl:586
 [2] multiple call sites at unknown:0
Reason: unsupported call to the Julia runtime (call to jl_f_getfield)
Stacktrace:
 [1] overdub at /home/alir/.julia/packages/Cassette/158rp/src/overdub.jl:586
 [2] multiple call sites at unknown:0
Stacktrace:
 [1] check_ir(::GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget,CUDA.CUDACompilerParams}, ::LLVM.Module) at /home/alir/.julia/packages/GPUCompiler/4e9CU/src/validation.jl:123
 [2] macro expansion at /home/alir/.julia/packages/GPUCompiler/4e9CU/src/driver.jl:241 [inlined]
 [3] macro expansion at /home/alir/.julia/packages/TimerOutputs/dVnaw/src/TimerOutput.jl:206 [inlined]
 [4] codegen(::Symbol, ::GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget,CUDA.CUDACompilerParams}; libraries::Bool, deferred_codegen::Bool, optimize::Bool, strip::Bool, validate::Bool, only_entry::Bool) at /home/alir/.julia/packages/GPUCompiler/4e9CU/src/driver.jl:239
 [5] compile(::Symbol, ::GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget,CUDA.CUDACompilerParams}; libraries::Bool, deferred_codegen::Bool, optimize::Bool, strip::Bool, validate::Bool, only_entry::Bool) at /home/alir/.julia/packages/GPUCompiler/4e9CU/src/driver.jl:39
 [6] compile at /home/alir/.julia/packages/GPUCompiler/4e9CU/src/driver.jl:35 [inlined]
 [7] _cufunction(::GPUCompiler.FunctionSpec{typeof(Cassette.overdub),Tuple{Cassette.Context{nametype(CUDACtx),KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.StaticSize{(16, 16, 16)},KernelAbstractions.NDIteration.DynamicCheck,Nothing,Nothing,KernelAbstractions.NDIteration.NDRange{3,KernelAbstractions.NDIteration.StaticSize{(1, 1, 16)},KernelAbstractions.NDIteration.StaticSize{(16, 16, 1)},Nothing,Nothing}},Nothing,KernelAbstractions.var"##PassType#253",Nothing,Cassette.DisableHooks},typeof(Oceananigans.Solvers.gpu_calculate_pressure_right_hand_side!),CUDA.CuDeviceArray{Complex{Float64},3,CUDA.AS.Global},Oceananigans.Solvers.HorizontallyPeriodic,GPU,RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},Int64,NamedTuple{(:u, :v, :w),Tuple{OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}},OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}},OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}}}}}}; kwargs::Base.Iterators.Pairs{Symbol,Int64,Tuple{Symbol},NamedTuple{(:maxthreads,),Tuple{Int64}}}) at /home/alir/.julia/packages/CUDA/h38pe/src/compiler/execution.jl:308
 [8] #87 at /home/alir/.julia/packages/GPUCompiler/4e9CU/src/cache.jl:21 [inlined]
 [9] get!(::GPUCompiler.var"#87#88"{Base.Iterators.Pairs{Symbol,Int64,Tuple{Symbol},NamedTuple{(:maxthreads,),Tuple{Int64}}},typeof(CUDA._cufunction),GPUCompiler.FunctionSpec{typeof(Cassette.overdub),Tuple{Cassette.Context{nametype(CUDACtx),KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.StaticSize{(16, 16, 16)},KernelAbstractions.NDIteration.DynamicCheck,Nothing,Nothing,KernelAbstractions.NDIteration.NDRange{3,KernelAbstractions.NDIteration.StaticSize{(1, 1, 16)},KernelAbstractions.NDIteration.StaticSize{(16, 16, 1)},Nothing,Nothing}},Nothing,KernelAbstractions.var"##PassType#253",Nothing,Cassette.DisableHooks},typeof(Oceananigans.Solvers.gpu_calculate_pressure_right_hand_side!),CUDA.CuDeviceArray{Complex{Float64},3,CUDA.AS.Global},Oceananigans.Solvers.HorizontallyPeriodic,GPU,RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},Int64,NamedTuple{(:u, :v, :w),Tuple{OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}},OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}},OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}}}}}}}, ::Dict{UInt64,Any}, ::UInt64) at ./dict.jl:450
 [10] macro expansion at ./lock.jl:183 [inlined]
 [11] check_cache(::typeof(CUDA._cufunction), ::GPUCompiler.FunctionSpec{typeof(Cassette.overdub),Tuple{Cassette.Context{nametype(CUDACtx),KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.StaticSize{(16, 16, 16)},KernelAbstractions.NDIteration.DynamicCheck,Nothing,Nothing,KernelAbstractions.NDIteration.NDRange{3,KernelAbstractions.NDIteration.StaticSize{(1, 1, 16)},KernelAbstractions.NDIteration.StaticSize{(16, 16, 1)},Nothing,Nothing}},Nothing,KernelAbstractions.var"##PassType#253",Nothing,Cassette.DisableHooks},typeof(Oceananigans.Solvers.gpu_calculate_pressure_right_hand_side!),CUDA.CuDeviceArray{Complex{Float64},3,CUDA.AS.Global},Oceananigans.Solvers.HorizontallyPeriodic,GPU,RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},Int64,NamedTuple{(:u, :v, :w),Tuple{OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}},OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}},OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}}}}}}, ::UInt64; kwargs::Base.Iterators.Pairs{Symbol,Int64,Tuple{Symbol},NamedTuple{(:maxthreads,),Tuple{Int64}}}) at /home/alir/.julia/packages/GPUCompiler/4e9CU/src/cache.jl:19
 [12] + at ./int.jl:86 [inlined]
 [13] hash_64_64 at ./hashing.jl:35 [inlined]
 [14] hash_uint64 at ./hashing.jl:62 [inlined]
 [15] hx at ./float.jl:568 [inlined]
 [16] hash at ./float.jl:571 [inlined]
 [17] cached_compilation(::typeof(CUDA._cufunction), ::GPUCompiler.FunctionSpec{typeof(Cassette.overdub),Tuple{Cassette.Context{nametype(CUDACtx),KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.StaticSize{(16, 16, 16)},KernelAbstractions.NDIteration.DynamicCheck,Nothing,Nothing,KernelAbstractions.NDIteration.NDRange{3,KernelAbstractions.NDIteration.StaticSize{(1, 1, 16)},KernelAbstractions.NDIteration.StaticSize{(16, 16, 1)},Nothing,Nothing}},Nothing,KernelAbstractions.var"##PassType#253",Nothing,Cassette.DisableHooks},typeof(Oceananigans.Solvers.gpu_calculate_pressure_right_hand_side!),CUDA.CuDeviceArray{Complex{Float64},3,CUDA.AS.Global},Oceananigans.Solvers.HorizontallyPeriodic,GPU,RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},Int64,NamedTuple{(:u, :v, :w),Tuple{OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}},OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}},OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}}}}}}, ::UInt64; kwargs::Base.Iterators.Pairs{Symbol,Int64,Tuple{Symbol},NamedTuple{(:maxthreads,),Tuple{Int64}}}) at /home/alir/.julia/packages/GPUCompiler/4e9CU/src/cache.jl:0
 [18] cufunction(::Function, ::Type{T} where T; name::String, kwargs::Base.Iterators.Pairs{Symbol,Int64,Tuple{Symbol},NamedTuple{(:maxthreads,),Tuple{Int64}}}) at /home/alir/.julia/packages/CUDA/h38pe/src/compiler/execution.jl:296
 [19] macro expansion at /home/alir/.julia/packages/CUDA/h38pe/src/compiler/execution.jl:108 [inlined]
 [20] (::KernelAbstractions.Kernel{KernelAbstractions.CUDADevice,KernelAbstractions.NDIteration.StaticSize{(16, 16)},KernelAbstractions.NDIteration.StaticSize{(16, 16, 16)},typeof(Oceananigans.Solvers.gpu_calculate_pressure_right_hand_side!)})(::CUDA.CuArray{Complex{Float64},3,Nothing}, ::Vararg{Any,N} where N; ndrange::Nothing, dependencies::KernelAbstractions.CudaEvent, workgroupsize::Nothing, progress::Function) at /home/alir/.julia/packages/KernelAbstractions/yw9SF/src/backends/cuda.jl:211
 [21] launch!(::GPU, ::RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}}, ::Symbol, ::typeof(Oceananigans.Solvers.calculate_pressure_right_hand_side!), ::CUDA.CuArray{Complex{Float64},3,Nothing}, ::Vararg{Any,N} where N; dependencies::KernelAbstractions.CudaEvent, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/alir/Oceananigans.jl/src/Utils/kernel_launching.jl:67
 [22] solve_for_pressure!(::OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}}, ::Oceananigans.Solvers.PressureSolver{Oceananigans.Solvers.HorizontallyPeriodic,GPU,NamedTuple{(:kx², :ky², :kz²),Tuple{CUDA.CuArray{Float64,3,Nothing},CUDA.CuArray{Float64,3,Nothing},CUDA.CuArray{Float64,3,Nothing}}},CUDA.CuArray{Complex{Float64},3,Nothing},NamedTuple{(:FFTxy!, :FFTz!, :IFFTxy!, :IFFTz!),Tuple{CUDA.CUFFT.cCuFFTPlan{Complex{Float64},-1,true,3},CUDA.CUFFT.cCuFFTPlan{Complex{Float64},-1,true,3},AbstractFFTs.ScaledPlan{Complex{Float64},CUDA.CUFFT.cCuFFTPlan{Complex{Float64},1,true,3},Float64},AbstractFFTs.ScaledPlan{Complex{Float64},CUDA.CUFFT.cCuFFTPlan{Complex{Float64},1,true,3},Float64}}},NamedTuple{(:ω_4Nz⁺, :ω_4Nz⁻),Tuple{CUDA.CuArray{Complex{Float64},3,Nothing},CUDA.CuArray{Complex{Float64},3,Nothing}}}}, ::GPU, ::RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}}, ::Int64, ::NamedTuple{(:u, :v, :w),Tuple{OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}}}}) at /home/alir/Oceananigans.jl/src/Solvers/solve_for_pressure.jl:11
 [23] calculate_pressure_correction!(::OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}}, ::Int64, ::NamedTuple{(:u, :v, :w),Tuple{OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}}}}, ::IncompressibleModel{Oceananigans.TimeSteppers.AdamsBashforthTimeStepper{Float64,NamedTuple{(:u, :v, :w, :T, :S),Tuple{Field{Oceananigans.Grids.Face,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Face,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Face,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{NormalFlow,Nothing},BoundaryCondition{NormalFlow,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}}}},NamedTuple{(:u, :v, :w),Tuple{Field{Oceananigans.Grids.Face,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Face,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Face,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{NormalFlow,Nothing},BoundaryCondition{NormalFlow,Nothing}}}}}}}},Oceananigans.TurbulenceClosures.IsotropicDiffusivity{Float64,NamedTuple{(:T, :S),Tuple{Float64,Float64}}},GPU,RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},Float64,SeawaterBuoyancy{Float64,LinearEquationOfState{Float64},Nothing,Nothing},Nothing,Nothing,NamedTuple{(:u, :v, :w),Tuple{Field{Oceananigans.Grids.Face,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Face,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Face,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{NormalFlow,Nothing},BoundaryCondition{NormalFlow,Nothing}}}}}}},NamedTuple{(:T, :S),Tuple{Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}}}},NamedTuple{(:pHY′, :pNHS),Tuple{Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}}}},NamedTuple{(:u, :v, :w, :T, :S),NTuple{5,typeof(Oceananigans.Forcing.zeroforcing)}},Oceananigans.Advection.CenteredSecondOrder,Oceananigans.Solvers.PressureSolver{Oceananigans.Solvers.HorizontallyPeriodic,GPU,NamedTuple{(:kx², :ky², :kz²),Tuple{CUDA.CuArray{Float64,3,Nothing},CUDA.CuArray{Float64,3,Nothing},CUDA.CuArray{Float64,3,Nothing}}},CUDA.CuArray{Complex{Float64},3,Nothing},NamedTuple{(:FFTxy!, :FFTz!, :IFFTxy!, :IFFTz!),Tuple{CUDA.CUFFT.cCuFFTPlan{Complex{Float64},-1,true,3},CUDA.CUFFT.cCuFFTPlan{Complex{Float64},-1,true,3},AbstractFFTs.ScaledPlan{Complex{Float64},CUDA.CUFFT.cCuFFTPlan{Complex{Float64},1,true,3},Float64},AbstractFFTs.ScaledPlan{Complex{Float64},CUDA.CUFFT.cCuFFTPlan{Complex{Float64},1,true,3},Float64}}},NamedTuple{(:ω_4Nz⁺, :ω_4Nz⁻),Tuple{CUDA.CuArray{Complex{Float64},3,Nothing},CUDA.CuArray{Complex{Float64},3,Nothing}}}},Nothing}) at /home/alir/Oceananigans.jl/src/TimeSteppers/generic_time_stepping.jl:75
 [24] time_step!(::IncompressibleModel{Oceananigans.TimeSteppers.AdamsBashforthTimeStepper{Float64,NamedTuple{(:u, :v, :w, :T, :S),Tuple{Field{Oceananigans.Grids.Face,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Face,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Face,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{NormalFlow,Nothing},BoundaryCondition{NormalFlow,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}}}},NamedTuple{(:u, :v, :w),Tuple{Field{Oceananigans.Grids.Face,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Face,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Face,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{NormalFlow,Nothing},BoundaryCondition{NormalFlow,Nothing}}}}}}}},Oceananigans.TurbulenceClosures.IsotropicDiffusivity{Float64,NamedTuple{(:T, :S),Tuple{Float64,Float64}}},GPU,RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},Float64,SeawaterBuoyancy{Float64,LinearEquationOfState{Float64},Nothing,Nothing},Nothing,Nothing,NamedTuple{(:u, :v, :w),Tuple{Field{Oceananigans.Grids.Face,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Face,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Face,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{NormalFlow,Nothing},BoundaryCondition{NormalFlow,Nothing}}}}}}},NamedTuple{(:T, :S),Tuple{Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}}}},NamedTuple{(:pHY′, :pNHS),Tuple{Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}}}},NamedTuple{(:u, :v, :w, :T, :S),NTuple{5,typeof(Oceananigans.Forcing.zeroforcing)}},Oceananigans.Advection.CenteredSecondOrder,Oceananigans.Solvers.PressureSolver{Oceananigans.Solvers.HorizontallyPeriodic,GPU,NamedTuple{(:kx², :ky², :kz²),Tuple{CUDA.CuArray{Float64,3,Nothing},CUDA.CuArray{Float64,3,Nothing},CUDA.CuArray{Float64,3,Nothing}}},CUDA.CuArray{Complex{Float64},3,Nothing},NamedTuple{(:FFTxy!, :FFTz!, :IFFTxy!, :IFFTz!),Tuple{CUDA.CUFFT.cCuFFTPlan{Complex{Float64},-1,true,3},CUDA.CUFFT.cCuFFTPlan{Complex{Float64},-1,true,3},AbstractFFTs.ScaledPlan{Complex{Float64},CUDA.CUFFT.cCuFFTPlan{Complex{Float64},1,true,3},Float64},AbstractFFTs.ScaledPlan{Complex{Float64},CUDA.CUFFT.cCuFFTPlan{Complex{Float64},1,true,3},Float64}}},NamedTuple{(:ω_4Nz⁺, :ω_4Nz⁻),Tuple{CUDA.CuArray{Complex{Float64},3,Nothing},CUDA.CuArray{Complex{Float64},3,Nothing}}}},Nothing}, ::Int64; euler::Bool) at /home/alir/Oceananigans.jl/src/TimeSteppers/adams_bashforth.jl:57
 [25] time_step!(::IncompressibleModel{Oceananigans.TimeSteppers.AdamsBashforthTimeStepper{Float64,NamedTuple{(:u, :v, :w, :T, :S),Tuple{Field{Oceananigans.Grids.Face,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Face,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Face,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{NormalFlow,Nothing},BoundaryCondition{NormalFlow,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}}}},NamedTuple{(:u, :v, :w),Tuple{Field{Oceananigans.Grids.Face,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Face,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Face,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{NormalFlow,Nothing},BoundaryCondition{NormalFlow,Nothing}}}}}}}},Oceananigans.TurbulenceClosures.IsotropicDiffusivity{Float64,NamedTuple{(:T, :S),Tuple{Float64,Float64}}},GPU,RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},Float64,SeawaterBuoyancy{Float64,LinearEquationOfState{Float64},Nothing,Nothing},Nothing,Nothing,NamedTuple{(:u, :v, :w),Tuple{Field{Oceananigans.Grids.Face,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Face,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Face,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{NormalFlow,Nothing},BoundaryCondition{NormalFlow,Nothing}}}}}}},NamedTuple{(:T, :S),Tuple{Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}}}},NamedTuple{(:pHY′, :pNHS),Tuple{Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3,Nothing}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}}}},NamedTuple{(:u, :v, :w, :T, :S),NTuple{5,typeof(Oceananigans.Forcing.zeroforcing)}},Oceananigans.Advection.CenteredSecondOrder,Oceananigans.Solvers.PressureSolver{Oceananigans.Solvers.HorizontallyPeriodic,GPU,NamedTuple{(:kx², :ky², :kz²),Tuple{CUDA.CuArray{Float64,3,Nothing},CUDA.CuArray{Float64,3,Nothing},CUDA.CuArray{Float64,3,Nothing}}},CUDA.CuArray{Complex{Float64},3,Nothing},NamedTuple{(:FFTxy!, :FFTz!, :IFFTxy!, :IFFTz!),Tuple{CUDA.CUFFT.cCuFFTPlan{Complex{Float64},-1,true,3},CUDA.CUFFT.cCuFFTPlan{Complex{Float64},-1,true,3},AbstractFFTs.ScaledPlan{Complex{Float64},CUDA.CUFFT.cCuFFTPlan{Complex{Float64},1,true,3},Float64},AbstractFFTs.ScaledPlan{Complex{Float64},CUDA.CUFFT.cCuFFTPlan{Complex{Float64},1,true,3},Float64}}},NamedTuple{(:ω_4Nz⁺, :ω_4Nz⁻),Tuple{CUDA.CuArray{Complex{Float64},3,Nothing},CUDA.CuArray{Complex{Float64},3,Nothing}}}},Nothing}, ::Int64) at /home/alir/Oceananigans.jl/src/TimeSteppers/adams_bashforth.jl:41
 [26] top-level scope at REPL[1]:1

The text was updated successfully, but these errors were encountered:

glwagner · 2020-08-03T14:50:12Z

Could be a failure of Adapt.jl's method for tuples and named tuples:

https://github.com/JuliaGPU/Adapt.jl/blob/a62a2568f1199d0e7b154eb6001afa629d31038e/src/base.jl#L3

ali-ramadhan · 2020-08-13T13:20:59Z

@maleadt Was wondering if you've seen this issue before or if you have any ideas on how to fix it?

maleadt · 2020-08-13T13:30:56Z

Adapt.jl isn't relevant here, that's for CPU->GPU conversions. This is a tuple operation not getting inferred properly, leading to a dynamic call to a runtime function. Those aren't supported. With 1.5, there's been a bunch of latency optimizations that have affected inference quality, e.g., when not forcing specialization on functions with ::F where F.

Your best bet here would be to step through using Cthulhu and figure out which code isn't inferring properly. If it's tough code to get though, you could always add the call to Cthulhu.descend_code_warntype at the place where the InvalidIRError is thrown.

maleadt · 2020-08-13T13:31:43Z

Also note the scalar iteration; you might want to look into that too.

ali-ramadhan · 2020-08-13T14:30:03Z

Thanks @maleadt! If it's actually in the calculate_pressure_right_hand_side! kernel then might not be hard to find as it's pretty small...

Mostly just documenting what I did in this comment so I can pick up later on:

Tried playing around with Cthulhu.jl a little bit, didn't understand everything it was telling me but ended up putting Cthulhu.@descend_code_warntype in front of the call to solve_for_pressure! in generic_time_stepping.jl (other places seemed "too complex" for Cthulhu.jl and it errored).

Not sure if this is useful but it seems like the worksize tuple and some args tuple aren't being inferred correctly but worksize is just used to launch the kernel so maybe args is the problem. I'll have a deeper look when I have more time.

Also note the scalar iteration; you might want to look into that too.

True haha, we only finally started dynamically disabling scalar ops yesterday in PR #851 after being burnt by scalar ops one too many times!

maleadt · 2020-08-13T14:33:18Z

Not sure if this is useful but it seems like the worksize tuple and some args tuple aren't being inferred correctly but worksize is just used to launch the kernel so maybe args is the problem. I'll have a deeper look when I have more time.

You're looking at CPU code here; you rather want to prefix your application with @device_code_warntype interactive=true (which will spawn a Cthulhu UI for every kernel), or use descend_code_warntype directly passing the kernel's function and argument typle type. It should be possible to reconstruct the tuple type from the error message:

ERROR: InvalidIRError: compiling kernel gpu_calculate_pressure_right_hand_side!(Cassette.Context{nametype(CUDACtx),KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.StaticSize{(16, 16, 16)},KernelAbstractions.NDIteration.DynamicCheck,Nothing,Nothing,KernelAbstractions.NDIteration.NDRange{3,KernelAbstractions.NDIteration.StaticSize{(1, 1, 16)},KernelAbstractions.NDIteration.StaticSize{(16, 16, 1)},Nothing,Nothing}},Nothing,KernelAbstractions.var"##PassType#253",Nothing,Cassette.DisableHooks}, typeof(Oceananigans.Solvers.gpu_calculate_pressure_right_hand_side!), CUDA.CuDeviceArray{Complex{Float64},3,CUDA.AS.Global}, Oceananigans.Solvers.HorizontallyPeriodic, GPU, RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}}, Int64, NamedTuple{(:u, :v, :w),Tuple{OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}},OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}},OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}}}}) resulted in invalid LLVM IR

But seeing how it's fairly complex you might be better off using @device_code_warntype interactive=true (after having imported Cthulhu).

glwagner · 2020-09-24T10:58:24Z

Thank you @maleadt for the very helpful comments. We toyed around a bit with forced specialization on functions in the past and in other contexts when we had difficulties with inlining. We should look into that. I'm not sure we are forcing specialization everywhere and it sounds like we may want to, to be conservative regarding GPU compilation.

Looking at the error, it seems like the problem function is calculate_pressure_right_hand_side:

Oceananigans.jl/src/Solvers/solve_for_pressure.jl

Lines 36 to 42 in ea33ab0

    
           @kernel function calculate_pressure_right_hand_side!(RHS, solver_type, arch, grid, Δt, U★) 
        
               i, j, k = @index(Global, NTuple) 
        
               i′, j′, k′ = permute_index(solver_type, arch, i, j, k, grid.Nx, grid.Ny, grid.Nz) 
        
               @inbounds RHS[i′, j′, k′] = divᶜᶜᶜ(i, j, k, grid, U★.u, U★.v, U★.w) / Δt 
        
           end

right? Since the default model is "horizontally periodic", permute_index ends up calling:

Oceananigans.jl/src/Solvers/index_permutations.jl

Line 36 in ea33ab0

    
           @inline   permute_index(::HorizontallyPeriodic, ::GPU, i, j, k, Nx, Ny, Nz) = i, j,   _permute_index(k, Nz)

The function _permute_index is

Oceananigans.jl/src/Solvers/index_permutations.jl

Lines 11 to 17 in ea33ab0

    
           @inline function _permute_index(i, N) 
        
               if (i & 1) == 1  # Same as isodd(i) 
        
                   return floor(Int, i/2) + 1 
        
               else 
        
                   return N - floor(Int, (i-1)/2) 
        
               end 
        
           end

Is it worth toying around with different ways to write permute_index? Or maybe it would help to give the compiler a hint as the types of the arguments of _permute_index...

maleadt · 2020-09-24T12:01:36Z

I think the easiest is to check with Cthulhu which of these expressions are badly inferred and result in a call to jl_f_tuple. It's not necessarily any of the definitions shown here, it could as well be, e.g., a getindex or conversion method implicitly being called on one of your kernel arguments.

ali-ramadhan · 2020-09-24T15:11:06Z

If we're not sure what the cause of the problem is and Cthulu.jl doesn't help (or in my case too much of a noob) we could try commenting out the pressure solve (or parts of it) to try and isolate the problematic line(s) or kernel(s).

I did try playing around with refactoring permute_index to resolve this issue but didn't really get anywhere.

ali-ramadhan · 2020-09-28T22:38:50Z

So I was able to isolate the issue to the _permute_index(i, N) function, and more specifically to the float to int conversion that it does. Something about Casette not being able to overdub a call to Int64?

@maleadt @vchuravy Does this error message say anything useful?

Was not able to reproduce in a minimal working example for CUDA.jl or KernelAbstractions.jl unfortunately...

I'll open a PR to make this issue a priority.

              _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.5.0 (2020-08-01)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using Oceananigans; @info 1; grid = RegularCartesianGrid(topology=(Periodic, Periodic, Bounded), size=(16, 16, 16), extent=(1, 1, 1)); @info 2; model = IncompressibleModel(architecture=GPU(), grid=grid); @info 3; time_step!(model, 1)
[ Info: Precompiling Oceananigans [9e8cae18-63c1-5223-a75c-80ca9d6e9a09]
[ Info: 1
[ Info: 2
[ Info: 3
ERROR: InvalidIRError: compiling kernel gpu_calculate_pressure_right_hand_side!(Cassette.Context{nametype(CUDACtx),KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.StaticSize{(16, 16, 16)},KernelAbstractions.NDIteration.DynamicCheck,Nothing,Nothing,KernelAbstractions.NDIteration.NDRange{3,KernelAbstractions.NDIteration.StaticSize{(1, 1, 16)},KernelAbstractions.NDIteration.StaticSize{(16, 16, 1)},Nothing,Nothing}},Nothing,KernelAbstractions.var"##PassType#253",Nothing,Cassette.DisableHooks}, typeof(Oceananigans.Solvers.gpu_calculate_pressure_right_hand_side!), CUDA.CuDeviceArray{Complex{Float64},3,CUDA.AS.Global}, Oceananigans.Solvers.HorizontallyPeriodic, GPU, RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}}, Int64, NamedTuple{(:u, :v, :w),Tuple{OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}},OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}},OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}}}}) resulted in invalid LLVM IR                                                                                          
Reason: unsupported call to the Julia runtime (call to jl_f_tuple)
Stacktrace:
 [1] overdub at /home/alir/.julia/packages/Cassette/158rp/src/overdub.jl:586
 [2] Int64 at float.jl:710
 [3] _permute_index at /home/alir/Oceananigans.jl/src/Solvers/index_permutations.jl:11
 [4] permute_index at /home/alir/Oceananigans.jl/src/Solvers/index_permutations.jl:24
 [5] macro expansion at /home/alir/Oceananigans.jl/src/Solvers/solve_for_pressure.jl:39
 [6] gpu_calculate_pressure_right_hand_side! at /home/alir/.julia/packages/KernelAbstractions/xslEz/src/macros.jl:80
 [7] overdub at /home/alir/.julia/packages/Cassette/158rp/src/overdub.jl:0
Reason: unsupported call to the Julia runtime (call to jl_f_getfield)
Stacktrace:
 [1] overdub at /home/alir/.julia/packages/Cassette/158rp/src/overdub.jl:586
 [2] Int64 at float.jl:710
 [3] _permute_index at /home/alir/Oceananigans.jl/src/Solvers/index_permutations.jl:11
 [4] permute_index at /home/alir/Oceananigans.jl/src/Solvers/index_permutations.jl:24
 [5] macro expansion at /home/alir/Oceananigans.jl/src/Solvers/solve_for_pressure.jl:39
 [6] gpu_calculate_pressure_right_hand_side! at /home/alir/.julia/packages/KernelAbstractions/xslEz/src/macros.jl:80
 [7] overdub at /home/alir/.julia/packages/Cassette/158rp/src/overdub.jl:0
Stacktrace:
 [1] check_ir(::GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget,CUDA.CUDACompilerParams}, ::LLVM.Module) at /home/alir/.julia/packages/GPUCompiler/GKp4B/src/validation.jl:123
 [2] macro expansion at /home/alir/.julia/packages/GPUCompiler/GKp4B/src/driver.jl:241 [inlined]
 [3] macro expansion at /home/alir/.julia/packages/TimerOutputs/dVnaw/src/TimerOutput.jl:206 [inlined]
 [4] codegen(::Symbol, ::GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget,CUDA.CUDACompilerParams}; libraries::Bool, deferred_codegen::Bool, optimize::Bool, strip::Bool, validate::Bool, only_entry::Bool) at /home/alir/.julia/packages/GPUCompiler/GKp4B/src/driver.jl:239
 [5] compile(::Symbol, ::GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget,CUDA.CUDACompilerParams}; libraries::Bool, deferred_codegen::Bool, optimize::Bool, strip::Bool, validate::Bool, only_entry::Bool) at /home/alir/.julia/packages/GPUCompiler/GKp4B/src/driver.jl:39
 [6] compile at /home/alir/.julia/packages/GPUCompiler/GKp4B/src/driver.jl:35 [inlined]
 [7] _cufunction(::GPUCompiler.FunctionSpec{typeof(Cassette.overdub),Tuple{Cassette.Context{nametype(CUDACtx),KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.StaticSize{(16, 16, 16)},KernelAbstractions.NDIteration.DynamicCheck,Nothing,Nothing,KernelAbstractions.NDIteration.NDRange{3,KernelAbstractions.NDIteration.StaticSize{(1, 1, 16)},KernelAbstractions.NDIteration.StaticSize{(16, 16, 1)},Nothing,Nothing}},Nothing,KernelAbstractions.var"##PassType#253",Nothing,Cassette.DisableHooks},typeof(Oceananigans.Solvers.gpu_calculate_pressure_right_hand_side!),CUDA.CuDeviceArray{Complex{Float64},3,CUDA.AS.Global},Oceananigans.Solvers.HorizontallyPeriodic,GPU,RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},Int64,NamedTuple{(:u, :v, :w),Tuple{OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}},OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}},OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}}}}}}; kwargs::Base.Iterators.Pairs{Symbol,Int64,Tuple{Symbol},NamedTuple{(:maxthreads,),Tuple{Int64}}}) at /home/alir/.julia/packages/CUDA/dZvbp/src/compiler/execution.jl:310
 [8] check_cache(::typeof(CUDA._cufunction), ::GPUCompiler.FunctionSpec{typeof(Cassette.overdub),Tuple{Cassette.Context{nametype(CUDACtx),KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.StaticSize{(16, 16, 16)},KernelAbstractions.NDIteration.DynamicCheck,Nothing,Nothing,KernelAbstractions.NDIteration.NDRange{3,KernelAbstractions.NDIteration.StaticSize{(1, 1, 16)},KernelAbstractions.NDIteration.StaticSize{(16, 16, 1)},Nothing,Nothing}},Nothing,KernelAbstractions.var"##PassType#253",Nothing,Cassette.DisableHooks},typeof(Oceananigans.Solvers.gpu_calculate_pressure_right_hand_side!),CUDA.CuDeviceArray{Complex{Float64},3,CUDA.AS.Global},Oceananigans.Solvers.HorizontallyPeriodic,GPU,RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},Int64,NamedTuple{(:u, :v, :w),Tuple{OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}},OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}},OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}}}}}}, ::UInt64; kwargs::Base.Iterators.Pairs{Symbol,Int64,Tuple{Symbol},NamedTuple{(:maxthreads,),Tuple{Int64}}}) at /home/alir/.julia/packages/GPUCompiler/GKp4B/src/cache.jl:24
 [9] gpu_calculate_pressure_right_hand_side! at ./none:0 [inlined]
 [10] cached_compilation(::typeof(CUDA._cufunction), ::GPUCompiler.FunctionSpec{typeof(Cassette.overdub),Tuple{Cassette.Context{nametype(CUDACtx),KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.StaticSize{(16, 16, 16)},KernelAbstractions.NDIteration.DynamicCheck,Nothing,Nothing,KernelAbstractions.NDIteration.NDRange{3,KernelAbstractions.NDIteration.StaticSize{(1, 1, 16)},KernelAbstractions.NDIteration.StaticSize{(16, 16, 1)},Nothing,Nothing}},Nothing,KernelAbstractions.var"##PassType#253",Nothing,Cassette.DisableHooks},typeof(Oceananigans.Solvers.gpu_calculate_pressure_right_hand_side!),CUDA.CuDeviceArray{Complex{Float64},3,CUDA.AS.Global},Oceananigans.Solvers.HorizontallyPeriodic,GPU,RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},Int64,NamedTuple{(:u, :v, :w),Tuple{OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}},OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}},OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}}}}}}, ::UInt64; kwargs::Base.Iterators.Pairs{Symbol,Int64,Tuple{Symbol},NamedTuple{(:maxthreads,),Tuple{Int64}}}) at /home/alir/.julia/packages/GPUCompiler/GKp4B/src/cache.jl:0
 [11] cufunction(::typeof(Cassette.overdub), ::Type{Tuple{Cassette.Context{nametype(CUDACtx),KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.StaticSize{(16, 16, 16)},KernelAbstractions.NDIteration.DynamicCheck,Nothing,Nothing,KernelAbstractions.NDIteration.NDRange{3,KernelAbstractions.NDIteration.StaticSize{(1, 1, 16)},KernelAbstractions.NDIteration.StaticSize{(16, 16, 1)},Nothing,Nothing}},Nothing,KernelAbstractions.var"##PassType#253",Nothing,Cassette.DisableHooks},typeof(Oceananigans.Solvers.gpu_calculate_pressure_right_hand_side!),CUDA.CuDeviceArray{Complex{Float64},3,CUDA.AS.Global},Oceananigans.Solvers.HorizontallyPeriodic,GPU,RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},Int64,NamedTuple{(:u, :v, :w),Tuple{OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}},OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}},OffsetArrays.OffsetArray{Float64,3,CUDA.CuDeviceArray{Float64,3,CUDA.AS.Global}}}}}}; name::String, kwargs::Base.Iterators.Pairs{Symbol,Int64,Tuple{Symbol},NamedTuple{(:maxthreads,),Tuple{Int64}}}) at /home/alir/.julia/packages/CUDA/dZvbp/src/compiler/execution.jl:298
 [12] macro expansion at /home/alir/.julia/packages/CUDA/dZvbp/src/compiler/execution.jl:109 [inlined]
 [13] (::KernelAbstractions.Kernel{KernelAbstractions.CUDADevice,KernelAbstractions.NDIteration.StaticSize{(16, 16)},KernelAbstractions.NDIteration.StaticSize{(16, 16, 16)},typeof(Oceananigans.Solvers.gpu_calculate_pressure_right_hand_side!)})(::CUDA.CuArray{Complex{Float64},3}, ::Vararg{Any,N} where N; ndrange::Nothing, dependencies::KernelAbstractions.CudaEvent, workgroupsize::Nothing, progress::Function) at /home/alir/.julia/packages/KernelAbstractions/xslEz/src/backends/cuda.jl:172
 [14] launch!(::GPU, ::RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}}, ::Symbol, ::typeof(Oceananigans.Solvers.calculate_pressure_right_hand_side!), ::CUDA.CuArray{Complex{Float64},3}, ::Vararg{Any,N} where N; dependencies::KernelAbstractions.CudaEvent, include_right_boundaries::Bool, location::Nothing, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/alir/Oceananigans.jl/src/Utils/kernel_launching.jl:80
 [15] solve_for_pressure!(::OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}}, ::Oceananigans.Solvers.PressureSolver{Oceananigans.Solvers.HorizontallyPeriodic,GPU,NamedTuple{(:kx², :ky², :kz²),Tuple{CUDA.CuArray{Float64,3},CUDA.CuArray{Float64,3},CUDA.CuArray{Float64,3}}},CUDA.CuArray{Complex{Float64},3},NamedTuple{(:FFTxy!, :FFTz!, :IFFTxy!, :IFFTz!),Tuple{CUDA.CUFFT.cCuFFTPlan{Complex{Float64},-1,true,3},CUDA.CUFFT.cCuFFTPlan{Complex{Float64},-1,true,3},AbstractFFTs.ScaledPlan{Complex{Float64},CUDA.CUFFT.cCuFFTPlan{Complex{Float64},1,true,3},Float64},AbstractFFTs.ScaledPlan{Complex{Float64},CUDA.CUFFT.cCuFFTPlan{Complex{Float64},1,true,3},Float64}}},NamedTuple{(:ω_4Nz⁺, :ω_4Nz⁻),Tuple{CUDA.CuArray{Complex{Float64},3},CUDA.CuArray{Complex{Float64},3}}}}, ::GPU, ::RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}}, ::Int64, ::NamedTuple{(:u, :v, :w),Tuple{OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}}}}) at /home/alir/Oceananigans.jl/src/Solvers/solve_for_pressure.jl:11
 [16] calculate_pressure_correction!(::OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}}, ::Int64, ::NamedTuple{(:u, :v, :w),Tuple{OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}}}}, ::IncompressibleModel{Oceananigans.TimeSteppers.QuasiAdamsBashforth2TimeStepper{Float64,NamedTuple{(:u, :v, :w, :T, :S),Tuple{Field{Oceananigans.Grids.Face,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Face,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Face,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{NormalFlow,Nothing},BoundaryCondition{NormalFlow,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}}}}},IsotropicDiffusivity{Float64,NamedTuple{(:T, :S),Tuple{Float64,Float64}}},GPU,RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},Float64,SeawaterBuoyancy{Float64,LinearEquationOfState{Float64},Nothing,Nothing},Nothing,Nothing,NamedTuple{(:u, :v, :w),Tuple{Field{Oceananigans.Grids.Face,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Face,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Face,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{NormalFlow,Nothing},BoundaryCondition{NormalFlow,Nothing}}}}}}},NamedTuple{(:T, :S),Tuple{Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}}}},NamedTuple{(:pHY′, :pNHS),Tuple{Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}}}},NamedTuple{(:u, :v, :w, :T, :S),NTuple{5,typeof(Oceananigans.Forcing.zeroforcing)}},Oceananigans.Advection.CenteredSecondOrder,Oceananigans.Solvers.PressureSolver{Oceananigans.Solvers.HorizontallyPeriodic,GPU,NamedTuple{(:kx², :ky², :kz²),Tuple{CUDA.CuArray{Float64,3},CUDA.CuArray{Float64,3},CUDA.CuArray{Float64,3}}},CUDA.CuArray{Complex{Float64},3},NamedTuple{(:FFTxy!, :FFTz!, :IFFTxy!, :IFFTz!),Tuple{CUDA.CUFFT.cCuFFTPlan{Complex{Float64},-1,true,3},CUDA.CUFFT.cCuFFTPlan{Complex{Float64},-1,true,3},AbstractFFTs.ScaledPlan{Complex{Float64},CUDA.CUFFT.cCuFFTPlan{Complex{Float64},1,true,3},Float64},AbstractFFTs.ScaledPlan{Complex{Float64},CUDA.CUFFT.cCuFFTPlan{Complex{Float64},1,true,3},Float64}}},NamedTuple{(:ω_4Nz⁺, :ω_4Nz⁻),Tuple{CUDA.CuArray{Complex{Float64},3},CUDA.CuArray{Complex{Float64},3}}}},Nothing}) at /home/alir/Oceananigans.jl/src/TimeSteppers/generic_time_stepping.jl:75
 [17] time_step!(::IncompressibleModel{Oceananigans.TimeSteppers.QuasiAdamsBashforth2TimeStepper{Float64,NamedTuple{(:u, :v, :w, :T, :S),Tuple{Field{Oceananigans.Grids.Face,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Face,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Face,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{NormalFlow,Nothing},BoundaryCondition{NormalFlow,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}}}}},IsotropicDiffusivity{Float64,NamedTuple{(:T, :S),Tuple{Float64,Float64}}},GPU,RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},Float64,SeawaterBuoyancy{Float64,LinearEquationOfState{Float64},Nothing,Nothing},Nothing,Nothing,NamedTuple{(:u, :v, :w),Tuple{Field{Oceananigans.Grids.Face,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Face,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Face,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{NormalFlow,Nothing},BoundaryCondition{NormalFlow,Nothing}}}}}}},NamedTuple{(:T, :S),Tuple{Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}}}},NamedTuple{(:pHY′, :pNHS),Tuple{Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}}}},NamedTuple{(:u, :v, :w, :T, :S),NTuple{5,typeof(Oceananigans.Forcing.zeroforcing)}},Oceananigans.Advection.CenteredSecondOrder,Oceananigans.Solvers.PressureSolver{Oceananigans.Solvers.HorizontallyPeriodic,GPU,NamedTuple{(:kx², :ky², :kz²),Tuple{CUDA.CuArray{Float64,3},CUDA.CuArray{Float64,3},CUDA.CuArray{Float64,3}}},CUDA.CuArray{Complex{Float64},3},NamedTuple{(:FFTxy!, :FFTz!, :IFFTxy!, :IFFTz!),Tuple{CUDA.CUFFT.cCuFFTPlan{Complex{Float64},-1,true,3},CUDA.CUFFT.cCuFFTPlan{Complex{Float64},-1,true,3},AbstractFFTs.ScaledPlan{Complex{Float64},CUDA.CUFFT.cCuFFTPlan{Complex{Float64},1,true,3},Float64},AbstractFFTs.ScaledPlan{Complex{Float64},CUDA.CUFFT.cCuFFTPlan{Complex{Float64},1,true,3},Float64}}},NamedTuple{(:ω_4Nz⁺, :ω_4Nz⁻),Tuple{CUDA.CuArray{Complex{Float64},3},CUDA.CuArray{Complex{Float64},3}}}},Nothing}, ::Int64; euler::Bool) at /home/alir/Oceananigans.jl/src/TimeSteppers/quasi_adams_bashforth_2.jl:54
 [18] time_step!(::IncompressibleModel{Oceananigans.TimeSteppers.QuasiAdamsBashforth2TimeStepper{Float64,NamedTuple{(:u, :v, :w, :T, :S),Tuple{Field{Oceananigans.Grids.Face,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Face,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Face,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{NormalFlow,Nothing},BoundaryCondition{NormalFlow,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}}}}},IsotropicDiffusivity{Float64,NamedTuple{(:T, :S),Tuple{Float64,Float64}}},GPU,RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},Float64,SeawaterBuoyancy{Float64,LinearEquationOfState{Float64},Nothing,Nothing},Nothing,Nothing,NamedTuple{(:u, :v, :w),Tuple{Field{Oceananigans.Grids.Face,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Face,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Face,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{NormalFlow,Nothing},BoundaryCondition{NormalFlow,Nothing}}}}}}},NamedTuple{(:T, :S),Tuple{Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}}}},NamedTuple{(:pHY′, :pNHS),Tuple{Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}},Field{Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,Oceananigans.Grids.Cell,OffsetArrays.OffsetArray{Float64,3,CUDA.CuArray{Float64,3}},RegularCartesianGrid{Float64,Periodic,Periodic,Bounded,OffsetArrays.OffsetArray{Float64,1,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}},NamedTuple{(:x, :y, :z),Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing},BoundaryCondition{Oceananigans.BoundaryConditions.Periodic,Nothing}},CoordinateBoundaryConditions{BoundaryCondition{Flux,Nothing},BoundaryCondition{Flux,Nothing}}}}}}},NamedTuple{(:u, :v, :w, :T, :S),NTuple{5,typeof(Oceananigans.Forcing.zeroforcing)}},Oceananigans.Advection.CenteredSecondOrder,Oceananigans.Solvers.PressureSolver{Oceananigans.Solvers.HorizontallyPeriodic,GPU,NamedTuple{(:kx², :ky², :kz²),Tuple{CUDA.CuArray{Float64,3},CUDA.CuArray{Float64,3},CUDA.CuArray{Float64,3}}},CUDA.CuArray{Complex{Float64},3},NamedTuple{(:FFTxy!, :FFTz!, :IFFTxy!, :IFFTz!),Tuple{CUDA.CUFFT.cCuFFTPlan{Complex{Float64},-1,true,3},CUDA.CUFFT.cCuFFTPlan{Complex{Float64},-1,true,3},AbstractFFTs.ScaledPlan{Complex{Float64},CUDA.CUFFT.cCuFFTPlan{Complex{Float64},1,true,3},Float64},AbstractFFTs.ScaledPlan{Complex{Float64},CUDA.CUFFT.cCuFFTPlan{Complex{Float64},1,true,3},Float64}}},NamedTuple{(:ω_4Nz⁺, :ω_4Nz⁻),Tuple{CUDA.CuArray{Complex{Float64},3},CUDA.CuArray{Complex{Float64},3}}}},Nothing}, ::Int64) at /home/alir/Oceananigans.jl/src/TimeSteppers/quasi_adams_bashforth_2.jl:40
 [19] top-level scope at REPL[1]:1

ali-ramadhan added bug 🐞 Even a perfect program still has bugs GPU 👾 Where Oceananigans gets its powers from labels Aug 3, 2020

ali-ramadhan mentioned this issue Aug 13, 2020

Failed to precompile GPUArrays, CUDA, Oceananigans on macOS #854

Closed

ali-ramadhan mentioned this issue Aug 19, 2020

Error with NetCDF output - Julia v1.5, Oceananigans v0.34.1 #862

Closed

ali-ramadhan mentioned this issue Sep 28, 2020

Attempting to get Oceananigans to work on GPUs with Julia 1.5 #997

Merged

ali-ramadhan added the 🚨 high priority 🚨 label Oct 1, 2020

ali-ramadhan mentioned this issue Oct 13, 2020

Run on GPU without network access #1035

Closed

ali-ramadhan closed this as completed in #997 Oct 14, 2020

ali-ramadhan mentioned this issue Dec 4, 2020

Complex AbstractOperations cannot be computed on GPU #1241

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Obscure GPU error when time stepping a model with Julia 1.5 and Oceananigans v0.33.0 #828

Obscure GPU error when time stepping a model with Julia 1.5 and Oceananigans v0.33.0 #828

ali-ramadhan commented Aug 3, 2020 •

edited

Loading

glwagner commented Aug 3, 2020

ali-ramadhan commented Aug 13, 2020

maleadt commented Aug 13, 2020 •

edited

Loading

maleadt commented Aug 13, 2020

ali-ramadhan commented Aug 13, 2020

maleadt commented Aug 13, 2020

glwagner commented Sep 24, 2020 •

edited

Loading

maleadt commented Sep 24, 2020

ali-ramadhan commented Sep 24, 2020

ali-ramadhan commented Sep 28, 2020

Obscure GPU error when time stepping a model with Julia 1.5 and Oceananigans v0.33.0 #828

Obscure GPU error when time stepping a model with Julia 1.5 and Oceananigans v0.33.0 #828

Comments

ali-ramadhan commented Aug 3, 2020 • edited Loading

Julia 1.4.2

Julia 1.5.0

glwagner commented Aug 3, 2020

ali-ramadhan commented Aug 13, 2020

maleadt commented Aug 13, 2020 • edited Loading

maleadt commented Aug 13, 2020

ali-ramadhan commented Aug 13, 2020

maleadt commented Aug 13, 2020

glwagner commented Sep 24, 2020 • edited Loading

maleadt commented Sep 24, 2020

ali-ramadhan commented Sep 24, 2020

ali-ramadhan commented Sep 28, 2020

ali-ramadhan commented Aug 3, 2020 •

edited

Loading

maleadt commented Aug 13, 2020 •

edited

Loading

glwagner commented Sep 24, 2020 •

edited

Loading