Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Testing GPU compatible SF in ClimaAtmos #2235

Closed
3 tasks done
Tracked by #1980
akshaysridhar opened this issue Oct 12, 2023 · 6 comments
Closed
3 tasks done
Tracked by #1980

Testing GPU compatible SF in ClimaAtmos #2235

akshaysridhar opened this issue Oct 12, 2023 · 6 comments
Assignees
@akshaysridhar @sriharshakandala
Labels
enhancement New feature or request
Milestone
O1.2.1 Clean up C...

Comments

@akshaysridhar
Copy link
Member

akshaysridhar commented Oct 12, 2023 •
edited
Loading

This issue tracks outcomes from a series of WIP branches (most of which have corresponding PRs) related to GPU compatibility in ClimaAtmos. Testing on NVIDIA A100-SXM4-80GB.

Relevant branches

  • ClimaAtmos main
  • SurfaceFluxes v0.8.0
  • ClimaAtmos as/up-sf-fluxes.

Tasks
Beta Give feedback

cc @sriharshakandala
@akshaysridhar akshaysridhar added the enhancement New feature or request label Oct 12, 2023
@akshaysridhar
Copy link
Member Author

akshaysridhar commented Oct 12, 2023 •
edited
Loading

AP Test Case (Fail) : Stacktrace

asridhar@clima ~/Codes/ClimaAtmos.jl (as/up-sf-fluxes)$ julia --project=examples examples/hybrid/driver.jl --config config/longrun_configs/longrun_aquaplanet_rhoe_equil_clearsky_highres_hightop_rayleigh35e3_ft64.yml
[ Info: Running on CUDADevice.
[ Info: Setting up single-process ClimaAtmos run
[ Info: Running ClimaCore in unthreaded mode.
┌ Info: AtmosModel:
│
│                    `model_config`::`ClimaAtmos.SphericalModel`
│                       `perf_mode`::`ClimaAtmos.PerfStandard`
│                  `moisture_model`::`ClimaAtmos.EquilMoistModel`
│                     `energy_form`::`ClimaAtmos.TotalEnergy`
│                    `precip_model`::`ClimaAtmos.Microphysics0Moment`
│                    `forcing_type`::`Nothing`
│                      `subsidence`::`Nothing`
│                  `radiation_mode`::`ClimaAtmos.RRTMGPInterface.ClearSkyRadiation`
│                          `ls_adv`::`Nothing`
│                   `edmf_coriolis`::`Nothing`
│                `edmfx_entr_model`::`ClimaAtmos.NoEntrainment`
│                `edmfx_detr_model`::`ClimaAtmos.NoDetrainment`
│                  `turbconv_model`::`Nothing`
│     `non_orographic_gravity_wave`::`Nothing`
│         `orographic_gravity_wave`::`Nothing`
│                       `hyperdiff`::`ClimaAtmos.ClimaHyperdiffusion{Float32}`
│                       `vert_diff`::`ClimaAtmos.VerticalDiffusion{true, Float32}`
│                  `viscous_sponge`::`Nothing`
│                 `rayleigh_sponge`::`ClimaAtmos.RayleighSponge{Float32}`
│                 `sfc_temperature`::`ClimaAtmos.ZonallySymmetricSST`
└                   `surface_model`::`ClimaAtmos.PrescribedSurfaceTemperature`
┌ Info: numerics
│   energy_upwinding = Val{:none}()
│   tracer_upwinding = Val{:none}()
│   density_upwinding = Val{:none}()
│   edmfx_upwinding = Val{:none}()
│   apply_limiter = false
└   bubble = true
┌ Info: Time info:
│   dt = "150secs"
│   t_end = "300days"
└   floor_n_steps = 172800
┌ Info: Topography
└   topography = "NoWarp"
┌ Info: z heights
│   z_mesh.faces =
│    51-element Vector{ClimaCore.Geometry.ZPoint{Float32}}:
│     ZPoint(0.0)
│     ZPoint(30.0)
│     ZPoint(76.95174)
│     ZPoint(141.39532)
│     ZPoint(223.902)
│     ZPoint(325.0765)
│     ZPoint(445.55963)
│     ZPoint(586.0296)
│     ZPoint(747.2027)
│     ZPoint(929.832)
│     ZPoint(1134.7045)
│     ZPoint(1362.6383)
│     ZPoint(1614.4788)
│     ZPoint(1891.0958)
│     ⋮
│     ZPoint(20277.682)
│     ZPoint(21690.9)
│     ZPoint(23187.03)
│     ZPoint(24772.832)
│     ZPoint(26456.123)
│     ZPoint(28246.012)
│     ZPoint(30153.2)
│     ZPoint(32190.39)
│     ZPoint(34372.832)
│     ZPoint(36719.05)
│     ZPoint(39251.89)
│     ZPoint(42000.0)
└     ZPoint(45000.0)
┌ Info: Resolution stats:
│   Nq = 4
│   h_elem = 16
│   z_elem = 50
│   ncols = 24576
└   ndofs_total = 1228800
[ Info: Allocating Y: 24.993 s (44201897 allocations: 2.52 GiB)
┌ Error: @cuprint does not support values of type Nothing
└ @ CUDA ~/.julia/packages/CUDA/35NC6/src/device/intrinsics/output.jl:168
ERROR: LoadError: GPU compilation of MethodInstance for ClimaCore.DataLayouts.knl_copyto!(::ClimaCore.DataLayouts.VIJFH{Float32, 4, CUDA.CuDeviceArray{Float32, 5, 1}}, ::Base.Broadcast.Broadcasted{ClimaCore.DataLayouts.VIJFHStyle{4, CUDA.CuArray{Float32, N, CUDA.Mem.DeviceBuffer} where N}, NTuple{5, Base.OneTo{Int64}}, ClimaCore.Fields.var"#3#4"{Float32}, Tuple{Base.Broadcast.Broadcasted{ClimaCore.DataLayouts.VIJFHStyle{4, CUDA.CuArray{Float32, N, CUDA.Mem.DeviceBuffer} where N}, Nothing, Dierckx.Spline1D, Tuple{ClimaCore.DataLayouts.VIJFH{Float32, 4, CUDA.CuDeviceArray{Float32, 5, 1}}}}}}) failed
KernelError: passing and using non-bitstype argument

Argument 3 to your kernel function is of type Base.Broadcast.Broadcasted{ClimaCore.DataLayouts.VIJFHStyle{4, CUDA.CuArray{Float32, N, CUDA.Mem.DeviceBuffer} where N}, NTuple{5, Base.OneTo{Int64}}, ClimaCore.Fields.var"#3#4"{Float32}, Tuple{Base.Broadcast.Broadcasted{ClimaCore.DataLayouts.VIJFHStyle{4, CUDA.CuArray{Float32, N, CUDA.Mem.DeviceBuffer} where N}, Nothing, Dierckx.Spline1D, Tuple{ClimaCore.DataLayouts.VIJFH{Float32, 4, CUDA.CuDeviceArray{Float32, 5, 1}}}}}}, which is not isbits:
  .args is of type Tuple{Base.Broadcast.Broadcasted{ClimaCore.DataLayouts.VIJFHStyle{4, CUDA.CuArray{Float32, N, CUDA.Mem.DeviceBuffer} where N}, Nothing, Dierckx.Spline1D, Tuple{ClimaCore.DataLayouts.VIJFH{Float32, 4, CUDA.CuDeviceArray{Float32, 5, 1}}}}} which is not isbits.
    .1 is of type Base.Broadcast.Broadcasted{ClimaCore.DataLayouts.VIJFHStyle{4, CUDA.CuArray{Float32, N, CUDA.Mem.DeviceBuffer} where N}, Nothing, Dierckx.Spline1D, Tuple{ClimaCore.DataLayouts.VIJFH{Float32, 4, CUDA.CuDeviceArray{Float32, 5, 1}}}} which is not isbits.
      .f is of type Dierckx.Spline1D which is not isbits.
        .t is of type Vector{Float64} which is not isbits.
        .c is of type Vector{Float64} which is not isbits.
        .wrk is of type Vector{Float64} which is not isbits.


Stacktrace:
  [1] check_invocation(job::GPUCompiler.CompilerJob)
    @ GPUCompiler ~/.julia/packages/GPUCompiler/YO8Uj/src/validation.jl:96
  [2] macro expansion
    @ ~/.julia/packages/GPUCompiler/YO8Uj/src/driver.jl:123 [inlined]
  [3] macro expansion
    @ ~/.julia/packages/TimerOutputs/RsWnF/src/TimerOutput.jl:253 [inlined]
  [4] codegen(output::Symbol, job::GPUCompiler.CompilerJob; libraries::Bool, toplevel::Bool, optimize::Bool, cleanup::Bool, strip::Bool, validate::Bool, only_entry::Bool, parent_job::Nothing)
    @ GPUCompiler ~/.julia/packages/GPUCompiler/YO8Uj/src/driver.jl:121
  [5] codegen
    @ ~/.julia/packages/GPUCompiler/YO8Uj/src/driver.jl:110 [inlined]
  [6] compile(target::Symbol, job::GPUCompiler.CompilerJob; libraries::Bool, toplevel::Bool, optimize::Bool, cleanup::Bool, strip::Bool, validate::Bool, only_entry::Bool)
    @ GPUCompiler ~/.julia/packages/GPUCompiler/YO8Uj/src/driver.jl:106
  [7] compile
    @ ~/.julia/packages/GPUCompiler/YO8Uj/src/driver.jl:98 [inlined]
  [8] #1037
    @ ~/.julia/packages/CUDA/35NC6/src/compiler/compilation.jl:104 [inlined]
  [9] JuliaContext(f::CUDA.var"#1037#1040"{GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams}})
    @ GPUCompiler ~/.julia/packages/GPUCompiler/YO8Uj/src/driver.jl:47
 [10] compile(job::GPUCompiler.CompilerJob)
    @ CUDA ~/.julia/packages/CUDA/35NC6/src/compiler/compilation.jl:103
 [11] actual_compilation(cache::Dict{Any, CUDA.CuFunction}, src::Core.MethodInstance, world::UInt64, cfg::GPUCompiler.CompilerConfig{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams}, compiler::typeof(CUDA.compile), linker::typeof(CUDA.link))
    @ GPUCompiler ~/.julia/packages/GPUCompiler/YO8Uj/src/execution.jl:125
 [12] cached_compilation(cache::Dict{Any, CUDA.CuFunction}, src::Core.MethodInstance, cfg::GPUCompiler.CompilerConfig{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams}, compiler::Function, linker::Function)
    @ GPUCompiler ~/.julia/packages/GPUCompiler/YO8Uj/src/execution.jl:103
 [13] macro expansion
    @ ~/.julia/packages/CUDA/35NC6/src/compiler/execution.jl:318 [inlined]
 [14] macro expansion
    @ ./lock.jl:267 [inlined]
 [15] cufunction(f::typeof(ClimaCore.DataLayouts.knl_copyto!), tt::Type{Tuple{ClimaCore.DataLayouts.VIJFH{Float32, 4, CUDA.CuDeviceArray{Float32, 5, 1}}, Base.Broadcast.Broadcasted{ClimaCore.DataLayouts.VIJFHStyle{4, CUDA.CuArray{Float32, N, CUDA.Mem.DeviceBuffer} where N}, NTuple{5, Base.OneTo{Int64}}, ClimaCore.Fields.var"#3#4"{Float32}, Tuple{Base.Broadcast.Broadcasted{ClimaCore.DataLayouts.VIJFHStyle{4, CUDA.CuArray{Float32, N, CUDA.Mem.DeviceBuffer} where N}, Nothing, Dierckx.Spline1D, Tuple{ClimaCore.DataLayouts.VIJFH{Float32, 4, CUDA.CuDeviceArray{Float32, 5, 1}}}}}}}}; kwargs::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:always_inline,), Tuple{Bool}}})
    @ CUDA ~/.julia/packages/CUDA/35NC6/src/compiler/execution.jl:313
 [16] macro expansion
    @ ~/.julia/packages/CUDA/35NC6/src/compiler/execution.jl:104 [inlined]
 [17] copyto!(dest::ClimaCore.DataLayouts.VIJFH{Float32, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}}, bc::Base.Broadcast.Broadcasted{ClimaCore.DataLayouts.VIJFHStyle{4, CUDA.CuArray{Float32, N, CUDA.Mem.DeviceBuffer} where N}, NTuple{5, Base.OneTo{Int64}}, ClimaCore.Fields.var"#3#4"{Float32}, Tuple{Base.Broadcast.Broadcasted{ClimaCore.DataLayouts.VIJFHStyle{4, CUDA.CuArray{Float32, N, CUDA.Mem.DeviceBuffer} where N}, Nothing, Dierckx.Spline1D, Tuple{ClimaCore.DataLayouts.VIJFH{Float32, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}}}}}})
    @ ClimaCore.DataLayouts ~/.julia/packages/ClimaCore/MGWpt/src/DataLayouts/cuda.jl:107
 [18] copyto!
    @ ~/.julia/packages/ClimaCore/MGWpt/src/Fields/broadcast.jl:152 [inlined]
 [19] copy
    @ ~/.julia/packages/ClimaCore/MGWpt/src/Fields/broadcast.jl:98 [inlined]
 [20] materialize(bc::Base.Broadcast.Broadcasted{ClimaCore.Fields.FieldStyle{ClimaCore.DataLayouts.VIJFHStyle{4, CUDA.CuArray{Float32, N, CUDA.Mem.DeviceBuffer} where N}}, Nothing, ClimaCore.Fields.var"#3#4"{Float32}, Tuple{Base.Broadcast.Broadcasted{ClimaCore.Fields.FieldStyle{ClimaCore.DataLayouts.VIJFHStyle{4, CUDA.CuArray{Float32, N, CUDA.Mem.DeviceBuffer} where N}}, Nothing, Dierckx.Spline1D, Tuple{Field{} (trunc disp)}}}})
    @ Base.Broadcast ./broadcast.jl:873
 [21] (::ClimaAtmos.var"#96#97"{ClimaAtmos.RRTMGPInterface.BestFit, ClimaAtmos.RRTMGPInterface.SameAsInterpolation, typeof(ClimaAtmos.rrtmgp_data_loader), ClimaCore.Fields.FieldVector{Float32, NamedTuple{(:c, :f), Tuple{Field{(:ρ, :uₕ, :ρe_tot, :ρq_tot)} (trunc disp), Field{(:u₃,)} (trunc disp)}}}, NamedTuple{(:is_init, :simulation, :atmos, :comms_ctx, :sfc_setup, :test, :moisture_model, :model_config, :limiter, :ᶜΦ, :ᶠgradᵥ_ᶜΦ, :ᶜρ_ref, :ᶜp_ref, :ᶜT, :ᶜf, :∂ᶜK∂ᶠu₃_data, :params, :energy_upwinding, :tracer_upwinding, :density_upwinding, :edmfx_upwinding, :do_dss, :ghost_buffer, :net_energy_flux_toa, :net_energy_flux_sfc, :env_thermo_quad, :ᶜspecific, :ᶜu, :ᶠu³, :ᶜK, :ᶜts, :ᶜp, :ᶜh_tot, :sfc_conditions, :ᶠtemp_scalar, :ᶜtemp_scalar, :ᶜtemp_scalar_2, :temp_data_level, :temp_data_level_2, :temp_data_level_3, :ᶜtemp_CT3, :ᶠtemp_CT3, :ᶠtemp_CT12, :ᶠtemp_CT12ʲs, :ᶠtemp_C123, :ᶜtemp_UVWxUVW, :ᶠtemp_UVWxUVW, :sfc_temp_C3, :ᶜ∇²u, :ᶜ∇²specific_energy, :ᶜ∇²specific_tracers, :hyperdiffusion_ghost_buffer, :ᶜ∇²uʲs), Tuple{Base.RefValue{Bool}, NamedTuple{(:comms_ctx, :is_debugging_tc, :output_dir, :restart, :job_id, :dt, :start_date, :t_end), Tuple{ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, Bool, String, Bool, String, Float32, Dates.DateTime, Float32}}, ClimaAtmos.AtmosModel{ClimaAtmos.SphericalModel, ClimaAtmos.PerfStandard, ClimaAtmos.EquilMoistModel, ClimaAtmos.TotalEnergy, ClimaAtmos.Microphysics0Moment, Nothing, Nothing, ClimaAtmos.RRTMGPInterface.ClearSkyRadiation, Nothing, Nothing, Bool, ClimaAtmos.NoEntrainment, ClimaAtmos.NoDetrainment, Bool, Bool, Bool, Nothing, Nothing, Nothing, ClimaAtmos.ClimaHyperdiffusion{Float32}, ClimaAtmos.VerticalDiffusion{true, Float32}, Nothing, ClimaAtmos.RayleighSponge{Float32}, ClimaAtmos.ZonallySymmetricSST, ClimaAtmos.PrescribedSurfaceTemperature}, ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, ClimaAtmos.SurfaceConditions.SurfaceState{Float32, ClimaAtmos.SurfaceConditions.MoninObukhov{Float32, Nothing, Nothing}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Nothing, ClimaAtmos.EquilMoistModel, ClimaAtmos.SphericalModel, Nothing, Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), ClimaAtmos.Parameters.ClimaAtmosParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}, RRTMGP.Parameters.RRTMGPParameters{Float32}, Insolation.Parameters.InsolationParameters{Float32}, CloudMicrophysics.Parameters.CloudMicrophysicsParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, ClimaAtmos.TurbulenceConvection.Parameters.TurbulenceConvectionParameters{Float32, CloudMicrophysics.Parameters.CloudMicrophysicsParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}}}, Val{:none}, Val{:none}, Val{:none}, Val{:none}, Bool, NamedTuple{(:c, :f), Tuple{ClimaCore.Spaces.DSSBuffer{NamedTuple{(:ρ, :uₕ, :ρe_tot, :ρq_tot), Tuple{Float32, ClimaCore.Geometry.Covariant12Vector{Float32}, Float32, Float32}}, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{NamedTuple{(:ρ, :uₕ, :ρe_tot, :ρq_tot), Tuple{Float32, ClimaCore.Geometry.UVVector{Float32}, Float32, Float32}}, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}, ClimaCore.Spaces.DSSBuffer{NamedTuple{(:u₃,), Tuple{ClimaCore.Geometry.Covariant3Vector{Float32}}}, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{NamedTuple{(:u₃,), Tuple{ClimaCore.Geometry.Covariant3Vector{Float32}}}, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}}}, Vector{ClimaCore.Geometry.WVector{Float32}}, Vector{ClimaCore.Geometry.WVector{Float32}}, ClimaAtmos.SGSQuadrature{3, ClimaAtmos.GaussianQuad, StaticArraysCore.SVector{3, Float32}, StaticArraysCore.SVector{3, Float32}}, Field{(:e_tot, :q_tot)} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{(:ts, :ustar, :obukhov_length, :buoyancy_flux, :ρ_flux_uₕ, :ρ_flux_h_tot, :ρ_flux_q_tot)} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), ClimaCore.DataLayouts.IJFH{Float32, 4, SubArray{Float32, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}, Tuple{Int64, Vararg{Base.Slice{Base.OneTo{Int64}}, 4}}, true}}, ClimaCore.DataLayouts.IJFH{Float32, 4, SubArray{Float32, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}, Tuple{Int64, Vararg{Base.Slice{Base.OneTo{Int64}}, 4}}, true}}, ClimaCore.DataLayouts.IJFH{Float32, 4, SubArray{Float32, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}, Tuple{Int64, Vararg{Base.Slice{Base.OneTo{Int64}}, 4}}, true}}, Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{(:q_tot,)} (trunc disp), NamedTuple{(:ᶜ∇²u, :ᶜ∇²specific_energy, :ᶜ∇²specific_tracers), Tuple{ClimaCore.Spaces.DSSBuffer{ClimaCore.Geometry.Covariant123Vector{Float32}, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{ClimaCore.Geometry.UVWVector{Float32}, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}, ClimaCore.Spaces.DSSBuffer{Float32, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{Float32, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}, ClimaCore.Spaces.DSSBuffer{NamedTuple{(:q_tot,), Tuple{Float32}}, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{NamedTuple{(:q_tot,), Tuple{Float32}}, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}}}, NamedTuple{(), Tuple{}}}}, ClimaAtmos.RRTMGPInterface.ClearSkyRadiation, RRTMGP.Parameters.RRTMGPParameters{Float32}, DataType, Bool, Bool, ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}})(input_data::NCDatasets.NCDataset{Nothing})
    @ ClimaAtmos ~/.julia/packages/ClimaAtmos/Y6nUf/src/parameterized_tendencies/radiation/radiation.jl:86
 [22] #92
    @ ~/.julia/packages/ClimaAtmos/Y6nUf/src/parameterized_tendencies/radiation/radiation_utilities.jl:34 [inlined]
 [23] NCDatasets.NCDataset(::ClimaAtmos.var"#92#93"{ClimaAtmos.var"#96#97"{ClimaAtmos.RRTMGPInterface.BestFit, ClimaAtmos.RRTMGPInterface.SameAsInterpolation, typeof(ClimaAtmos.rrtmgp_data_loader), ClimaCore.Fields.FieldVector{Float32, NamedTuple{(:c, :f), Tuple{Field{(:ρ, :uₕ, :ρe_tot, :ρq_tot)} (trunc disp), Field{(:u₃,)} (trunc disp)}}}, NamedTuple{(:is_init, :simulation, :atmos, :comms_ctx, :sfc_setup, :test, :moisture_model, :model_config, :limiter, :ᶜΦ, :ᶠgradᵥ_ᶜΦ, :ᶜρ_ref, :ᶜp_ref, :ᶜT, :ᶜf, :∂ᶜK∂ᶠu₃_data, :params, :energy_upwinding, :tracer_upwinding, :density_upwinding, :edmfx_upwinding, :do_dss, :ghost_buffer, :net_energy_flux_toa, :net_energy_flux_sfc, :env_thermo_quad, :ᶜspecific, :ᶜu, :ᶠu³, :ᶜK, :ᶜts, :ᶜp, :ᶜh_tot, :sfc_conditions, :ᶠtemp_scalar, :ᶜtemp_scalar, :ᶜtemp_scalar_2, :temp_data_level, :temp_data_level_2, :temp_data_level_3, :ᶜtemp_CT3, :ᶠtemp_CT3, :ᶠtemp_CT12, :ᶠtemp_CT12ʲs, :ᶠtemp_C123, :ᶜtemp_UVWxUVW, :ᶠtemp_UVWxUVW, :sfc_temp_C3, :ᶜ∇²u, :ᶜ∇²specific_energy, :ᶜ∇²specific_tracers, :hyperdiffusion_ghost_buffer, :ᶜ∇²uʲs), Tuple{Base.RefValue{Bool}, NamedTuple{(:comms_ctx, :is_debugging_tc, :output_dir, :restart, :job_id, :dt, :start_date, :t_end), Tuple{ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, Bool, String, Bool, String, Float32, Dates.DateTime, Float32}}, ClimaAtmos.AtmosModel{ClimaAtmos.SphericalModel, ClimaAtmos.PerfStandard, ClimaAtmos.EquilMoistModel, ClimaAtmos.TotalEnergy, ClimaAtmos.Microphysics0Moment, Nothing, Nothing, ClimaAtmos.RRTMGPInterface.ClearSkyRadiation, Nothing, Nothing, Bool, ClimaAtmos.NoEntrainment, ClimaAtmos.NoDetrainment, Bool, Bool, Bool, Nothing, Nothing, Nothing, ClimaAtmos.ClimaHyperdiffusion{Float32}, ClimaAtmos.VerticalDiffusion{true, Float32}, Nothing, ClimaAtmos.RayleighSponge{Float32}, ClimaAtmos.ZonallySymmetricSST, ClimaAtmos.PrescribedSurfaceTemperature}, ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, ClimaAtmos.SurfaceConditions.SurfaceState{Float32, ClimaAtmos.SurfaceConditions.MoninObukhov{Float32, Nothing, Nothing}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Nothing, ClimaAtmos.EquilMoistModel, ClimaAtmos.SphericalModel, Nothing, Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), ClimaAtmos.Parameters.ClimaAtmosParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}, RRTMGP.Parameters.RRTMGPParameters{Float32}, Insolation.Parameters.InsolationParameters{Float32}, CloudMicrophysics.Parameters.CloudMicrophysicsParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, ClimaAtmos.TurbulenceConvection.Parameters.TurbulenceConvectionParameters{Float32, CloudMicrophysics.Parameters.CloudMicrophysicsParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}}}, Val{:none}, Val{:none}, Val{:none}, Val{:none}, Bool, NamedTuple{(:c, :f), Tuple{ClimaCore.Spaces.DSSBuffer{NamedTuple{(:ρ, :uₕ, :ρe_tot, :ρq_tot), Tuple{Float32, ClimaCore.Geometry.Covariant12Vector{Float32}, Float32, Float32}}, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{NamedTuple{(:ρ, :uₕ, :ρe_tot, :ρq_tot), Tuple{Float32, ClimaCore.Geometry.UVVector{Float32}, Float32, Float32}}, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}, ClimaCore.Spaces.DSSBuffer{NamedTuple{(:u₃,), Tuple{ClimaCore.Geometry.Covariant3Vector{Float32}}}, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{NamedTuple{(:u₃,), Tuple{ClimaCore.Geometry.Covariant3Vector{Float32}}}, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}}}, Vector{ClimaCore.Geometry.WVector{Float32}}, Vector{ClimaCore.Geometry.WVector{Float32}}, ClimaAtmos.SGSQuadrature{3, ClimaAtmos.GaussianQuad, StaticArraysCore.SVector{3, Float32}, StaticArraysCore.SVector{3, Float32}}, Field{(:e_tot, :q_tot)} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{(:ts, :ustar, :obukhov_length, :buoyancy_flux, :ρ_flux_uₕ, :ρ_flux_h_tot, :ρ_flux_q_tot)} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), ClimaCore.DataLayouts.IJFH{Float32, 4, SubArray{Float32, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}, Tuple{Int64, Vararg{Base.Slice{Base.OneTo{Int64}}, 4}}, true}}, ClimaCore.DataLayouts.IJFH{Float32, 4, SubArray{Float32, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}, Tuple{Int64, Vararg{Base.Slice{Base.OneTo{Int64}}, 4}}, true}}, ClimaCore.DataLayouts.IJFH{Float32, 4, SubArray{Float32, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}, Tuple{Int64, Vararg{Base.Slice{Base.OneTo{Int64}}, 4}}, true}}, Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{(:q_tot,)} (trunc disp), NamedTuple{(:ᶜ∇²u, :ᶜ∇²specific_energy, :ᶜ∇²specific_tracers), Tuple{ClimaCore.Spaces.DSSBuffer{ClimaCore.Geometry.Covariant123Vector{Float32}, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{ClimaCore.Geometry.UVWVector{Float32}, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}, ClimaCore.Spaces.DSSBuffer{Float32, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{Float32, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}, ClimaCore.Spaces.DSSBuffer{NamedTuple{(:q_tot,), Tuple{Float32}}, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{NamedTuple{(:q_tot,), Tuple{Float32}}, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}}}, NamedTuple{(), Tuple{}}}}, ClimaAtmos.RRTMGPInterface.ClearSkyRadiation, RRTMGP.Parameters.RRTMGPParameters{Float32}, DataType, Bool, Bool, ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}}}, ::String, ::Vararg{String}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ NCDatasets ~/.julia/packages/NCDatasets/c8XyT/src/dataset.jl:220
 [24] NCDataset
    @ ~/.julia/packages/NCDatasets/c8XyT/src/dataset.jl:217 [inlined]
 [25] rrtmgp_data_loader
    @ ~/.julia/packages/ClimaAtmos/Y6nUf/src/parameterized_tendencies/radiation/radiation_utilities.jl:33 [inlined]
 [26] radiation_model_cache(Y::ClimaCore.Fields.FieldVector{Float32, NamedTuple{(:c, :f), Tuple{Field{(:ρ, :uₕ, :ρe_tot, :ρq_tot)} (trunc disp), Field{(:u₃,)} (trunc disp)}}}, default_cache::NamedTuple{(:is_init, :simulation, :atmos, :comms_ctx, :sfc_setup, :test, :moisture_model, :model_config, :limiter, :ᶜΦ, :ᶠgradᵥ_ᶜΦ, :ᶜρ_ref, :ᶜp_ref, :ᶜT, :ᶜf, :∂ᶜK∂ᶠu₃_data, :params, :energy_upwinding, :tracer_upwinding, :density_upwinding, :edmfx_upwinding, :do_dss, :ghost_buffer, :net_energy_flux_toa, :net_energy_flux_sfc, :env_thermo_quad, :ᶜspecific, :ᶜu, :ᶠu³, :ᶜK, :ᶜts, :ᶜp, :ᶜh_tot, :sfc_conditions, :ᶠtemp_scalar, :ᶜtemp_scalar, :ᶜtemp_scalar_2, :temp_data_level, :temp_data_level_2, :temp_data_level_3, :ᶜtemp_CT3, :ᶠtemp_CT3, :ᶠtemp_CT12, :ᶠtemp_CT12ʲs, :ᶠtemp_C123, :ᶜtemp_UVWxUVW, :ᶠtemp_UVWxUVW, :sfc_temp_C3, :ᶜ∇²u, :ᶜ∇²specific_energy, :ᶜ∇²specific_tracers, :hyperdiffusion_ghost_buffer, :ᶜ∇²uʲs), Tuple{Base.RefValue{Bool}, NamedTuple{(:comms_ctx, :is_debugging_tc, :output_dir, :restart, :job_id, :dt, :start_date, :t_end), Tuple{ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, Bool, String, Bool, String, Float32, Dates.DateTime, Float32}}, ClimaAtmos.AtmosModel{ClimaAtmos.SphericalModel, ClimaAtmos.PerfStandard, ClimaAtmos.EquilMoistModel, ClimaAtmos.TotalEnergy, ClimaAtmos.Microphysics0Moment, Nothing, Nothing, ClimaAtmos.RRTMGPInterface.ClearSkyRadiation, Nothing, Nothing, Bool, ClimaAtmos.NoEntrainment, ClimaAtmos.NoDetrainment, Bool, Bool, Bool, Nothing, Nothing, Nothing, ClimaAtmos.ClimaHyperdiffusion{Float32}, ClimaAtmos.VerticalDiffusion{true, Float32}, Nothing, ClimaAtmos.RayleighSponge{Float32}, ClimaAtmos.ZonallySymmetricSST, ClimaAtmos.PrescribedSurfaceTemperature}, ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, ClimaAtmos.SurfaceConditions.SurfaceState{Float32, ClimaAtmos.SurfaceConditions.MoninObukhov{Float32, Nothing, Nothing}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Nothing, ClimaAtmos.EquilMoistModel, ClimaAtmos.SphericalModel, Nothing, Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), ClimaAtmos.Parameters.ClimaAtmosParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}, RRTMGP.Parameters.RRTMGPParameters{Float32}, Insolation.Parameters.InsolationParameters{Float32}, CloudMicrophysics.Parameters.CloudMicrophysicsParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, ClimaAtmos.TurbulenceConvection.Parameters.TurbulenceConvectionParameters{Float32, CloudMicrophysics.Parameters.CloudMicrophysicsParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}}}, Val{:none}, Val{:none}, Val{:none}, Val{:none}, Bool, NamedTuple{(:c, :f), Tuple{ClimaCore.Spaces.DSSBuffer{NamedTuple{(:ρ, :uₕ, :ρe_tot, :ρq_tot), Tuple{Float32, ClimaCore.Geometry.Covariant12Vector{Float32}, Float32, Float32}}, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{NamedTuple{(:ρ, :uₕ, :ρe_tot, :ρq_tot), Tuple{Float32, ClimaCore.Geometry.UVVector{Float32}, Float32, Float32}}, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}, ClimaCore.Spaces.DSSBuffer{NamedTuple{(:u₃,), Tuple{ClimaCore.Geometry.Covariant3Vector{Float32}}}, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{NamedTuple{(:u₃,), Tuple{ClimaCore.Geometry.Covariant3Vector{Float32}}}, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}}}, Vector{ClimaCore.Geometry.WVector{Float32}}, Vector{ClimaCore.Geometry.WVector{Float32}}, ClimaAtmos.SGSQuadrature{3, ClimaAtmos.GaussianQuad, StaticArraysCore.SVector{3, Float32}, StaticArraysCore.SVector{3, Float32}}, Field{(:e_tot, :q_tot)} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{(:ts, :ustar, :obukhov_length, :buoyancy_flux, :ρ_flux_uₕ, :ρ_flux_h_tot, :ρ_flux_q_tot)} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), ClimaCore.DataLayouts.IJFH{Float32, 4, SubArray{Float32, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}, Tuple{Int64, Vararg{Base.Slice{Base.OneTo{Int64}}, 4}}, true}}, ClimaCore.DataLayouts.IJFH{Float32, 4, SubArray{Float32, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}, Tuple{Int64, Vararg{Base.Slice{Base.OneTo{Int64}}, 4}}, true}}, ClimaCore.DataLayouts.IJFH{Float32, 4, SubArray{Float32, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}, Tuple{Int64, Vararg{Base.Slice{Base.OneTo{Int64}}, 4}}, true}}, Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{(:q_tot,)} (trunc disp), NamedTuple{(:ᶜ∇²u, :ᶜ∇²specific_energy, :ᶜ∇²specific_tracers), Tuple{ClimaCore.Spaces.DSSBuffer{ClimaCore.Geometry.Covariant123Vector{Float32}, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{ClimaCore.Geometry.UVWVector{Float32}, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}, ClimaCore.Spaces.DSSBuffer{Float32, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{Float32, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}, ClimaCore.Spaces.DSSBuffer{NamedTuple{(:q_tot,), Tuple{Float32}}, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{NamedTuple{(:q_tot,), Tuple{Float32}}, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}}}, NamedTuple{(), Tuple{}}}}, params::ClimaAtmos.Parameters.ClimaAtmosParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}, RRTMGP.Parameters.RRTMGPParameters{Float32}, Insolation.Parameters.InsolationParameters{Float32}, CloudMicrophysics.Parameters.CloudMicrophysicsParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, ClimaAtmos.TurbulenceConvection.Parameters.TurbulenceConvectionParameters{Float32, CloudMicrophysics.Parameters.CloudMicrophysicsParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}}}, radiation_mode::ClimaAtmos.RRTMGPInterface.ClearSkyRadiation; interpolation::ClimaAtmos.RRTMGPInterface.BestFit, bottom_extrapolation::ClimaAtmos.RRTMGPInterface.SameAsInterpolation, data_loader::typeof(ClimaAtmos.rrtmgp_data_loader))
    @ ClimaAtmos ~/.julia/packages/ClimaAtmos/Y6nUf/src/parameterized_tendencies/radiation/radiation.jl:60
 [27] radiation_model_cache
    @ ~/.julia/packages/ClimaAtmos/Y6nUf/src/parameterized_tendencies/radiation/radiation.jl:29 [inlined]
 [28] additional_cache(Y::ClimaCore.Fields.FieldVector{Float32, NamedTuple{(:c, :f), Tuple{Field{(:ρ, :uₕ, :ρe_tot, :ρq_tot)} (trunc disp), Field{(:u₃,)} (trunc disp)}}}, default_cache::NamedTuple{(:is_init, :simulation, :atmos, :comms_ctx, :sfc_setup, :test, :moisture_model, :model_config, :limiter, :ᶜΦ, :ᶠgradᵥ_ᶜΦ, :ᶜρ_ref, :ᶜp_ref, :ᶜT, :ᶜf, :∂ᶜK∂ᶠu₃_data, :params, :energy_upwinding, :tracer_upwinding, :density_upwinding, :edmfx_upwinding, :do_dss, :ghost_buffer, :net_energy_flux_toa, :net_energy_flux_sfc, :env_thermo_quad, :ᶜspecific, :ᶜu, :ᶠu³, :ᶜK, :ᶜts, :ᶜp, :ᶜh_tot, :sfc_conditions, :ᶠtemp_scalar, :ᶜtemp_scalar, :ᶜtemp_scalar_2, :temp_data_level, :temp_data_level_2, :temp_data_level_3, :ᶜtemp_CT3, :ᶠtemp_CT3, :ᶠtemp_CT12, :ᶠtemp_CT12ʲs, :ᶠtemp_C123, :ᶜtemp_UVWxUVW, :ᶠtemp_UVWxUVW, :sfc_temp_C3, :ᶜ∇²u, :ᶜ∇²specific_energy, :ᶜ∇²specific_tracers, :hyperdiffusion_ghost_buffer, :ᶜ∇²uʲs), Tuple{Base.RefValue{Bool}, NamedTuple{(:comms_ctx, :is_debugging_tc, :output_dir, :restart, :job_id, :dt, :start_date, :t_end), Tuple{ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, Bool, String, Bool, String, Float32, Dates.DateTime, Float32}}, ClimaAtmos.AtmosModel{ClimaAtmos.SphericalModel, ClimaAtmos.PerfStandard, ClimaAtmos.EquilMoistModel, ClimaAtmos.TotalEnergy, ClimaAtmos.Microphysics0Moment, Nothing, Nothing, ClimaAtmos.RRTMGPInterface.ClearSkyRadiation, Nothing, Nothing, Bool, ClimaAtmos.NoEntrainment, ClimaAtmos.NoDetrainment, Bool, Bool, Bool, Nothing, Nothing, Nothing, ClimaAtmos.ClimaHyperdiffusion{Float32}, ClimaAtmos.VerticalDiffusion{true, Float32}, Nothing, ClimaAtmos.RayleighSponge{Float32}, ClimaAtmos.ZonallySymmetricSST, ClimaAtmos.PrescribedSurfaceTemperature}, ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, ClimaAtmos.SurfaceConditions.SurfaceState{Float32, ClimaAtmos.SurfaceConditions.MoninObukhov{Float32, Nothing, Nothing}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Nothing, ClimaAtmos.EquilMoistModel, ClimaAtmos.SphericalModel, Nothing, Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), ClimaAtmos.Parameters.ClimaAtmosParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}, RRTMGP.Parameters.RRTMGPParameters{Float32}, Insolation.Parameters.InsolationParameters{Float32}, CloudMicrophysics.Parameters.CloudMicrophysicsParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, ClimaAtmos.TurbulenceConvection.Parameters.TurbulenceConvectionParameters{Float32, CloudMicrophysics.Parameters.CloudMicrophysicsParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}}}, Val{:none}, Val{:none}, Val{:none}, Val{:none}, Bool, NamedTuple{(:c, :f), Tuple{ClimaCore.Spaces.DSSBuffer{NamedTuple{(:ρ, :uₕ, :ρe_tot, :ρq_tot), Tuple{Float32, ClimaCore.Geometry.Covariant12Vector{Float32}, Float32, Float32}}, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{NamedTuple{(:ρ, :uₕ, :ρe_tot, :ρq_tot), Tuple{Float32, ClimaCore.Geometry.UVVector{Float32}, Float32, Float32}}, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}, ClimaCore.Spaces.DSSBuffer{NamedTuple{(:u₃,), Tuple{ClimaCore.Geometry.Covariant3Vector{Float32}}}, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{NamedTuple{(:u₃,), Tuple{ClimaCore.Geometry.Covariant3Vector{Float32}}}, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}}}, Vector{ClimaCore.Geometry.WVector{Float32}}, Vector{ClimaCore.Geometry.WVector{Float32}}, ClimaAtmos.SGSQuadrature{3, ClimaAtmos.GaussianQuad, StaticArraysCore.SVector{3, Float32}, StaticArraysCore.SVector{3, Float32}}, Field{(:e_tot, :q_tot)} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{(:ts, :ustar, :obukhov_length, :buoyancy_flux, :ρ_flux_uₕ, :ρ_flux_h_tot, :ρ_flux_q_tot)} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), ClimaCore.DataLayouts.IJFH{Float32, 4, SubArray{Float32, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}, Tuple{Int64, Vararg{Base.Slice{Base.OneTo{Int64}}, 4}}, true}}, ClimaCore.DataLayouts.IJFH{Float32, 4, SubArray{Float32, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}, Tuple{Int64, Vararg{Base.Slice{Base.OneTo{Int64}}, 4}}, true}}, ClimaCore.DataLayouts.IJFH{Float32, 4, SubArray{Float32, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}, Tuple{Int64, Vararg{Base.Slice{Base.OneTo{Int64}}, 4}}, true}}, Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{} (trunc disp), Field{(:q_tot,)} (trunc disp), NamedTuple{(:ᶜ∇²u, :ᶜ∇²specific_energy, :ᶜ∇²specific_tracers), Tuple{ClimaCore.Spaces.DSSBuffer{ClimaCore.Geometry.Covariant123Vector{Float32}, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{ClimaCore.Geometry.UVWVector{Float32}, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}, ClimaCore.Spaces.DSSBuffer{Float32, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{Float32, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}, ClimaCore.Spaces.DSSBuffer{NamedTuple{(:q_tot,), Tuple{Float32}}, ClimaComms.SingletonGraphContext, ClimaCore.DataLayouts.VIFH{NamedTuple{(:q_tot,), Tuple{Float32}}, 12, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}}}}, NamedTuple{(), Tuple{}}}}, parsed_args::Dict{Any, Any}, params::ClimaAtmos.Parameters.ClimaAtmosParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}, RRTMGP.Parameters.RRTMGPParameters{Float32}, Insolation.Parameters.InsolationParameters{Float32}, CloudMicrophysics.Parameters.CloudMicrophysicsParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, ClimaAtmos.TurbulenceConvection.Parameters.TurbulenceConvectionParameters{Float32, CloudMicrophysics.Parameters.CloudMicrophysicsParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}}}, atmos::ClimaAtmos.AtmosModel{ClimaAtmos.SphericalModel, ClimaAtmos.PerfStandard, ClimaAtmos.EquilMoistModel, ClimaAtmos.TotalEnergy, ClimaAtmos.Microphysics0Moment, Nothing, Nothing, ClimaAtmos.RRTMGPInterface.ClearSkyRadiation, Nothing, Nothing, Bool, ClimaAtmos.NoEntrainment, ClimaAtmos.NoDetrainment, Bool, Bool, Bool, Nothing, Nothing, Nothing, ClimaAtmos.ClimaHyperdiffusion{Float32}, ClimaAtmos.VerticalDiffusion{true, Float32}, Nothing, ClimaAtmos.RayleighSponge{Float32}, ClimaAtmos.ZonallySymmetricSST, ClimaAtmos.PrescribedSurfaceTemperature}, dt::Float32, initial_condition::ClimaAtmos.InitialConditions.DecayingProfile)
    @ ClimaAtmos ~/.julia/packages/ClimaAtmos/Y6nUf/src/cache/cache.jl:134
 [29] get_cache(Y::ClimaCore.Fields.FieldVector{Float32, NamedTuple{(:c, :f), Tuple{Field{(:ρ, :uₕ, :ρe_tot, :ρq_tot)} (trunc disp), Field{(:u₃,)} (trunc disp)}}}, parsed_args::Dict{Any, Any}, params::ClimaAtmos.Parameters.ClimaAtmosParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}, RRTMGP.Parameters.RRTMGPParameters{Float32}, Insolation.Parameters.InsolationParameters{Float32}, CloudMicrophysics.Parameters.CloudMicrophysicsParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, ClimaAtmos.TurbulenceConvection.Parameters.TurbulenceConvectionParameters{Float32, CloudMicrophysics.Parameters.CloudMicrophysicsParameters{Float32, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}, SurfaceFluxes.Parameters.SurfaceFluxesParameters{Float32, SurfaceFluxes.UniversalFunctions.BusingerParams{Float32}, Thermodynamics.Parameters.ThermodynamicsParameters{Float32}}}}, spaces::NamedTuple{(:center_space, :face_space, :horizontal_mesh, :quad, :z_max, :z_elem, :z_stretch), Tuple{ClimaCore.Spaces.ExtrudedFiniteDifferenceSpace{ClimaCore.Spaces.CellCenter, ClimaCore.Spaces.SpectralElementSpace2D{ClimaCore.Topologies.Topology2D{ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, ClimaCore.Meshes.EquiangularCubedSphere{ClimaCore.Domains.SphereDomain{Float32}, ClimaCore.Meshes.NormalizedBilinearMap}, Vector{CartesianIndex{3}}, Array{Int64, 3}, CUDA.CuArray{Tuple{Int64, Int64, Int64, Int64, Bool}, 1, CUDA.Mem.DeviceBuffer}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, CUDA.CuArray{Tuple{Int64, Int64}, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Tuple{Bool, Int64, Int64}, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, NamedTuple{(), Tuple{}}, CUDA.CuArray{Tuple{Int64, Int64}, 1, CUDA.Mem.DeviceBuffer}}, ClimaCore.Spaces.Quadratures.GLL{4}, ClimaCore.Geometry.SphericalGlobalGeometry{Float32}, ClimaCore.DataLayouts.IJFH{ClimaCore.Geometry.LocalGeometry{(1, 2), ClimaCore.Geometry.LatLongPoint{Float32}, Float32, StaticArraysCore.SMatrix{2, 2, Float32, 4}}, 4, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, ClimaCore.DataLayouts.IJFH{Float32, 4, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, ClimaCore.DataLayouts.IFH{ClimaCore.Geometry.SurfaceGeometry{Float32, ClimaCore.Geometry.UVVector{Float32}}, 4, CUDA.CuArray{Float32, 3, CUDA.Mem.DeviceBuffer}}, NamedTuple{(), Tuple{}}}, ClimaCore.Topologies.IntervalTopology{ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.ZPoint{Float32}, Tuple{Symbol, Symbol}}, Vector{ClimaCore.Geometry.ZPoint{Float32}}}, NamedTuple{(:bottom, :top), Tuple{Int64, Int64}}}, ClimaCore.Spaces.Flat, ClimaCore.Geometry.SphericalGlobalGeometry{Float32}, ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.LatLongZPoint{Float32}, Float32, StaticArraysCore.SMatrix{3, 3, Float32, 9}}, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}}, ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.LatLongZPoint{Float32}, Float32, StaticArraysCore.SMatrix{3, 3, Float32, 9}}, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}}}, ClimaCore.Spaces.ExtrudedFiniteDifferenceSpace{ClimaCore.Spaces.CellFace, ClimaCore.Spaces.SpectralElementSpace2D{ClimaCore.Topologies.Topology2D{ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, ClimaCore.Meshes.EquiangularCubedSphere{ClimaCore.Domains.SphereDomain{Float32}, ClimaCore.Meshes.NormalizedBilinearMap}, Vector{CartesianIndex{3}}, Array{Int64, 3}, CUDA.CuArray{Tuple{Int64, Int64, Int64, Int64, Bool}, 1, CUDA.Mem.DeviceBuffer}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, CUDA.CuArray{Tuple{Int64, Int64}, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Tuple{Bool, Int64, Int64}, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Int64, 1, CUDA.Mem.DeviceBuffer}, NamedTuple{(), Tuple{}}, CUDA.CuArray{Tuple{Int64, Int64}, 1, CUDA.Mem.DeviceBuffer}}, ClimaCore.Spaces.Quadratures.GLL{4}, ClimaCore.Geometry.SphericalGlobalGeometry{Float32}, ClimaCore.DataLayouts.IJFH{ClimaCore.Geometry.LocalGeometry{(1, 2), ClimaCore.Geometry.LatLongPoint{Float32}, Float32, StaticArraysCore.SMatrix{2, 2, Float32, 4}}, 4, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, ClimaCore.DataLayouts.IJFH{Float32, 4, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}}, ClimaCore.DataLayouts.IFH{ClimaCore.Geometry.SurfaceGeometry{Float32, ClimaCore.Geometry.UVVector{Float32}}, 4, CUDA.CuArray{Float32, 3, CUDA.Mem.DeviceBuffer}}, NamedTuple{(), Tuple{}}}, ClimaCore.Topologies.IntervalTopology{ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, ClimaCore.Meshes.IntervalMesh{ClimaCore.Domains.IntervalDomain{ClimaCore.Geometry.ZPoint{Float32}, Tuple{Symbol, Symbol}}, Vector{ClimaCore.Geometry.ZPoint{Float32}}}, NamedTuple{(:bottom, :top), Tuple{Int64, Int64}}}, ClimaCore.Spaces.Flat, ClimaCore.Geometry.SphericalGlobalGeometry{Float32}, ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.LatLongZPoint{Float32}, Float32, StaticArraysCore.SMatrix{3, 3, Float32, 9}}, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}}, ClimaCore.DataLayouts.VIJFH{ClimaCore.Geometry.LocalGeometry{(1, 2, 3), ClimaCore.Geometry.LatLongZPoint{Float32}, Float32, StaticArraysCore.SMatrix{3, 3, Float32, 9}}, 4, CUDA.CuArray{Float32, 5, CUDA.Mem.DeviceBuffer}}}, ClimaCore.Meshes.EquiangularCubedSphere{ClimaCore.Domains.SphereDomain{Float32}, ClimaCore.Meshes.NormalizedBilinearMap}, ClimaCore.Spaces.Quadratures.GLL{4}, Float32, Int64, ClimaCore.Meshes.GeneralizedExponentialStretching{Float32}}}, atmos::ClimaAtmos.AtmosModel{ClimaAtmos.SphericalModel, ClimaAtmos.PerfStandard, ClimaAtmos.EquilMoistModel, ClimaAtmos.TotalEnergy, ClimaAtmos.Microphysics0Moment, Nothing, Nothing, ClimaAtmos.RRTMGPInterface.ClearSkyRadiation, Nothing, Nothing, Bool, ClimaAtmos.NoEntrainment, ClimaAtmos.NoDetrainment, Bool, Bool, Bool, Nothing, Nothing, Nothing, ClimaAtmos.ClimaHyperdiffusion{Float32}, ClimaAtmos.VerticalDiffusion{true, Float32}, Nothing, ClimaAtmos.RayleighSponge{Float32}, ClimaAtmos.ZonallySymmetricSST, ClimaAtmos.PrescribedSurfaceTemperature}, numerics::NamedTuple{(:energy_upwinding, :tracer_upwinding, :density_upwinding, :edmfx_upwinding, :apply_limiter, :bubble), Tuple{Val{:none}, Val{:none}, Val{:none}, Val{:none}, Bool, Bool}}, simulation::NamedTuple{(:comms_ctx, :is_debugging_tc, :output_dir, :restart, :job_id, :dt, :start_date, :t_end), Tuple{ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, Bool, String, Bool, String, Float32, Dates.DateTime, Float32}}, initial_condition::ClimaAtmos.InitialConditions.DecayingProfile, surface_setup::ClimaAtmos.SurfaceConditions.DefaultMoninObukhov)
    @ ClimaAtmos ~/.julia/packages/ClimaAtmos/Y6nUf/src/solver/type_getters.jl:562
 [30] macro expansion
    @ ~/.julia/packages/ClimaAtmos/Y6nUf/src/solver/type_getters.jl:819 [inlined]
 [31] macro expansion
    @ ./timing.jl:501 [inlined]
 [32] macro expansion
    @ ~/.julia/packages/ClimaAtmos/Y6nUf/src/utils/utilities.jl:257 [inlined]
 [33] get_integrator(config::ClimaAtmos.AtmosConfig{Float32, CLIMAParameters.AliasParamDict{Float32}, Dict{Any, Any}, ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}})
    @ ClimaAtmos ~/.julia/packages/ClimaAtmos/Y6nUf/src/solver/type_getters.jl:818
 [34] top-level scope
    @ ~/Codes/ClimaAtmos.jl/examples/hybrid/driver.jl:8
in expression starting at /home/asridhar/Codes/ClimaAtmos.jl/examples/hybrid/driver.jl:8

@akshaysridhar
Copy link
Member Author

akshaysridhar commented Oct 12, 2023 •
edited
Loading

Baroclinic Wave SF Interface (similar outcome for Held-Suarez w/ MO) Test (Pass)

asridhar@clima ~/Codes/ClimaAtmos.jl (as/up-sf-fluxes)$ julia --project=examples examples/hybrid/driver.jl --config config/gpu_configs/target_gpu_implicit_moist_baroclinic_wave_sfmo.yml
[ Info: Running on CUDADevice.
[ Info: Setting up single-process ClimaAtmos run
[ Info: Running ClimaCore in unthreaded mode.
┌ Info: AtmosModel:
│
│                    `model_config`::`ClimaAtmos.SphericalModel`
│                       `perf_mode`::`ClimaAtmos.PerfStandard`
│                  `moisture_model`::`ClimaAtmos.EquilMoistModel`
│                     `energy_form`::`ClimaAtmos.TotalEnergy`
│                    `precip_model`::`ClimaAtmos.Microphysics0Moment`
│                    `forcing_type`::`Nothing`
│                      `subsidence`::`Nothing`
│                  `radiation_mode`::`Nothing`
│                          `ls_adv`::`Nothing`
│                   `edmf_coriolis`::`Nothing`
│                `edmfx_entr_model`::`ClimaAtmos.NoEntrainment`
│                `edmfx_detr_model`::`ClimaAtmos.NoDetrainment`
│                  `turbconv_model`::`Nothing`
│     `non_orographic_gravity_wave`::`Nothing`
│         `orographic_gravity_wave`::`Nothing`
│                       `hyperdiff`::`ClimaAtmos.ClimaHyperdiffusion{Float32}`
│                       `vert_diff`::`Nothing`
│                  `viscous_sponge`::`Nothing`
│                 `rayleigh_sponge`::`Nothing`
│                 `sfc_temperature`::`ClimaAtmos.ZonallySymmetricSST`
└                   `surface_model`::`ClimaAtmos.PrescribedSurfaceTemperature`
┌ Info: numerics
│   energy_upwinding = Val{:none}()
│   tracer_upwinding = Val{:none}()
│   density_upwinding = Val{:none}()
│   edmfx_upwinding = Val{:none}()
│   apply_limiter = false
└   bubble = true
┌ Info: Time info:
│   dt = "100secs"
│   t_end = "15days"
└   floor_n_steps = 12960
┌ Info: Topography
└   topography = "NoWarp"
┌ Info: z heights
│   z_mesh.faces =
│    64-element Vector{ClimaCore.Geometry.ZPoint{Float32}}:
│     ZPoint(0.0)
│     ZPoint(30.0)
│     ZPoint(64.432526)
│     ZPoint(103.41123)
│     ZPoint(147.05489)
│     ZPoint(195.48775)
│     ZPoint(248.8398)
│     ZPoint(307.24713)
│     ZPoint(370.8522)
│     ZPoint(439.80435)
│     ZPoint(514.26)
│     ZPoint(594.383)
│     ZPoint(680.34534)
│     ZPoint(772.3272)
│     ⋮
│     ZPoint(12489.93)
│     ZPoint(13257.988)
│     ZPoint(14084.975)
│     ZPoint(14979.377)
│     ZPoint(15951.814)
│     ZPoint(17015.83)
│     ZPoint(18189.092)
│     ZPoint(19495.287)
│     ZPoint(20967.266)
│     ZPoint(22652.527)
│     ZPoint(24623.646)
│     ZPoint(27000.0)
└     ZPoint(30000.0)
┌ Info: Resolution stats:
│   Nq = 4
│   h_elem = 30
│   z_elem = 63
│   ncols = 86400
└   ndofs_total = 5443200
[ Info: Allocating Y: 27.883 s (56900391 allocations: 2.96 GiB)
┌ Error: @cuprint does not support values of type Nothing
└ @ CUDA ~/.julia/packages/CUDA/35NC6/src/device/intrinsics/output.jl:168
[ Info: Allocating cache (p): 36.319 s (158948327 allocations: 6.56 GiB)
[ Info: Using ODE config: `ClimaTimeSteppers.ARS343`
[ Info: ode_configuration: 822.960 ms (921421 allocations: 60.39 MiB)
[ Info: get_callbacks: 115.899 ms (94339 allocations: 6.09 MiB)
[ Info: initializing diagnostics: 1.108 s (2543375 allocations: 130.26 MiB)
[ Info: Prepared diagnostic callbacks: 208.033 ms (314419 allocations: 16.13 MiB)
[ Info: Prepared SciMLBase.CallbackSet callbacks: 7.296 ms (1624 allocations: 110.04 KiB)
[ Info: n_steps_per_cycle_per_cb (non diagnostics): [864, 1]
[ Info: n_steps_per_cycle_per_cb_diagnostic: Any[]
[ Info: n_steps_per_cycle (non diagnostics): 864
[ Info: Define ode function: 5.046 s (415532 allocations: 28.68 MiB)
[ Info: dt_save_to_sol: Inf, length(saveat): 1
[ Info: Saving diagnostics to HDF5 file on day 0 second 0
[ Info: init integrator: 69.210 s (61083128 allocations: 3.72 GiB)
[ Info: Init diagnostics: 2.500 μs (0 allocations: 0 bytes)
┌ Info: Running
│   job_id = "target_gpu_implicit_moist_baroclinic_wave_mo"
│   output_dir = "output/target_gpu_implicit_moist_baroclinic_wave_mo"
└   tspan = (0.0f0, 1.296f6)
[ Info: Saving diagnostics to HDF5 file on day 1 second 0
[ Info: Saving diagnostics to HDF5 file on day 2 second 0
[ Info: Saving diagnostics to HDF5 file on day 3 second 0
...

@simonbyrne
Copy link
Member

Dierckx.Spline1D isn't going to work on the GPU. We can either figure out some other mechanism to construct topography, or we initialize it all the CPU (#2241)

@akshaysridhar
Copy link
Member Author

akshaysridhar commented Oct 16, 2023 •
edited
Loading

Dierckx.Spline1D isn't going to work on the GPU. We can either figure out some other mechanism to construct topography, or we initialize it all the CPU (#2241)

Thanks @simonbyrne This specific issue appears to be in the radiation_cache here:

            # interpolate the ozone concentrations to our initial pressures
            pressure2ozone = Spline1D(
                input_center_pressure,
                input_center_volume_mixing_ratio_o3,
            )

(Clearsky radiation parameterisation) This is in the ClimaAtmos interface as opposed to the radiation source code. We expect to hit similar issues when enabling topography. cc @sriharshakandala

@sriharshakandala sriharshakandala mentioned this issue Oct 19, 2023
Open
@simonbyrne
Copy link
Member

See #2257

@lawann lawann mentioned this issue Oct 30, 2023
Closed
@akshaysridhar
Copy link
Member Author

akshaysridhar commented Oct 30, 2023 •
edited
Loading

@sriharshakandala With the CPU init updates, we can show that SF is compatible with the ClimaAtmos interfaces as in as/up-sf-fluxes > commit 7325e5b0 (running on single A100s). I think we can close this (since the other interpolation or GPU initialization issues are tracked elsewhere).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@akshaysridhar akshaysridhar

@sriharshakandala sriharshakandala

Labels
enhancement New feature or request
Projects
None yet
Milestone
O1.2.1 Clean up ClimaAtmos and run it...
Development

No branches or pull requests
3 participants
@simonbyrne @akshaysridhar @sriharshakandala