Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Change default output array type for JLD2OutputWriter to Array{Float64} #2890

Merged
merged 12 commits into from
Feb 6, 2023
Merged

Change default output array type for JLD2OutputWriter to Array{Float64} #2890

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
18d6e64
Change default output array type for `JLD2OutputWriter` to Array{Floa…
glwagner Feb 1, 2023
4fbe4c7
fix doctests
navidcy Feb 1, 2023
9232dd4
Update netcdf_output_writer.jl
glwagner Feb 2, 2023
d0b1e27
fix more doctests
navidcy Feb 2, 2023
5eea8f4
fix tests
navidcy Feb 3, 2023
221a413
Update test_netcdf_output_writer.jl
glwagner Feb 3, 2023
ee856ea
Update test_netcdf_output_writer.jl
glwagner Feb 3, 2023
5158df6
fix tests
navidcy Feb 3, 2023
2b9831d
Update Project.toml
glwagner Feb 5, 2023
ad74379
Update Project.toml
glwagner Feb 5, 2023
bd07897
Update Project.toml
glwagner Feb 5, 2023
cf7b561
Merge branch 'main' into glw-xk/jld2-output-float64
navidcy Feb 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions docs/src/model_setup/output_writers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ Other important keyword arguments are
portion of model data is saved to disk.

* `array_type` for specifying the type of the array that holds outputted field data. The default is
`Array{Float32}`, or arrays of single-precision floating point numbers.
`Array{Float64}`, or arrays of single-precision floating point numbers.

Once an `OutputWriter` is created, it can be used to write output by adding it the
ordered dictionary `simulation.output_writers`. prior to calling `run!(simulation)`.
Expand Down Expand Up @@ -70,7 +70,7 @@ NetCDFOutputWriter scheduled on TimeInterval(1 minute):
├── filepath: ./more_fields.nc
├── dimensions: zC(16), zF(17), xC(16), yF(16), xF(16), yC(16), time(0)
├── 2 outputs: (c, u)
└── array type: Array{Float32}
└── array type: Array{Float64}
```

```jldoctest netcdf1
Expand All @@ -83,7 +83,7 @@ NetCDFOutputWriter scheduled on TimeInterval(1 minute):
├── filepath: ./another_surface_xy_slice.nc
├── dimensions: zC(1), zF(1), xC(16), yF(16), xF(16), yC(16), time(0)
├── 2 outputs: (c, u)
└── array type: Array{Float32}
└── array type: Array{Float64}
```

```jldoctest netcdf1
Expand All @@ -98,7 +98,7 @@ NetCDFOutputWriter scheduled on TimeInterval(1 minute):
├── filepath: ./another_averaged_z_profile.nc
├── dimensions: zC(16), zF(17), xC(1), yF(1), xF(1), yC(1), time(0)
├── 2 outputs: (c, u) averaged on AveragedTimeInterval(window=20 seconds, stride=1, interval=1 minute)
└── array type: Array{Float32}
└── array type: Array{Float64}
```

`NetCDFOutputWriter` also accepts output functions that write scalars and arrays to disk,
Expand Down Expand Up @@ -140,7 +140,7 @@ NetCDFOutputWriter scheduled on IterationInterval(1):
├── filepath: ./some_things.nc
├── dimensions: zC(16), zF(17), xC(16), yF(16), xF(16), yC(16), time(0)
├── 3 outputs: (profile, slice, scalar)
└── array type: Array{Float32}
└── array type: Array{Float64}
```

See [`NetCDFOutputWriter`](@ref) for more information.
Expand Down Expand Up @@ -187,7 +187,7 @@ simulation.output_writers[:velocities] = JLD2OutputWriter(model, model.velocitie
JLD2OutputWriter scheduled on TimeInterval(20 minutes):
├── filepath: ./some_more_data.jld2
├── 3 outputs: (u, v, w)
├── array type: Array{Float32}
├── array type: Array{Float64}
├── including: [:grid, :coriolis, :buoyancy, :closure]
└── max filesize: Inf YiB
```
Expand All @@ -204,7 +204,7 @@ simulation.output_writers[:avg_c] = JLD2OutputWriter(model, (; c=c_avg),
JLD2OutputWriter scheduled on TimeInterval(20 minutes):
├── filepath: ./some_more_averaged_data.jld2
├── 1 outputs: c averaged on AveragedTimeInterval(window=5 minutes, stride=1, interval=20 minutes)
├── array type: Array{Float32}
├── array type: Array{Float64}
├── including: [:grid, :coriolis, :buoyancy, :closure]
└── max filesize: Inf YiB
```
Expand Down Expand Up @@ -261,7 +261,7 @@ simulation.output_writers[:velocities] = JLD2OutputWriter(model, model.velocitie
JLD2OutputWriter scheduled on TimeInterval(4 years):
├── filepath: ./even_more_averaged_velocity_data.jld2
├── 3 outputs: (u, v, w) averaged on AveragedTimeInterval(window=1 year, stride=2, interval=4 years)
├── array type: Array{Float32}
├── array type: Array{Float64}
├── including: [:grid, :coriolis, :buoyancy, :closure]
└── max filesize: Inf YiB
```
10 changes: 5 additions & 5 deletions src/OutputWriters/jld2_output_writer.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ ext(::Type{JLD2OutputWriter}) = ".jld2"
dir = ".",
indices = (:, :, :),
with_halos = false,
array_type = Array{Float32},
array_type = Array{Float64},
max_filesize = Inf,
overwrite_existing = false,
init = noinit,
Expand Down Expand Up @@ -76,7 +76,7 @@ Keyword arguments
you might need to set `with_halos = true`.

- `array_type`: The array type to which output arrays are converted to prior to saving.
Default: `Array{Float32}`.
Default: `Array{Float64}`.

## File management

Expand Down Expand Up @@ -136,7 +136,7 @@ simulation.output_writers[:velocities] = JLD2OutputWriter(model, model.velocitie
JLD2OutputWriter scheduled on TimeInterval(20 minutes):
├── filepath: ./some_data.jld2
├── 3 outputs: (u, v, w)
├── array type: Array{Float32}
├── array type: Array{Float64}
├── including: [:grid, :coriolis, :buoyancy, :closure]
└── max filesize: Inf YiB
```
Expand All @@ -153,7 +153,7 @@ simulation.output_writers[:avg_c] = JLD2OutputWriter(model, (; c=c_avg),
JLD2OutputWriter scheduled on TimeInterval(20 minutes):
├── filepath: ./some_averaged_data.jld2
├── 1 outputs: c averaged on AveragedTimeInterval(window=5 minutes, stride=1, interval=20 minutes)
├── array type: Array{Float32}
├── array type: Array{Float64}
├── including: [:grid, :coriolis, :buoyancy, :closure]
└── max filesize: Inf YiB
```
Expand All @@ -162,7 +162,7 @@ function JLD2OutputWriter(model, outputs; filename, schedule,
dir = ".",
indices = (:, :, :),
with_halos = false,
array_type = Array{Float32},
array_type = Array{Float64},
max_filesize = Inf,
overwrite_existing = false,
init = noinit,
Expand Down
14 changes: 7 additions & 7 deletions src/OutputWriters/netcdf_output_writer.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@ end
"""
NetCDFOutputWriter(model, outputs; filename, schedule
dir = ".",
array_type = Array{Float32},
array_type = Array{Float64},
indices = nothing,
with_halos = false,
global_attributes = Dict(),
Expand All @@ -162,7 +162,7 @@ Keyword arguments
- `dir`: Directory to save output to.

- `array_type`: The array type to which output arrays are converted to prior to saving.
Default: `Array{Float32}`.
Default: `Array{Float64}`.

- `indices`: Tuple of indices of the output variables to include. Default is `(:, :, :)`, which
includes the full fields.
Expand Down Expand Up @@ -210,7 +210,7 @@ NetCDFOutputWriter scheduled on TimeInterval(1 minute):
├── filepath: ./fields.nc
├── dimensions: zC(16), zF(17), xC(16), yF(16), xF(16), yC(16), time(0)
├── 2 outputs: (c, u)
└── array type: Array{Float32}
└── array type: Array{Float64}
```

```jldoctest netcdf1
Expand All @@ -223,7 +223,7 @@ NetCDFOutputWriter scheduled on TimeInterval(1 minute):
├── filepath: ./surface_xy_slice.nc
├── dimensions: zC(1), zF(1), xC(16), yF(16), xF(16), yC(16), time(0)
├── 2 outputs: (c, u)
└── array type: Array{Float32}
└── array type: Array{Float64}
```

```jldoctest netcdf1
Expand All @@ -238,7 +238,7 @@ NetCDFOutputWriter scheduled on TimeInterval(1 minute):
├── filepath: ./averaged_z_profile.nc
├── dimensions: zC(16), zF(17), xC(1), yF(1), xF(1), yC(1), time(0)
├── 2 outputs: (c, u) averaged on AveragedTimeInterval(window=20 seconds, stride=1, interval=1 minute)
└── array type: Array{Float32}
└── array type: Array{Float64}
```

`NetCDFOutputWriter` also accepts output functions that write scalars and arrays to disk,
Expand Down Expand Up @@ -282,12 +282,12 @@ NetCDFOutputWriter scheduled on IterationInterval(1):
├── filepath: ./things.nc
├── dimensions: zC(16), zF(17), xC(16), yF(16), xF(16), yC(16), time(0)
├── 3 outputs: (profile, slice, scalar)
└── array type: Array{Float32}
└── array type: Array{Float64}
```
"""
function NetCDFOutputWriter(model, outputs; filename, schedule,
dir = ".",
array_type = Array{Float32},
array_type = Array{Float64},
indices = (:, :, :),
with_halos = false,
global_attributes = Dict(),
Expand Down
2 changes: 1 addition & 1 deletion src/OutputWriters/windowed_time_average.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ simulation.output_writers[:velocities] = JLD2OutputWriter(model, model.velocitie
JLD2OutputWriter scheduled on TimeInterval(4 years):
├── filepath: ./averaged_velocity_data.jld2
├── 3 outputs: (u, v, w) averaged on AveragedTimeInterval(window=1 year, stride=2, interval=4 years)
├── array type: Array{Float32}
├── array type: Array{Float64}
├── including: [:grid, :coriolis, :buoyancy, :closure]
└── max filesize: Inf YiB
```
Expand Down
103 changes: 53 additions & 50 deletions test/test_netcdf_output_writer.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,12 +137,12 @@ function test_thermal_bubble_netcdf_output(arch)

@test eltype(ds3["time"]) == eltype(model.clock.time)

@test eltype(ds3["xC"]) == Float32
@test eltype(ds3["xF"]) == Float32
@test eltype(ds3["yC"]) == Float32
@test eltype(ds3["yF"]) == Float32
@test eltype(ds3["zC"]) == Float32
@test eltype(ds3["zF"]) == Float32
@test eltype(ds3["xC"]) == Float64
@test eltype(ds3["xF"]) == Float64
@test eltype(ds3["yC"]) == Float64
@test eltype(ds3["yF"]) == Float64
@test eltype(ds3["zC"]) == Float64
@test eltype(ds3["zF"]) == Float64

@test length(ds3["xC"]) == Nx
@test length(ds3["yC"]) == Ny
Expand All @@ -165,11 +165,11 @@ function test_thermal_bubble_netcdf_output(arch)
@test ds3["zC"][end] == grid.zᵃᵃᶜ[Nz]
@test ds3["zF"][end] == grid.zᵃᵃᶠ[Nz+1] # z is Bounded

@test eltype(ds3["u"]) == Float32
@test eltype(ds3["v"]) == Float32
@test eltype(ds3["w"]) == Float32
@test eltype(ds3["T"]) == Float32
@test eltype(ds3["S"]) == Float32
@test eltype(ds3["u"]) == Float64
@test eltype(ds3["v"]) == Float64
@test eltype(ds3["w"]) == Float64
@test eltype(ds3["T"]) == Float64
@test eltype(ds3["S"]) == Float64

u = ds3["u"][:, :, :, end]
v = ds3["v"][:, :, :, end]
Expand All @@ -196,12 +196,12 @@ function test_thermal_bubble_netcdf_output(arch)

@test eltype(ds2["time"]) == eltype(model.clock.time)

@test eltype(ds2["xC"]) == Float32
@test eltype(ds2["xF"]) == Float32
@test eltype(ds2["yC"]) == Float32
@test eltype(ds2["yF"]) == Float32
@test eltype(ds2["zC"]) == Float32
@test eltype(ds2["zF"]) == Float32
@test eltype(ds2["xC"]) == Float64
@test eltype(ds2["xF"]) == Float64
@test eltype(ds2["yC"]) == Float64
@test eltype(ds2["yF"]) == Float64
@test eltype(ds2["zC"]) == Float64
@test eltype(ds2["zF"]) == Float64
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ds2 is nc_sliced_writer which was constructed with explicit array_type = Array{Float32}

navidcy marked this conversation as resolved.
Show resolved Hide resolved

@test length(ds2["xC"]) == length(i_slice)
@test length(ds2["xF"]) == length(i_slice)
Expand Down Expand Up @@ -268,6 +268,7 @@ function test_thermal_bubble_netcdf_output_with_halos(arch)
view(model.tracers.T, i1:i2, j1:j2, k1:k2) .+= 0.01

nc_filepath = "test_dump_with_halos_$(typeof(arch)).nc"

nc_writer = NetCDFOutputWriter(model, merge(model.velocities, model.tracers),
filename = nc_filepath,
schedule = IterationInterval(10),
Expand All @@ -288,12 +289,13 @@ function test_thermal_bubble_netcdf_output_with_halos(arch)

@test eltype(ds["time"]) == eltype(model.clock.time)

@test eltype(ds["xC"]) == Float32
@test eltype(ds["xF"]) == Float32
@test eltype(ds["yC"]) == Float32
@test eltype(ds["yF"]) == Float32
@test eltype(ds["zC"]) == Float32
@test eltype(ds["zF"]) == Float32
# Using default array_type = Array{Float64}
@test eltype(ds["xC"]) == Float64
@test eltype(ds["xF"]) == Float64
@test eltype(ds["yC"]) == Float64
@test eltype(ds["yF"]) == Float64
@test eltype(ds["zC"]) == Float64
@test eltype(ds["zF"]) == Float64

Hx, Hy, Hz = grid.Hx, grid.Hy, grid.Hz
@test length(ds["xC"]) == Nx+2Hx
Expand All @@ -317,11 +319,11 @@ function test_thermal_bubble_netcdf_output_with_halos(arch)
@test ds["zC"][end] == grid.zᵃᵃᶜ[Nz+Hz]
@test ds["zF"][end] == grid.zᵃᵃᶠ[Nz+Hz+1] # z is Bounded

@test eltype(ds["u"]) == Float32
@test eltype(ds["v"]) == Float32
@test eltype(ds["w"]) == Float32
@test eltype(ds["T"]) == Float32
@test eltype(ds["S"]) == Float32
@test eltype(ds["u"]) == Float64
@test eltype(ds["v"]) == Float64
@test eltype(ds["w"]) == Float64
@test eltype(ds["T"]) == Float64
@test eltype(ds["S"]) == Float64

u = ds["u"][:, :, :, end]
v = ds["v"][:, :, :, end]
Expand Down Expand Up @@ -349,8 +351,7 @@ function test_netcdf_function_output(arch)
iters = 3

grid = RectilinearGrid(arch, size=(N, N, N), extent=(L, 2L, 3L))
model = NonhydrostaticModel(grid=grid,
buoyancy=SeawaterBuoyancy(), tracers=(:T, :S))
model = NonhydrostaticModel(; grid, buoyancy=SeawaterBuoyancy(), tracers=(:T, :S))

simulation = Simulation(model, Δt=Δt, stop_iteration=iters)
grid = model.grid
Expand All @@ -377,19 +378,22 @@ function test_netcdf_function_output(arch)
nc_filepath = "test_function_outputs_$(typeof(arch)).nc"

simulation.output_writers[:food] =
NetCDFOutputWriter(model, outputs; filename=nc_filepath,
schedule=TimeInterval(Δt), dimensions=dims, array_type=Array{Float64}, verbose=true,
global_attributes=global_attributes, output_attributes=output_attributes)
NetCDFOutputWriter(model, outputs; global_attributes, output_attributes,
filename = nc_filepath,
schedule = TimeInterval(Δt),
dimensions = dims,
array_type = Array{Float64},
verbose=true)

run!(simulation)

ds = Dataset(nc_filepath, "r")

@test haskey(ds.attrib, "date") && !isnothing(ds.attrib["date"])
@test haskey(ds.attrib, "Julia") && !isnothing(ds.attrib["Julia"])
@test haskey(ds.attrib, "Oceananigans") && !isnothing(ds.attrib["Oceananigans"])
@test haskey(ds.attrib, "schedule") && !isnothing(ds.attrib["schedule"])
@test haskey(ds.attrib, "interval") && !isnothing(ds.attrib["interval"])
@test haskey(ds.attrib, "date") && !isnothing(ds.attrib["date"])
@test haskey(ds.attrib, "Julia") && !isnothing(ds.attrib["Julia"])
@test haskey(ds.attrib, "Oceananigans") && !isnothing(ds.attrib["Oceananigans"])
@test haskey(ds.attrib, "schedule") && !isnothing(ds.attrib["schedule"])
@test haskey(ds.attrib, "interval") && !isnothing(ds.attrib["interval"])
@test haskey(ds.attrib, "output time interval") && !isnothing(ds.attrib["output time interval"])

@test eltype(ds["time"]) == eltype(model.clock.time)
Expand Down Expand Up @@ -467,9 +471,13 @@ function test_netcdf_function_output(arch)
simulation = Simulation(model, Δt=Δt, stop_iteration=iters)

simulation.output_writers[:food] =
NetCDFOutputWriter(model, outputs; filename=nc_filepath, overwrite_existing=false,
schedule=IterationInterval(1), array_type=Array{Float64}, dimensions=dims, verbose=true,
global_attributes=global_attributes, output_attributes=output_attributes)
NetCDFOutputWriter(model, outputs; global_attributes, output_attributes,
filename = nc_filepath,
overwrite_existing = false,
schedule = IterationInterval(1),
array_type = Array{Float64},
dimensions = dims,
verbose = true)

run!(simulation)

Expand Down Expand Up @@ -510,15 +518,10 @@ function test_netcdf_time_averaging(arch)
c1_forcing = Forcing(Fc1, field_dependencies=:c1)
c2_forcing = Forcing(Fc2, field_dependencies=:c2)

model = NonhydrostaticModel(
grid = grid,
timestepper = :RungeKutta3,
tracers = (:c1, :c2),
forcing = (c1=c1_forcing, c2=c2_forcing),
coriolis = nothing,
buoyancy = nothing,
closure = nothing
)
model = NonhydrostaticModel(; grid,
timestepper = :RungeKutta3,
tracers = (:c1, :c2),
forcing = (c1=c1_forcing, c2=c2_forcing))

set!(model, c1=1, c2=1)

Expand Down