Tweak Distributed API #3314

glwagner · 2023-10-05T21:25:01Z

This PR changes up the API for Distributed. The changes are:

Removal of the kwarg enable_overlapping_computation which doesn't belong in architectures (which are model agnostic). (We can add that feature to specific models if desired.)
Automatic initialization if MPI.Initialized() is false
Implementation of default ranks (by default we distribute in x)

Perhaps more to come.

I'm also going to clean up up the examples, since they are the only source of information about running distributed simulations currently.

glwagner · 2023-10-05T21:27:17Z

Are there any tests for distributed hydrostatic models?

simone-silvestri · 2023-10-05T21:39:17Z

yeah in test/test_distributed_models.jl

glwagner · 2023-10-05T21:52:16Z

@simone-silvestri DistributedComputations is imported before models here:

Oceananigans.jl/src/Oceananigans.jl

Lines 221 to 224 in 82503b8

    
           include("DistributedComputations/DistributedComputations.jl") 
        
           include("TimeSteppers/TimeSteppers.jl") 
        
           include("Models/Models.jl")

What was "enable overlapping communication" intended to support? Is there something that depends on whether isnothing(mpi_requests)? Perhaps more direct specification of this feature would help.

glwagner · 2023-10-05T21:53:39Z

yeah in test/test_distributed_models.jl

There's no tests for HydrostaticFreeSurfaceModel there currently, just NonhydrostaticModel.

Is there another file with tests for HydrostaticFreeSurfaceModel?

glwagner · 2023-10-05T22:13:03Z

How important is it to call MPI.Finalize? What happens if we don't do it? If it's critical, how do we ensure this happens? (This seems like a bad design...)

glwagner · 2023-10-05T22:17:10Z

I think users should put the global grid size into the grid constructor, not the local size. If the partitioning is irregular, this should go into the architecture. The reason we need this is to support a "one line" change from local to distributed computations. How do we implement this?

Oceananigans.jl/src/DistributedComputations/distributed_grids.jl

Line 39 in d7b9728

global_size = map(sum, concatenate_local_sizes(size, arch))

glwagner · 2023-10-06T11:38:44Z

Ok, I found in the docs for MPI.jl that

Calling MPI.Finalize() at the end of the program is optional, as it will be called automatically when Julia exits.

This is good news. So we can call MPI.Init() when we build the architecture, and avoid MPI.Finalize(). This brings us one step closer to the "one line to distributed simulations" goal.

https://github.com/JuliaParallel/MPI.jl

glwagner · 2023-10-06T11:49:53Z

Hopefully we can close #3318 here too, let's discuss.

validation/distributed_simulations/distributed_geostrophic_adjustment.jl

simone-silvestri · 2023-10-06T17:03:02Z

src/DistributedComputations/distributed_architectures.jl

+Partition(Rx::Number, Ry::Number=1, Rz::Number=1) = Partition{Rx, Ry, Rz}()
+Partition(lengths::Array{Int, 1}) = Partition(reshape(lengths, length(lengths), 1))
+Base.size(::Partition{<:Any, Rx, Ry, Rz}) where {Rx, Ry, Rz} = (Rx, Ry, Rz)
+


Suggested change

Partition(; Rx = 1, Ry = 1, Rz = 1) = Partition(Rx, Ry, Rz)

src/DistributedComputations/distributed_architectures.jl

simone-silvestri · 2023-10-06T17:07:33Z

Oceananigans.jl/src/DistributedComputations/distributed_grids.jl

Line 326 in 5dd96cf

    
           return RectilinearGrid(arch, eltype(global_grid); size=local_size, x=x, y=y, z=z, halo=halo, topology=topo)

here we probably need something like

function scatter_local_grids(arch::Distributed, global_grid::RectilinearGrid, local_size)
    x, y, z, topo, halo = scatter_grid_properties(global_grid)
    global_size = map(sum, concatenate_local_sizes(local_size, arch))
    return RectilinearGrid(arch, eltype(global_grid); size=global_size, x=x, y=y, z=z, halo=halo, topology=topo)
end

same for the LatitudeLongitudeGrid

…ustment.jl Co-authored-by: Simone Silvestri <silvestri.simone0@gmail.com>

glwagner · 2023-10-07T22:03:19Z

I don’t think we should fix the untested features in this PR. We need to merge this ASAP so we can open a new PR with tests.

glwagner · 2023-10-07T22:04:22Z

@simone-silvestri can you fix the failing test? As soon as tests pass please merge. We need a another PR that implements tests following this one.

src/DistributedComputations/distributed_architectures.jl

navidcy · 2023-10-10T15:35:14Z

src/DistributedComputations/distributed_fields.jl

+    return nothing
 end


Suggested change

return nothing

end

return nothing

end

navidcy · 2023-10-10T15:35:32Z

src/DistributedComputations/distributed_grids.jl

@@ -84,7 +99,7 @@ function LatitudeLongitudeGrid(arch::Distributed,
                               radius = R_Earth,
                               halo = (1, 1, 1))

-    global_size = map(sum, concatenate_local_sizes(size, arch))
+    # global_size = map(sum, concatenate_local_sizes(size, arch))


why comment out? delete?

src/ImmersedBoundaries/distributed_immersed_boundaries.jl

navidcy · 2023-10-10T15:36:30Z

src/Oceananigans.jl

@@ -218,7 +222,7 @@ include("Biogeochemistry.jl")

 # TODO: move above
 include("ImmersedBoundaries/ImmersedBoundaries.jl")
-include("DistributedComputations/DistributedComputations.jl")
+# include("DistributedComputations/DistributedComputations.jl")


why comment out?

src/DistributedComputations/distributed_fields.jl

navidcy · 2023-10-10T15:39:39Z

test/test_distributed_models.jl

@@ -483,8 +485,9 @@ end
    @testset "Time stepping ShallowWaterModel" begin
        for child_arch in archs
            topo = (Periodic, Periodic, Flat)
-            arch = Distributed(child_arch; ranks=(1, 4, 1), topology = topo, devices = (0, 0, 0, 0))
-            grid = RectilinearGrid(arch, topology=topo, size=(8, 2), extent=(1, 2), halo=(3, 3))
+            #arch = Distributed(child_arch; ranks=(1, 4, 1), topology = topo, devices = (0, 0, 0, 0))


delete the comment?

navidcy · 2023-10-10T15:42:57Z

validation/distributed_simulations/distributed_shallow_water_turbulence.jl


-MPI.Barrier(MPI.COMM_WORLD)
+# ?


Co-authored-by: Navid C. Constantinou <navidcy@users.noreply.github.com>

…ceananigans.jl into glw/refactor-distributed

glwagner added 2 commits October 5, 2023 17:23

Remove enable_overlapping from distributed kwarg

9119d70

Import Distributed after Solvers

8023bb4

Dont switch import yet

c7874c4

Implement default rank distribution

d7b9728

glwagner added 2 commits October 6, 2023 05:28

Update validation scripts

25a809c

Update for new API

f509525

glwagner added 6 commits October 6, 2023 07:52

More cleanup of distributed examples

7b3416e

Restore 2D partitioning to hydrostatic turbulence example

2a0c32e

Implement Partition for Distributed architectures, messy state

7c77f1c

Fix issue with distributed tests for new API

94b2dc0

Add a comment about correctness checking

52400cb

Fix further tests

79c29fa

simone-silvestri reviewed Oct 6, 2023

View reviewed changes

validation/distributed_simulations/distributed_geostrophic_adjustment.jl Outdated Show resolved Hide resolved

simone-silvestri reviewed Oct 6, 2023

View reviewed changes

validation/distributed_simulations/distributed_geostrophic_adjustment.jl Outdated Show resolved Hide resolved

simone-silvestri reviewed Oct 6, 2023

View reviewed changes

src/DistributedComputations/distributed_architectures.jl Show resolved Hide resolved

glwagner and others added 2 commits October 6, 2023 13:13

Update validation/distributed_simulations/distributed_geostrophic_adj…

4032773

…ustment.jl Co-authored-by: Simone Silvestri <silvestri.simone0@gmail.com>

Update validation/distributed_simulations/distributed_geostrophic_adj…

02a45fb

…ustment.jl Co-authored-by: Simone Silvestri <silvestri.simone0@gmail.com>

glwagner mentioned this pull request Oct 6, 2023

Add HydrostaticFreeSurfaceModel quickstart to README #3308

Draft

glwagner and others added 7 commits October 7, 2023 16:04

Merge branch 'main' into glw/refactor-distributed

0772719

tests fixxed

3beefe9

bugfix

8cb2f01

try now

dd3b161

fixed tests

10f9071

bugfix

b6b2125

fixed tests

fab72cb

navidcy reviewed Oct 10, 2023

View reviewed changes

src/DistributedComputations/distributed_architectures.jl Outdated Show resolved Hide resolved

navidcy reviewed Oct 10, 2023

View reviewed changes

src/ImmersedBoundaries/distributed_immersed_boundaries.jl Outdated Show resolved Hide resolved

navidcy reviewed Oct 10, 2023

View reviewed changes

src/DistributedComputations/distributed_fields.jl Outdated Show resolved Hide resolved

navidcy reviewed Oct 10, 2023

View reviewed changes

validation/distributed_simulations/distributed_shallow_water_turbulence.jl

MPI.Barrier(MPI.COMM_WORLD)

# ?

Copy link

Collaborator

navidcy Oct 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

# ? ?

glwagner and others added 4 commits October 10, 2023 11:59

Change synched to syncronized

ff60167

Update src/DistributedComputations/distributed_fields.jl

d27624e

Co-authored-by: Navid C. Constantinou <navidcy@users.noreply.github.com>

Remove comment

37e037f

Merge branch 'glw/refactor-distributed' of https://github.com/CliMA/O…

6f5eaf4

…ceananigans.jl into glw/refactor-distributed

simone-silvestri approved these changes Oct 10, 2023

View reviewed changes

Apply suggestions from code review

8958a2c

simone-silvestri merged commit c823cb9 into main Oct 10, 2023
47 checks passed

simone-silvestri deleted the glw/refactor-distributed branch October 10, 2023 18:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tweak Distributed API #3314

Tweak Distributed API #3314

glwagner commented Oct 5, 2023 •

edited

Loading

glwagner commented Oct 5, 2023

simone-silvestri commented Oct 5, 2023

glwagner commented Oct 5, 2023

glwagner commented Oct 5, 2023

glwagner commented Oct 5, 2023

glwagner commented Oct 5, 2023

glwagner commented Oct 6, 2023

glwagner commented Oct 6, 2023

simone-silvestri Oct 6, 2023

simone-silvestri commented Oct 6, 2023

glwagner commented Oct 7, 2023

glwagner commented Oct 7, 2023

navidcy Oct 10, 2023

navidcy Oct 10, 2023

navidcy Oct 10, 2023

navidcy Oct 10, 2023

navidcy Oct 10, 2023

Tweak Distributed API #3314

Tweak Distributed API #3314

Conversation

glwagner commented Oct 5, 2023 • edited Loading

glwagner commented Oct 5, 2023

simone-silvestri commented Oct 5, 2023

glwagner commented Oct 5, 2023

glwagner commented Oct 5, 2023

glwagner commented Oct 5, 2023

glwagner commented Oct 5, 2023

glwagner commented Oct 6, 2023

glwagner commented Oct 6, 2023

simone-silvestri Oct 6, 2023

Choose a reason for hiding this comment

simone-silvestri commented Oct 6, 2023

glwagner commented Oct 7, 2023

glwagner commented Oct 7, 2023

navidcy Oct 10, 2023

Choose a reason for hiding this comment

navidcy Oct 10, 2023

Choose a reason for hiding this comment

navidcy Oct 10, 2023

Choose a reason for hiding this comment

navidcy Oct 10, 2023

Choose a reason for hiding this comment

navidcy Oct 10, 2023

Choose a reason for hiding this comment

glwagner commented Oct 5, 2023 •

edited

Loading