Should reductions exclude peripheral_node?

[glwagner]

Right now reductions only exclude immersed peripheral nodes:

Oceananigans.jl/src/ImmersedBoundaries/immersed_reductions.jl

Line 27 in 72e2197

return get_condition(condition.func, i, j, k, ibg, args...) & !(immersed_peripheral_node(i, j, k, ibg, LX(), LY(), LZ()))

It recently caused me a lot of pain and confusion and time that "ordinary" peripheral nodes are included in the reduction, but immersed peripheral nodes are not.

What is the logic for treating immersed boundaries differently from ordinary boundaries? I think we should either exclude only inactive nodes or peripheral nodes, but this behavior should be consistent between immersed and not immersed grids.

This change would mean we don't need special reductions (at least for fields that are not reduced) on immersed vs not-immersed grids.

@simone-silvestri may have the answer.

[glwagner]

Simple example why this doesn't make sense: immersed_peripheral_node is

Oceananigans.jl/src/ImmersedBoundaries/ImmersedBoundaries.jl

Lines 217 to 218 in 72e2197

	@inline immersed_peripheral_node(i, j, k, ibg::IBG, LX, LY, LZ) = peripheral_node(i, j, k, ibg, LX, LY, LZ) &
	!peripheral_node(i, j, k, ibg.underlying_grid, LX, LY, LZ)

So, for example, if the entire bottom of a grid is immersed, then it will be included in the reduction --- because those nodes are on the periphery of the underlying grid, so

!peripheral_node(i, j, k, ibg.underlying_grid, LX, LY, LZ)

is false. Including the bottom row of cells in the reduction makes no sense because these nodes are far beneath the immersed boundary.

As a quick fix we could add another condition that also ignores inactive_node.

But I'd also like to understand why we don't simply ignore all peripheral_nodes...

[navidcy]

cc @simone-silvestri