Diagnostics slow down modules running on GPU by a lot

[navidcy]

there must be something the gives GPU a serious bottleneck in diagnostics.jl in FourierFlows.jl

Where I run this test script I get:

we time-step without diagnostics
step: 0400, t: 1, walltime: 0.87 sec
step: 0800, t: 2, walltime: 1.66 sec
step: 1200, t: 2, walltime: 2.45 sec
step: 1600, t: 3, walltime: 3.24 sec
step: 2000, t: 4, walltime: 4.03 sec

re-initialize and run again with diagnostics
step: 0400, t: 1, walltime: 135.30 sec
step: 0800, t: 2, walltime: 305.39 sec
step: 1200, t: 2, walltime: 459.79 sec
step: 1600, t: 3, walltime: 599.51 sec
step: 2000, t: 4, walltime: 770.04 sec

If, on the other hand, I run on CPU the slowdown I get is just 1-2%.

[glwagner]

I bet its the functions energy and enstrophy that are the issue, rather than diagnostics.jl in FourierFlows. We can test this by using a trivial diagnostic, say, trivial(prob) = 1, and see if we observe the same slow down on the GPU. Since trivial(prob) = 1 does not involve GPU computation, I'd guess that its probably not diagnostics.jl.

[navidcy]

I did that:

we time-step without diagnostics
step: 0400, t: 1, walltime: 5.29 sec
step: 0800, t: 2, walltime: 6.27 sec
step: 1200, t: 2, walltime: 9.57 sec
step: 1600, t: 3, walltime: 10.48 sec
step: 2000, t: 4, walltime: 11.32 sec

re-initialize and run again with trivial diagnostic
step: 0400, t: 1, walltime: 0.92 sec
step: 0800, t: 2, walltime: 1.73 sec
step: 1200, t: 2, walltime: 2.54 sec
step: 1600, t: 3, walltime: 3.37 sec
step: 2000, t: 4, walltime: 4.18 sec

re-initialize and run again with energy diagnostic
step: 0400, t: 1, walltime: 1.16 sec
step: 0800, t: 2, walltime: 2.27 sec
step: 1200, t: 2, walltime: 3.37 sec
step: 1600, t: 3, walltime: 4.47 sec
step: 2000, t: 4, walltime: 5.58 sec

and last re-initialize and run again with enstrophy diagnostic
step: 0400, t: 1, walltime: 129.00 sec
step: 0800, t: 2, walltime: 258.86 sec
...

energy is OK but enstrophy leads to slowdown.

I guess this points towards parsevalsum2() vs parsevalsum()? Do you have any educated guess? Is it @views you reckon?

[navidcy]

I'm guessing that this is related to issue FourierFlows/FourierFlows.jl#128

[glwagner]

Oh. I thought the issue was non-contiguous memory, but since parsevalsum is fine, I guess that means its something else.

An easy solution is just to use parsevalsum in enstrophy.

parsevalsum2 is more of a convenience function than necessary. You could try removing @views (I don't see why theres an @views there but not on parsevalsum). The problem may be the abs2 element-wise command, however.

[navidcy]

Resolved by FourierFlows/FourierFlows.jl#165