GPUify the TwoDTurb module

[navidcy]

This PR modifies the TwoDTurb module so that it can run smoothly on both CPU and GPU.

[navidcy]

It'll be good if we can remove the boilerplate in

GeophysicalFlows.jl/src/twodturb.jl

Lines 99 to 127 in 733833b

	struct Vars{Aphys, Atrans} <: TwoDTurbVars
	zeta :: Aphys
	u :: Aphys
	v :: Aphys
	zetah :: Atrans
	uh :: Atrans
	vh :: Atrans
	end
	struct ForcedVars{Aphys, Atrans} <: TwoDTurbVars
	zeta :: Aphys
	u :: Aphys
	v :: Aphys
	zetah :: Atrans
	uh :: Atrans
	vh :: Atrans
	Fh :: Atrans
	end
	struct StochasticForcedVars{Aphys, Atrans} <: TwoDTurbVars
	zeta :: Aphys
	u :: Aphys
	v :: Aphys
	zetah :: Atrans
	uh :: Atrans
	vh :: Atrans
	Fh :: Atrans
	prevsol :: Atrans
	end

I tried replacing it with

const physicalvars = [:zeta, :u, :v]
const transformvars = [ Symbol(var, :h) for var in physicalvars ]
const forcedvars = [:Fh]
const stochforcedvars = [:prevsol]

varspecs = cat(
  getfieldspecs(physicalvars, :(Aphys)),
  getfieldspecs(transformvars, :(Atrans)),
  dims=1)

forcedvarspecs = cat(varspecs, getfieldspecs(forcedvars, :(Atrans)), dims=1)

stochforcedvarspecs = cat(forcedvarspecs, getfieldspecs(stochforcedvars, :(Atrans)), dims=1)

struct Vars{Aphys, Atrans} <: TwoDTurbVars
  [ :( $(spec[1])::$(spec[2]) ) for spec in varspecs]
end

struct ForcedVars{Aphys, Atrans} <: TwoDTurbVars
  [ :( $(spec[1])::$(spec[2]) ) for spec in forcedvarspecs]
end

struct StochasticForcedVars{Aphys, Atrans} <: TwoDTurbVars
  [ :( $(spec[1])::$(spec[2]) ) for spec in stochforcedvarspecs]
end

but it didn't work... Did I do something wrong?

[codecov]

Codecov Report

Merging #25 into master will increase coverage by 0.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master      #25      +/-   ##
==========================================
+ Coverage   98.63%   98.64%   +0.01%     
==========================================
  Files           6        6              
  Lines         513      518       +5     
==========================================
+ Hits          506      511       +5     
  Misses          7        7

Impacted Files	Coverage Δ
src/twodturb.jl	100% <100%> (ø)	⬆️
src/utils.jl	100% <100%> (ø)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b9621db...9276b42. Read the comment docs.

[coveralls]

Pull Request Test Coverage Report for Build 263

• 38 of 38 (100.0%) changed or added relevant lines in 2 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+0.6%) to 98.456%

---

Totals
Change from base Build 251:	0.6%
Covered Lines:	510
Relevant Lines:	518

---

💛 - Coveralls

[navidcy]

Also, when a 2D grid is created on the GPU I get this message:

┌ Warning: Performing scalar operations on GPU arrays: This is very slow, consider disallowing these operations with `allowscalar(false)`
└ @ GPUArrays ~/.julia/packages/GPUArrays/J4c3Q/src/indexing.jl:16

and I couldn't figure out what causes it and how to make sure the code is efficient.

[glwagner]

@navidcy I think you want to use @eval somehow to reduce this boiler plate.

But there's another way --- we can define Vars generally, but give the optional fields special parameter types so that they can be set to nothing if need be. Not sure what the best way forward is.

[glwagner]

Also, when a 2D grid is created on the GPU I get this message:

Ya that's annoying. I've been getting that too in different code using GPUifyLoops. Your guess is as good as mine.

[navidcy]

@navidcy I think you want to use @eval somehow to reduce this boiler plate.

But there's another way --- we can define Vars generally, but give the optional fields special parameter types so that they can be set to nothing if need be. Not sure what the best way forward is.

can you try do that? I've lost my patience with @eval...

[navidcy]

Btw, I tried running examples/twodturb_mcwilliams1984.jl on GPU and it was painfully slow...
Perhaps the diags being CPU arrays induce constant communication back and forth from GPU to CPU?

I removed saveoutput and run only with the diagnostics, i.e., stepforward!(prob, diags, nsubs). On my laptop I got:

step: 0200, t: 2, ΔE: 0.9909, ΔZ: 0.4213, τ: 0.04 min
step: 0400, t: 4, ΔE: 0.9884, ΔZ: 0.2665, τ: 0.06 min
step: 0600, t: 6, ΔE: 0.9873, ΔZ: 0.1979, τ: 0.07 min

while on bruno using GPU:

step: 0200, t: 2, ΔE: 0.9909, ΔZ: 0.4213, τ: 1.11 min
step: 0400, t: 4, ΔE: 0.9884, ΔZ: 0.2665, τ: 2.17 min
step: 0600, t: 6, ΔE: 0.9873, ΔZ: 0.1979, τ: 3.21 min

When I run without computing diagnostics on bruno, i.e., stepforward!(prob, nsubs), I got:

step: 0200, t: 2, ΔE: 1.0000, ΔZ: 1.0000, τ: 0.01 min
step: 0400, t: 4, ΔE: 1.0000, ΔZ: 1.0000, τ: 0.03 min
step: 0600, t: 6, ΔE: 1.0000, ΔZ: 1.0000, τ: 0.04 min

I think we should revisit the diagnostics.jl in FourierFlows.jl. I feel that the fact that diagnostics are saved in CPU arrays creates a bottleneck.

[glwagner]

I feel that the fact that diagnostics are saved in CPU arrays creates a bottleneck.

I don't think this by itself is a problem (you can have a CPU array of CuArrays, for example). But it does sound like something is awry. Diagnostics also have have to well-written to be efficient.

[navidcy]

I feel that the fact that diagnostics are saved in CPU arrays creates a bottleneck.

I don't think this by itself is a problem (you can have a CPU array of CuArrays, for example). But it does sound like something is awry. Diagnostics also have have to well-written to be efficient.

OK, so regarding this PR: should we merge? Anything else needed?