UVM dependency removal

[ldh4]

[WIP] Modifications to make Nalu-wind buildable without Cuda UVM

Pre-requisite work for converting Nalu-wind to be compatible with HIP.
This is to make Nalu wind to build without Cuda UVM

@tasmith4 @alanw0 @jhux2 @ddement

[tasmith4]

After further investigation, @ldh4 and I learned from @alanw0 that tests without NGP in the name are not expected to pass in any build. After running the unit tests with that filter, they all pass on ascicgpu both with and without UVM.

[alanw0]

After further investigation, @ldh4 and I learned from @alanw0 that tests without NGP in the name are not expected to pass in any build. After running the unit tests with that filter, they all pass on ascicgpu both with and without UVM.

@tasmith4 @ldh4 More precisely, tests without NGP in the name are not expected to pass in a Cuda build. All unit tests are expected to pass in a CPU build. However, I am surprised to hear that all of the NGP unit-tests pass on Cuda without UVM. For example, the *.NGP* filter would include at least 3 unit tests that have tpetra usage (MixtureFractionKernelHex8Mesh.NGP_adv_diff_edge_tpetra*). I guess it's possible those tests don't use the graph assembly code, but I would be surprised. (And the one thing I think we're pretty certain about is that the graph assembly code is not using the correct types for at least some of its Views, and fixing those would then require one or more deep_copy calls.)

In any case, I'll work with you and Dong Hun tomorrow to help establish where we're at with all this.

[tasmith4]

You're right @alanw0, I should have been more precise, thanks for clarifying. I was also very surprised all the NGP tests pass without UVM (and am willing to be skeptical of the result, although I double-checked my build), we'll have to discuss this more today as you say.

[alanw0]

@tasmith4 In particular we're passing those views (that this commit changes to host views) into the setAllIndices method on the tpetra crsgraph. I guess the explanation could be that we're getting those host views from a dual view, and inside setAllIndices tpetra is doing the sync for us and then using the device views. If that's what is happening then great!

[tasmith4]

I don't think that's what's happening, so I'm now even more confused by why these tests pass.

[alanw0]

I guess I didn't look closely... Now that I look at this commit again, it doesn't change the types of the views we're passing into setAllIndices. It is changing the types of other stuff. So we'll just have to work through this today and see what is going on. We'll probably want to put in a (temporary) check in the code, in TpetraLinearSystem.C, for whether uvm is actually on or not.

[PaulMullowney]

Why is this necessary? Are we leaking memory?

[ldh4]

On the TpetraLinearSystem class, we had to change it so that it is no longer holding onto the host/device coeff appliers as its member variables. So, a new coeff applier is constructed on each request. Because of this, we had to introduce a way to properly deallocate coeff appliers when no longer needed. This is one of the changes that we will likely to revisit to find a better way to handle this.
Just as a note, free_coeff_applier is an empty function in LinearSystem.h, so this line will not affect HypreLinearSystem.

[PaulMullowney]

We're doing this again. We must have been leaking memory?

[ldh4]

Same reason as the above.

[ldh4]

A couple notes on this commit:

• This builds and runs with and without unified memory support (tested only with cuda backend for now).
• Building without unified memory currently shows deficiencies in both memory usage and runtime performance. We are still investigating the source of this and will have follow-on tasks to resolve this.
• Building with unified memory does not have this problem and maintains the pervious performance. When tested with nrel5mw_refined_rcb problem (~700M nodes) on 90 nodes of summit, the build with this patch ran to its completion where as the build from Nalu-wind master timed out.

[ldh4]

I am actually running nrel5mw_refined_rcb problem again with a smaller termination count, just to see how comparable the runtimes between the two builds are.

[ldh4]

Checking numbers from nrel5mw_refined_rcb runs again with its termination count reduced by half showed that there isn't any significant difference in runtime and memory usage between this patched version (uvm turned on) and the master, as expected. Runtime had ~5% speedup.