views from Kokkos_ScratchSpace should use different alignment

[alanw0]

This link:
https://github.com/kokkos/kokkos/blob/master/core/src/Kokkos_ScratchSpace.hpp#L65
shows that views into Kokkos_ScratchSpace have 8-byte alignment.
We would like to request that this be changed to use KOKKOS_MEMORY_ALIGNMENT for the alignment value.

[crtrott]

Sure, Dan is doing it ;-)

[ibaned]

I tried to do a simple replacement of '8' with 'KOKKOS_MEMORY_ALIGNMENT', but that caused a lot more scratch memory to be allocated and crashed our tests. A more involved approach is needed, not sure when I'll have time to debug it.

[alanw0]

Just curious - did the memory increase by more than ((KOKKOS_MEMORY_ALIGNMENT) - (8)) per scratch-view ? Maybe that difference is multiplied by num-threads or something?

I guess when the user provides bytes-per-thread to the TeamPolicy, they don't account for alignment padding, nor do they say how many views they will be obtaining into that memory...

If this can't be quickly solved, perhaps nalu needs to move on to some other solution.

[crtrott]

One issue is that KOKKOS_MEMORY_ALIGNMENT is probably larger than it needs to be. On GPUs I think it might 128 bytes to align with the cache line. So if you create a lot fo small things this is gonna be costly. What we should do is align by max(8,sizeof(value_type)).

[crtrott]

Also we don't probably need KOKKOS_MEMORY_ALIGNMENT. I bet that the main issue is that SIMD types don't get aligned correctly and cause issues? Is that right?

[alanw0]

Yes the issue is with simd types.

[crtrott]

Ok so I have a fix ready for pull request, which will align each scratch view to sizeof(value_type). That should hopefully make this work.

[crtrott]

Marking this as a bug since it actually needs to be aligned for types which require alignment.