Slowdown due to tracking_enabled() in 2.04.00 (found by Albany app)

[ibaned]

@ikalash and I have been tracking down a slowdown in the Albany CISM climate integration, and unfortunately it bisects down to the last Kokkos promotion.

Currently we're seeing a consistent 7% slowdown in the application test we're working with, with the only difference being which version of Kokkos its compiling against.

gahansen/Albany#159 contains the detailed analysis we've done to date, which hints at several possible causes of this performance regression within Kokkos.

I suspect this will take some effort amongst the Kokkos team, but it would be good if we could find a way to restore performance.
I know there were already concerns at the time of promotion (namely, the global rendezvous performance) but its not immediately obvious to me that this is the culprit.

[crtrott]

Try it again with current develop (soon to be master) where some rendezvous stuff was improved again.

[ibaned]

Currently the suspect is actually the tracking_enabled function that was changed to query a thread-local variable during the early stages of the OpenMP backend refactor by @dsunder . Before and after commit 8216185, there is a 14% increase in the runtime of an ice sheet modeling test (50 seconds to 57 seconds). This suggests to me that portions of the code in Trilinos and Albany are doing large amounts of shallow copying of views, possibly within kernels. I suggest we do two things:

• discuss with @dsunder regarding the thread-private implementation and search for a compromise
• identify and reduce unecessary shallow copies in user code (for example, functions which are called in kernel code can take a view by const reference instead of by value).

[dholladay00]

Regarding your 2nd point, this is a question I have had for a while. Is it in fact true that functions within a kernel can and should take views by const reference?

[ibaned]

Although ideally we would have a near-zero-overhead shallow copy mechanism, it seems that the current mechanism is slow enough to affect a certain application. I would personally write my own code using const references, until the overhead of the system improves (which it very well might by next release).

[stanmoore1]

If you do any shallow copies inside the kernel, also make sure you are using unmanaged views, otherwise the overhead of allocation tracking can really kill performance.

[mhoemmen]

kokkos-kernels and Tpetra tend to use unmanaged Views in kernels, and take unmanaged Views before taking subviews.

[ibaned]

Current indications are that this issue affects unmanaged views as well.

[ibaned]

Now going to attempt various mitigations. First a baseline with the latest Kokkos develop, Trilinos develop, and Albany master:

1/1 Test #128: FO_GIS_Gis20km_Tpetra ............   Passed   54.55 sec
1/1 Test #128: FO_GIS_Gis20km_Tpetra ............   Passed   55.75 sec
1/1 Test #128: FO_GIS_Gis20km_Tpetra ............   Passed   53.76 sec

[ibaned]

Making tracking_enabled() an inline function (bcb78974):

1/1 Test #128: FO_GIS_Gis20km_Tpetra ............   Passed   51.34 sec
1/1 Test #128: FO_GIS_Gis20km_Tpetra ............   Passed   49.85 sec
1/1 Test #128: FO_GIS_Gis20km_Tpetra ............   Passed   49.81 sec

[ibaned]

Making t_tracking_enabled a global variable instead of a thread-local variable:

1/1 Test #128: FO_GIS_Gis20km_Tpetra ............   Passed   51.39 sec
1/1 Test #128: FO_GIS_Gis20km_Tpetra ............   Passed   48.53 sec
1/1 Test #128: FO_GIS_Gis20km_Tpetra ............   Passed   48.54 sec

Seems like a tiny improvement, not sure if its worth it or not, at least for this application.

[ibaned]

Keeping tracking_enabled() non-inline (i.e. slow) but reducing the number of calls to it by fixing certain things in Kokkos::View (2b046cc):

1/1 Test #128: FO_GIS_Gis20km_Tpetra ............   Passed   52.37 sec
1/1 Test #128: FO_GIS_Gis20km_Tpetra ............   Passed   51.26 sec
1/1 Test #128: FO_GIS_Gis20km_Tpetra ............   Passed   52.04 sec

[ibaned]

Both optimizations put together:

1/1 Test #128: FO_GIS_Gis20km_Tpetra ............   Passed   51.73 sec
1/1 Test #128: FO_GIS_Gis20km_Tpetra ............   Passed   51.17 sec
1/1 Test #128: FO_GIS_Gis20km_Tpetra ............   Passed   48.46 sec

[ibaned]

On the kokkos-dev machine:

Albany 7660bba3effb1ab19302330dc32f8b5f681272b4
Trilinos ec787f84c3458bd77130b4b52f76d1fff76b8b0b
Kokkos 0a89b4aab5dba2ae302a2bf964fb203702bcd9c3

1/1 Test #45: FO_GIS_Gis20km_Tpetra ............   Passed   66.79 sec
1/1 Test #45: FO_GIS_Gis20km_Tpetra ............   Passed   64.92 sec
1/1 Test #45: FO_GIS_Gis20km_Tpetra ............   Passed   65.48 sec

Kokkos 7fd4dcb03cb9f5af7e3ec4d05cd4a636096a4129

1/1 Test #45: FO_GIS_Gis20km_Tpetra ............   Passed   50.71 sec
1/1 Test #45: FO_GIS_Gis20km_Tpetra ............   Passed   50.87 sec
1/1 Test #45: FO_GIS_Gis20km_Tpetra ............   Passed   50.50 sec

[stanmoore1]

@ibaned can you comment if this is back to normal?

[ibaned]

@stanmoore1 I think we've removed the main source of overhead introduced in Kokkos 2.04.00, and we may even have made it faster than before because we found other issues and repaired them. Merge commit 8c8a483 is the key one. At the end of the day though, the only real truth is application benchmarking with old and new Kokkos versions.