Try to make sense of profiles

[ilectra]

First CPU , then GPU.
DONE:

• Identify what is where in the code (they're all LambdaApply!)
• Identify major bottlenecks

[ilectra]

@qiUip add summary of current understanding of profiles. @asifsamiarain add screenshots

[asifsamiarain]

Note : Profiles locations can be found in #4 (comment)

[qiUip]

The time consuming parts of the code are stemming out of the integrator loop (pseudo-Molecular Dynamics). These contain two main features, each doubled by the multi-step integrator, for the Gauge actions and the Fermions. The Fermions actions are represented by the two large flat-top sections to the left of the above screenshots while the gauge actions by the two 'chunks' each containing 2 hills that fragment into multiple small parts and a flat part next to them. In general, the Gauge interactions are larger than the Fermions when they are expected to be significantly lower. This points us towards investigating how to improve performance starting from where the integrator calculates the derivatives for the Gauges.

As the functions where the operators used for calculating the derivatives are mostly called from inside Lambda functions, the profile does not directly show the functions where the work itself takes place, but instead the functions where the Lambda's are called, making tracing the locations of the work done in the code challenging.

Testing both 8x8x8x8 grids and 16x16x16x16 showed the main balance remains the same, noting that enabling threading also keeps the balance generally the same, indicating that performance improvements need to be found higher up than where threading in present (which seems to be on low-level operators). This means that, for now, we can focus on single-threaded performance and work on improving the structure of the algorithms and data-structures.

The Gauge actions can be split further into the following: 1. deriv that comes from update_P which then calls WilsonLoops<Gimpl>::Staple in each of the two steps of the integrator. Staple then loops over the directions and calls multiple "shifts" (See Grid/qcd/utils/WilsonLoops.h:306); 2. update_field which is called from update_U and can be found in GaugeImplTypes.h:135 where ProjectOnGeneralGroup() that should also be investigated.

[ilectra]

Hint on looking at the horrible strings we get in VTune: When hovering over a box in the flame graph, there is often a string in the middle with something like GroupName2SpEEE with some other stuff after it. The first 'readable' work after that is actually a function name inside the file name that appears at the bottom of the box.