The performance curve of parallel GEMM with many cores shows significant up-down

[nakagawa-fj]

When we measured the performance of SGEMM on ARM Neoverse V1 with 64 cores,
the performance values were not smooth and significant up-down, as shown in the following graph.
For example, the performance drops about 35% at m=n=k=1100, compared to the points before and after.
The significant fluctuations in performance values depend on the number of parallel threads and data size,
leading me to believe that the issue causes in the thread partitioning control. I am planning to modify it.

[martin-frbg]

Thanks. You are probably aware of yamazakimitsufumi's previous work in #4655 ?

[martin-frbg]

Also note that the fix in #5133 for a misguided change I had made in #4920 was not in 0.3.29 yet - this could cause extra slowdown and made the IBM folks really unhappy

[nakagawa-fj]

Thanks. You are probably aware of yamazakimitsufumi's previous work in #4655 ?

Yes, I know about yamazakimitsufumi's work #4655.

Also note that the fix in #5133 for a misguided change I had made in #4920 was not in 0.3.29 yet - this could cause extra slowdown and made the IBM folks really unhappy

Thanks for your notice. I have been aware of the fixes in #5133 and #4920 from this work. I have been surprised to hear #4920 had an impact at IBM. Once my modify is ready, it might also need to be reviewed by IBM as well.

[brada4]

involved data size is 12MB, likely some cache is exceeded.
Can you overlap sawttoth from full 2**n core number space?