-
Notifications
You must be signed in to change notification settings - Fork 50
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to increase the performance when there is constraint on MM_TRANSPOSE_A=OFF #31
Comments
The ideal solution is increasing your overall tile size, but you could also try to shift some of your tile sizes from N to M. This is not ideal for communication volume, but it can keep the same number of inner tiles, while decreasing the number of columns that must be read from A per tile. For example, you could try |
Thanks. The problem for me is there are three GEMM computation tasks and the convention number of GEMM accelerator size is 512, and these three GEMM computation has different transpose format. My understanding of the transposeA implementation is to use parallel aSplit buffers to read columns of A and then in ConvertWidthA it will sequentially feed the element of outer tile A. The data feed time of A outer tile is DMM_MEMORY_TILE_SIZE_N, and the computation time of PE arrays is inner tile multiplication time(kInnerTileN*kInnerTileM). So the data feed time should be no more than computation time, is that understanding right? If so, will it work if I modify the transpose function to increase the parallelism for data feeding in tile A? If it cut half of the time, then I can double the PE number or B parallelism factor? |
What sizes does your matrix have? Even if you chain multiple multiplications of different sizes, it should still be possible for them to have different tile sizes. The problem is of course if your matrix is only 512x512, then you cannot make the tile bigger without losing computational efficiency.
Yes, correct
Yes, this could work. It's not super simple, since you would need to vectorize the buffering of A between and inside all the PEs. But it's definitely doable (actually the initial version had a version of this feature, but I removed it for simplicity since I never used it). You could give it a try :-) |
...since you would need to vectorize the buffering of A between and inside all the PEs.... Would you mind explaining a bit more about why we also need to modify PEs for vectorizing buffering of A? I'm a bit lost. Or is there any simpler way, for example, I only modify the data feeder of A to double the outer tile of A feeding speed? In that case, there is no extra modification for PEs, and constraint can be kInnerTileN*kInnerTileM >= MEMORY_TILE_SIZE_N/2. |
If you want to speed up buffering
I'm not sure how you could do this without making the width of the data path transporting elements of |
I got it. That sounds to set the Granularity of PE to more than 1 and performance is doubled. And do you mind sharing that code?
Since I want to keep that constraint working, if I need to double the PE number or double the parallelism of B, then kInnerTileN*kInnerTileM will be half. To satisfy the constraint, I also need to reduce the latency for loading and transposing A outer tile to half. The method I thought increased the performance by adding more PEs and parallelism of B, but it use more resources than the method you proposed. |
The code does not exist anymore, it was the idea that this should be supported when I wrote the transpose code, but in the end I decided that I didn't need it :-) |
Thanks. And what do you think of the other method? does it work?
|
It doesn't matter if your parallelism is in A or B, it only matters how many columns of |
Hi,
I'm using a GEMM with tile size 512 for xtot and ytot. But if I use C=AB mode(MM_TRANSPOSE_A=OFF), since A is column based and need to be transposed on chip, there is constraint that "the number of inner tiles must be greater than or equal to the outer tile size in N", this limited the number of PEs and the parallelism of B input, which also limited the GEMM performance. I also thought of using C=A.T*B instead but this doesn't work for me with changing the computation mathematically.
Is there any way to break the constraint, e.g. increase the A bandwidth, etc.? And how is the constraint formula deduced, what's its meaning? Thank you!
The text was updated successfully, but these errors were encountered: