-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Extend Matmul to support matrix multiplication with multiple heads #18570
Extend Matmul to support matrix multiplication with multiple heads #18570
Conversation
With the support of multiple head, the multiplication of two big matrixes is split into multiplication of several (head_number) small matrixes. e.g. if Mat A is [3, 24] and Mat B is [24, 4], when multiple A and B with head_number as 4, Mat A will be split as 4 matrix of [3, 6] and Mat B will be 4 matrix of [6, 4]. The result of final matrix will be 4 matrix of [3, 4], i.e. [3, 16]. test=develop
… matmul_with_multiple_head
To support matmul with multiple heads, it requires MKL stride feature of matrix multiplication. test=develop
To support matmul with multiple heads, it requires MKL stride feature of matrix multiplication. test=develop
…/Paddle into matmul_with_multiple_head
…/Paddle into matmul_with_multiple_head
…/Paddle into matmul_with_multiple_head
python/paddle/fluid/tests/unittests/test_matmul_op_with_head.py
Outdated
Show resolved
Hide resolved
python/paddle/fluid/tests/unittests/test_matmul_op_with_head.py
Outdated
Show resolved
Hide resolved
python/paddle/fluid/tests/unittests/test_matmul_op_with_head.py
Outdated
Show resolved
Hide resolved
python/paddle/fluid/tests/unittests/test_matmul_op_with_head.py
Outdated
Show resolved
Hide resolved
… matmul_with_multiple_head
5ce5a01
to
a8a3040
Compare
…/Paddle into matmul_with_multiple_head test=develop
numpy can handle it correctly without user's manual handling. test=develop
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
if transpose_X: | ||
shape_X = [M] | ||
else: | ||
shape_X = [K] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
reduce 4 lines to 1 line.
shape_X = [M] if transpose_X else [K]
You can update in your next PR
With the support of multiple head, the multiplication of two big matrixes is split into multiplication of several (head_number) small matrixes. e.g. if Mat A is [3, 24] and Mat B is [24, 4], when multiple A and B with head_number as 4, Mat A will be split as 4 matrix of [3, 6] and Mat B will be 4 matrix of [6, 4]. The result of final matrix will be 4 matrix of [3, 4], i.e. [3, 16].
Use this new extension, we can avoid head split/merge in Transformer.
first part of #16342