Skip to content

[Unity][DLight] Update GEMV rules#15429

Merged
junrushao merged 1 commit into
apache:unityfrom
Hzfengsy:update_gemv
Jul 29, 2023
Merged

[Unity][DLight] Update GEMV rules#15429
junrushao merged 1 commit into
apache:unityfrom
Hzfengsy:update_gemv

Conversation

@Hzfengsy

Copy link
Copy Markdown
Member

This PR updates the GEMV rules:

  1. improve for large workloads, speedup Vicuna 13B from 43.0 tok/s to 60.8 tok/s
  2. Fix the issue of unexpected unroll for allreduce parts.

cc @spectrometerHBH @MasterJH5574

This PR updates the GEMV rules:
1. improve for large workloads, speedup Vicuna 13B from 43.0 tok/s to 60.8 tok/s
2. Fix the issue of unexpected unroll for allreduce parts.
@tvm-bot

tvm-bot commented Jul 28, 2023

Copy link
Copy Markdown
Collaborator

Thanks for contributing to TVM! Please refer to the contributing guidelines https://tvm.apache.org/docs/contribute/ for useful information and tips. Please request code reviews from Reviewers by @-ing them in a comment.

Generated by tvm-bot

@junrushao

Copy link
Copy Markdown
Member

CC @masahi

@masahi

masahi commented Jul 29, 2023

Copy link
Copy Markdown
Member

On RTX 4080 this PR improved over my last week's result as follows:

7B

|     mode        |   seqlen   |   genlen   |  p50: tok/s |
|      FT         |    128     |    128     |     121.7   |
|  dlight (prev)  |    128     |    128     |     141.9   |
|  dlight (new)   |    128     |    128     |     146.7   |

13B

|     mode        |   seqlen   |   genlen   |  p50: tok/s |
|      FT         |    128     |    128     |     74.6    |
|  dlight (prev)  |    128     |    128     |     67.8    |
|  dlight (new)   |    128     |    128     |     80.6    |

@junrushao
junrushao merged commit 3f1469e into apache:unity Jul 29, 2023
@Hzfengsy
Hzfengsy deleted the update_gemv branch July 30, 2023 05:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants