CodeGeeX inference support oneflow backend #65
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BBuf
commented
Feb 13, 2023
| from oneflow.nn.parameter import Parameter | ||
|
|
||
|
|
||
| def fast_gelu(x): |
| # Query, Key, and Value | ||
| # ===================== | ||
|
|
||
| if hasattr(torch._C, 'grouped_matmul_bias'): |
| origin_key_layer = key_layer | ||
| origin_value_layer = value_layer | ||
|
|
||
| if hasattr(torch._C, 'fused_multi_head_attention_inference'): |
| context_length=None, | ||
| ): | ||
|
|
||
| # hidden_states: [sq, b, h] |
There was a problem hiding this comment.
TopQueryAttention和SelfAttrention优化方式相同
| from codegeex.oneflow import CodeGeeXModel | ||
| from codegeex.tokenizer import CodeGeeXTokenizer | ||
| from codegeex.quantization import quantize | ||
| os.environ["ONEFLOW_KERNEL_ENABLE_FUSED_LINEAR"] = "1" |
There was a problem hiding this comment.
优化4: matmul支持和bias_add进行融合。
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
以下是oneflow和Faster Transformer以及PyTorch在Fp16情况下的性能结果：
由于mock torch在这个例子的应用中存在一些问题，无法一键以 oneflow 的方式运行 pytorch 版本的代码，我们以单独新增脚本的方式支持 oneflow 后端的 codegeex 推理。