Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[cpu]llama avx model inference supports #8634

Merged
merged 1 commit into from
Jun 27, 2024

Conversation

bukejiyu
Copy link
Contributor

@bukejiyu bukejiyu commented Jun 20, 2024

PR types

PR changes

Description

paddle inference_mode 集成xft cpu kernel
机器8463B
输入/输出 128/15 bs=1
静态图llama 测速 next_tokens: 100+ms
48线程 动态图llama 测速 next_tokens: 70+ms

Copy link

paddle-bot bot commented Jun 20, 2024

Thanks for your contribution!

@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


395822456@qq.com seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

Copy link

codecov bot commented Jun 20, 2024

Codecov Report

Attention: Patch coverage is 0% with 282 lines in your changes missing coverage. Please review.

Project coverage is 55.63%. Comparing base (65e721e) to head (b791375).
Report is 11 commits behind head on develop.

Files Patch % Lines
...dlenlp/experimental/transformers/llama/modeling.py 0.00% 133 Missing ⚠️
...enlp/experimental/transformers/generation_utils.py 0.00% 101 Missing ⚠️
...erimental/transformers/fused_transformer_layers.py 0.00% 48 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #8634      +/-   ##
===========================================
- Coverage    55.80%   55.63%   -0.18%     
===========================================
  Files          620      620              
  Lines        96642    96940     +298     
===========================================
  Hits         53928    53928              
- Misses       42714    43012     +298     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@bukejiyu bukejiyu force-pushed the tmp_cpu branch 2 times, most recently from 426aa9d to c3c3d49 Compare June 25, 2024 04:20
@bukejiyu bukejiyu changed the title tmp support cpu [cpu]llama avx model inference supports Jun 25, 2024
csrc/cpu/src/setup.py Outdated Show resolved Hide resolved
csrc/cpu/README.md Outdated Show resolved Hide resolved
csrc/cpu/setup.sh Outdated Show resolved Hide resolved
csrc/cpu/setup.sh Outdated Show resolved Hide resolved
@bukejiyu bukejiyu force-pushed the tmp_cpu branch 5 times, most recently from 9225301 to ac28f8f Compare June 27, 2024 02:52
Copy link
Contributor

@DesmonDay DesmonDay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ZeyuChen ZeyuChen merged commit 69e04c0 into PaddlePaddle:develop Jun 27, 2024
8 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants