-
Notifications
You must be signed in to change notification settings - Fork 60
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Framework] Continuous Batching Support #357
Merged
Merged
Commits on May 15, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 1cc9c1d - Browse repository at this point
Copy the full SHA 1cc9c1dView commit details -
Configuration menu - View commit details
-
Copy full SHA for d01be1a - Browse repository at this point
Copy the full SHA d01be1aView commit details -
Configuration menu - View commit details
-
Copy full SHA for db0c4e9 - Browse repository at this point
Copy the full SHA db0c4e9View commit details -
Configuration menu - View commit details
-
Copy full SHA for a4f4b25 - Browse repository at this point
Copy the full SHA a4f4b25View commit details -
Configuration menu - View commit details
-
Copy full SHA for 45bcfa3 - Browse repository at this point
Copy the full SHA 45bcfa3View commit details -
Configuration menu - View commit details
-
Copy full SHA for a704873 - Browse repository at this point
Copy the full SHA a704873View commit details -
Configuration menu - View commit details
-
Copy full SHA for 987a874 - Browse repository at this point
Copy the full SHA 987a874View commit details -
Configuration menu - View commit details
-
Copy full SHA for 451ef21 - Browse repository at this point
Copy the full SHA 451ef21View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5e98e6d - Browse repository at this point
Copy the full SHA 5e98e6dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 2b5e266 - Browse repository at this point
Copy the full SHA 2b5e266View commit details -
Configuration menu - View commit details
-
Copy full SHA for e12ffa8 - Browse repository at this point
Copy the full SHA e12ffa8View commit details -
Configuration menu - View commit details
-
Copy full SHA for a4442f0 - Browse repository at this point
Copy the full SHA a4442f0View commit details -
Configuration menu - View commit details
-
Copy full SHA for fb52594 - Browse repository at this point
Copy the full SHA fb52594View commit details -
Configuration menu - View commit details
-
Copy full SHA for aa48f7e - Browse repository at this point
Copy the full SHA aa48f7eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 3f15904 - Browse repository at this point
Copy the full SHA 3f15904View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8c2e6b4 - Browse repository at this point
Copy the full SHA 8c2e6b4View commit details -
[Model/Layer] New forward to support CB (CommonDecoder->DecoderBlock-…
…>DecoderLayer->Attention/MLP) (#375)
Configuration menu - View commit details
-
Copy full SHA for aac0167 - Browse repository at this point
Copy the full SHA aac0167View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0e35c8f - Browse repository at this point
Copy the full SHA 0e35c8fView commit details -
Configuration menu - View commit details
-
Copy full SHA for f441906 - Browse repository at this point
Copy the full SHA f441906View commit details -
Configuration menu - View commit details
-
Copy full SHA for f9bfb49 - Browse repository at this point
Copy the full SHA f9bfb49View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3f232c5 - Browse repository at this point
Copy the full SHA 3f232c5View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6625b01 - Browse repository at this point
Copy the full SHA 6625b01View commit details -
Configuration menu - View commit details
-
Copy full SHA for f220fe0 - Browse repository at this point
Copy the full SHA f220fe0View commit details -
Configuration menu - View commit details
-
Copy full SHA for eb417af - Browse repository at this point
Copy the full SHA eb417afView commit details -
Configuration menu - View commit details
-
Copy full SHA for b5bda0c - Browse repository at this point
Copy the full SHA b5bda0cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 35562c0 - Browse repository at this point
Copy the full SHA 35562c0View commit details -
Configuration menu - View commit details
-
Copy full SHA for e67e455 - Browse repository at this point
Copy the full SHA e67e455View commit details -
Configuration menu - View commit details
-
Copy full SHA for c576aff - Browse repository at this point
Copy the full SHA c576affView commit details -
Configuration menu - View commit details
-
Copy full SHA for eff6a75 - Browse repository at this point
Copy the full SHA eff6a75View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2b374ff - Browse repository at this point
Copy the full SHA 2b374ffView commit details -
Configuration menu - View commit details
-
Copy full SHA for 7e9d731 - Browse repository at this point
Copy the full SHA 7e9d731View commit details -
[Layer] Better method to reinterpret KV cache (#397)
* [Common] Add sequenceMeta, sequenceGroup and sequenecePool. (#343) * merge batchSize and seqLen into one in TokenEembedding * merge batchSize and seqLen into one in TokenEembedding (#350) * [Common] Move Martix into xft namespace. (#351) * remove unsed function in DecoderLayer * [Layer] Remove unused functions in Decoder layer (#353) * fix compile error of embeddingForward * [Model] Fix compile error of embeddingForward in YaRNLlama (#358) * [Common] Add sampling params into group seq. (#356) * remove DecoderContext in computeSoftmax * [Util] Remove DecoderContext in computeSoftmax (#362) * [Common] Refactor sequence.h. (#363) * [kernels] refactor flash attention for continuous batching (#361) * [models] Add attnMeta for continuous batching (#364) * [Layers] fix build error (#365) * [Model] add interface for seq meta. (#366) * refactor resize function in DecoderContext to support CB, and qkScores member removed * [Common] Modify resize() in DecoderContext to support (#367) * add some code to CommonDecoder::forward() * SequenceMeta refactor * [Model] New CommonDecoder::forward impl. skeleton (#369) * new KVCacheMgr supporting CB * fix typo & set default prefixId to -1 in addSequence() * [Common] New KVCacheMgr to support CB (#371) * [Sampling] Add repetition penalty for new seq type. (#373) * New foward to support CB (CommonDecoder->DecoderBlock->DecoderLayer->Attention/MLP) * add todo * [Sampling] Add greedy search for cb path. (#376) * logic issue fix * code fix to make new forward work * add maxSeqLen limitation * cross attention impl. for CB * DecoderContext::resize fix * correct the output of the new forward * add cb_check * fix incorrect buffer size calculation * 2 sequences -> 3 sequences * better method to prepare KV cache --------- Co-authored-by: Changqing Li <changqing.li@intel.com> Co-authored-by: Duyi-Wang <duyi.wang@intel.com> Co-authored-by: Meng,Chen <chen.meng@intel.com>
Configuration menu - View commit details
-
Copy full SHA for 524bf32 - Browse repository at this point
Copy the full SHA 524bf32View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3865654 - Browse repository at this point
Copy the full SHA 3865654View commit details -
Configuration menu - View commit details
-
Copy full SHA for af0aae8 - Browse repository at this point
Copy the full SHA af0aae8View commit details -
Configuration menu - View commit details
-
Copy full SHA for cef27bc - Browse repository at this point
Copy the full SHA cef27bcView commit details
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.