-
Notifications
You must be signed in to change notification settings - Fork 50
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Model/Layer] New forward to support CB (CommonDecoder->DecoderBlock->DecoderLayer->Attention/MLP) #375
Conversation
@abenmao @Duyi-Wang pls help to review. |
@@ -265,10 +254,6 @@ class CommonDecoder : public AbstractDecoder { | |||
if (this->attnMask) free(this->attnMask); | |||
|
|||
delete this->predictor; | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Free the decoderBlock here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
src/layers/decoder_block.h
Outdated
@@ -0,0 +1,348 @@ | |||
// Copyright (c) 2023 Intel Corporation |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
2024
src/layers/decoder_block.h
Outdated
totInSeqLen += seqs[i]->getInputSeqLen(); | ||
} | ||
|
||
kvCacheMgr.prepareCache(seqIDs); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I prefer to prepare KVCache at a high level api.
We can assume all seqs' KVCache is ready.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
commented and added TODO
…>DecoderLayer->Attention/MLP) (#375)
No description provided.