Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Layer] Cross attention impl. for CB #382

Merged
merged 46 commits into from
May 11, 2024
Merged

[Layer] Cross attention impl. for CB #382

merged 46 commits into from
May 11, 2024

Conversation

pujiang2018
Copy link
Contributor

No description provided.

changqi1 and others added 30 commits April 25, 2024 14:08
@changqi1
[Common] Add sequenceMeta, sequenceGroup and sequenecePool. (#343)
8f351ba
@pujiang2018
Merge commit 'd2b8df0c85ba57b62169d74c88192de3bf6e4820' into cb_dev
5ab63df
@pujiang2018
merge batchSize and seqLen into one in TokenEembedding
dbcb267
@pujiang2018
merge batchSize and seqLen into one in TokenEembedding (#350)
3949abd
@Duyi-Wang
[Common] Move Martix into xft namespace. (#351)
9a53fb2
@pujiang2018
Merge commit '9a53fb2ea6b9141ba7c045bc0d135c1809e8f22c' into pujiang/…
25ee312 
…feature/cb_dev
@pujiang2018
remove unsed function in DecoderLayer
376b2bc
@pujiang2018
[Layer] Remove unused functions in Decoder layer (#353)
7112b33
@pujiang2018
Merge commit '819ecccfa06662bddb16e9e1cd1ec8775d1c0180' into cb_dev
4ff4707
@pujiang2018
Merge commit '4ff47074fc85a27e13251c3fb618f36e338c456f' into pujiang/…
d281a54 
…feature/cb_dev
@pujiang2018
fix compile error of embeddingForward
b5b225a
@pujiang2018
[Model] Fix compile error of embeddingForward in YaRNLlama (#358)
f8f8571
@Duyi-Wang
[Common] Add sampling params into group seq. (#356)
b95dac1
@pujiang2018
remove DecoderContext in computeSoftmax
d5c9407
@pujiang2018
Merge commit 'f8f85714331c0df2ce4a8344e06972316770ec11' into pujiang/…
be615b2 
…feature/cb_dev
@pujiang2018
[Util] Remove DecoderContext in computeSoftmax (#362)
e48ea1f
@Duyi-Wang
[Common] Refactor sequence.h. (#363)
7ad311e
@abenmao
[kernels] refactor flash attention for continuous batching (#361)
9e7bdca
@abenmao
[models] Add attnMeta for continuous batching (#364)
cfab63a
@abenmao
[Layers] fix build error (#365)
deabd33
@Duyi-Wang
[Model] add interface for seq meta. (#366)
2499f60
@pujiang2018
Merge commit '2499f602c22184ca5afaa2f013ae0ff4e3bd4263' into pujiang/…
5833d41 
…feature/cb_dev
@pujiang2018
refactor resize function in DecoderContext to support CB, and qkScore…
0514833 
…s member removed
@pujiang2018
[Common] Modify resize() in DecoderContext to support (#367)
c792aff
@pujiang2018
Merge commit 'c792aff5f7cc8554afa0399f4b6d241333b0b56c' into pujiang/…
5315a5a 
…feature/cb_dev
@pujiang2018
add some code to CommonDecoder::forward()
63e895a
@pujiang2018
SequenceMeta refactor
9cfc7c6
@pujiang2018
[Model] New CommonDecoder::forward impl. skeleton (#369)
2e05716
@pujiang2018
new KVCacheMgr supporting CB
41e692c
@pujiang2018
Merge commit '2e057165b456ef9b88591a880403ffe47e7500c3' into pujiang/…
32c845c 
…feature/cb_dev
pujiang2018 and others added 16 commits May 7, 2024 02:24
@pujiang2018
fix typo & set default prefixId to -1 in addSequence()
c2ac8d2
@pujiang2018
[Common] New KVCacheMgr to support CB (#371)
dfa1d0e
@Duyi-Wang
[Sampling] Add repetition penalty for new seq type. (#373)
e79c9c2
@pujiang2018
Merge commit 'e79c9c21825ad51116f49a290baa7eea80e25b07' into pujiang/…
1e93ef0 
…feature/cb_dev
@pujiang2018
New foward to support CB (CommonDecoder->DecoderBlock->DecoderLayer->…
689b41a 
…Attention/MLP)
@pujiang2018
add todo
3ce8d06
@Duyi-Wang
[Sampling] Add greedy search for cb path. (#376)
9377f83
@pujiang2018
Merge commit '9377f8371ebfe4999c2171eceab106a52ff2618a' into pujiang/…
b71ec05 
…feature/cb_dev
@pujiang2018
logic issue fix
3144675
@pujiang2018
Merge branch 'cb_dev' into pujiang/feature/cb_dev
fcee295
@pujiang2018
Merge commit '77dd36b9e5c6360be601e8d5cf76cbcddf32191b' into pujiang/…
24a8cb4 
…feature/cb_dev
@pujiang2018
Merge commit '3ef3c1e0f8f6032e30d8c67dbcfedfdc27a5e372' into pujiang/…
5512f3c 
…feature/cb_dev
@pujiang2018
code fix to make new forward work
32150c5
@pujiang2018
add maxSeqLen limitation
c55bf77
@pujiang2018
Merge commit '3f8a95d1935ae4043ed3a8b425bec23871629cfb' into pujiang/…
278fab4 
…feature/cb_dev
@pujiang2018
cross attention impl. for CB
727a1cb
@pujiang2018 pujiang2018 added the continuous batching continuous batching label May 10, 2024
@pujiang2018 pujiang2018 requested a review from changqi1 May 10, 2024 09:03
@pujiang2018 pujiang2018 merged commit 08cd8f7 into cb_dev May 11, 2024
Duyi-Wang pushed a commit that referenced this pull request May 15, 2024
@pujiang2018 @Duyi-Wang
[Layer] Cross attention impl. for CB (#382)
f220fe0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@changqi1 changqi1 Awaiting requested review from changqi1

Assignees
No one assigned
Labels
continuous batching continuous batching
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@pujiang2018 @changqi1 @Duyi-Wang @abenmao