Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MPT support #290

Closed
2 tasks done
generalsvr opened this issue May 6, 2023 · 8 comments · Fixed by #514
Closed
2 tasks done

MPT support #290

generalsvr opened this issue May 6, 2023 · 8 comments · Fixed by #514

Comments

@generalsvr
Copy link

Model description

Can you add a new MPT model? This looks very promising, especially the ability to extend context length by up to 85K tokens.

Open source status

  • The model implementation is available
  • The model weights are available

Provide useful links for the implementation

https://github.com/mosaicml/llm-foundry
@vvsotnikov
Copy link

Probably a related issue: huggingface/transformers#23174

@dongs0104 dongs0104 mentioned this issue May 16, 2023
Closed
2 tasks
@arnocandel arnocandel mentioned this issue Jun 22, 2023
Closed
@monuminu
Copy link

any plan on supporting mosaicml/mpt-30b-instruct support

@ccasimiro88
Copy link

I am also interested in deploying the mosaicml/mpt-30b-chat model. Would be really useful for the community! 🙏

@tim-a-davis
Copy link

Yes I am also interested in getting support for MPT models. I would love to assist in any way I can.

@mantrakp04
Copy link

+1

@alanxmay
Copy link

please🙏

@mantrakp04
Copy link

Yes I am also interested in getting support for MPT models. I would love to assist in any way I can.

Hey wanna work on the implementation then we can do a pr, add me on discord mantrakp- proud Logitech controller owner

@louis030195 louis030195 mentioned this issue Jun 27, 2023
Closed
@Narsil
Copy link
Collaborator

Narsil commented Jul 1, 2023

#514

@Narsil Narsil mentioned this issue Jul 1, 2023
Merged
5 tasks
OlivierDehaene pushed a commit that referenced this issue Jul 3, 2023
@Narsil
feat(server): Add Non flash MPT. (#514)
1da07e8 
# What does this PR do?


This adds a non flash version of MPT.
Flash is harder because we need to create a bias ready cuda kernel of
flash attention.

Fixes
#361
Fixes
#491
Fixes
#290
AIProphet added a commit to AIProphet/text-generation-inference that referenced this issue Jul 12, 2023
@AIProphet
feat(server): Add Non flash MPT. (#514)
b4def7f 
# What does this PR do?


This adds a non flash version of MPT.
Flash is harder because we need to create a bias ready cuda kernel of
flash attention.

Fixes
huggingface/text-generation-inference#361
Fixes
huggingface/text-generation-inference#491
Fixes
huggingface/text-generation-inference#290
verdant621 added a commit to verdant621/text-generation-inference that referenced this issue Oct 19, 2023
@verdant621
feat(server): Add Non flash MPT. (#514)
2aa866d 
# What does this PR do?


This adds a non flash version of MPT.
Flash is harder because we need to create a bias ready cuda kernel of
flash attention.

Fixes
huggingface/text-generation-inference#361
Fixes
huggingface/text-generation-inference#491
Fixes
huggingface/text-generation-inference#290
cr313 added a commit to cr313/text-generation-inference-load-test that referenced this issue Apr 19, 2024
@cr313
feat(server): Add Non flash MPT. (#514)
94dca74 
# What does this PR do?


This adds a non flash version of MPT.
Flash is harder because we need to create a bias ready cuda kernel of
flash attention.

Fixes
huggingface/text-generation-inference#361
Fixes
huggingface/text-generation-inference#491
Fixes
huggingface/text-generation-inference#290
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Non flash MPT. huggingface/text-generation-inference
8 participants
@Narsil @vvsotnikov @ccasimiro88 @generalsvr @monuminu @tim-a-davis @mantrakp04 @alanxmay