Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(transformer): transformer-based backend #1015

Closed
mudler opened this issue Sep 6, 2023 · 2 comments · Fixed by #1308
Closed

feat(transformer): transformer-based backend #1015

mudler opened this issue Sep 6, 2023 · 2 comments · Fixed by #1308
Assignees
@mudler
Labels
enhancement New feature or request roadmap

Comments

@mudler
Copy link
Owner

mudler commented Sep 6, 2023 •
edited

Is your feature request related to a problem? Please describe.
It makes sense to have a backend which is tied to https://github.com/huggingface/transformers. This would increase the range of support for LLM that LocalAI already offers

Describe the solution you'd like
Ideally would be preferred to use a wrapper around transformers and xformers to avoid duplicating efforts.

Describe alternatives you've considered

Additional context
@mudler mudler added the enhancement New feature or request label Sep 6, 2023
@mudler mudler self-assigned this Sep 6, 2023
mudler added a commit that referenced this issue Sep 9, 2023
@mudler
feat(vllm): Initial vllm backend implementation
a3fe9d2 
Related to: #1015

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler mudler mentioned this issue Sep 9, 2023
Merged
mudler added a commit that referenced this issue Sep 9, 2023
@mudler
feat(vllm): Initial vllm backend implementation
c0bb5c4 
Related to: #1015

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
mudler added a commit that referenced this issue Sep 10, 2023
@mudler
feat(vllm): Initial vllm backend implementation (#1026)
d6614f3 
Related to: #1015
@mudler mudler added the roadmap label Sep 14, 2023
@yhyu13
Copy link

yhyu13 commented Sep 21, 2023

Is Vllm fully integrated? vLLM begins to seem to be many AWQ quantized model's first choice recently

@mudler
Copy link
Owner Author

mudler commented Sep 21, 2023

Is Vllm fully integrated? vLLM begins to seem to be many AWQ quantized model's first choice recently

yep, going into the next release soon!

@mudler mudler mentioned this issue Sep 22, 2023
Merged
1 task
mudler added a commit that referenced this issue Sep 22, 2023
@mudler
feat(vllm): Allow to set quantization (#1094)
a28ab18 
This particularly useful to set AWQ

**Description**

Follow up of #1015 

**Notes for Reviewers**


**[Signed
commits](../CONTRIBUTING.md#signing-off-on-commits-developer-certificate-of-origin)**
- [ ] Yes, I signed my commits.
 

<!--
Thank you for contributing to LocalAI! 

Contributing Conventions:

1. Include descriptive PR titles with [<component-name>] prepended.
2. Build and test your changes before submitting a PR. 
3. Sign your commits

By following the community's contribution conventions upfront, the
review process will
be accelerated and your PR merged more quickly.
-->

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler mudler mentioned this issue Oct 2, 2023
Open
90 tasks
@mudler mudler changed the title feat(vllm): transformer-based backend feat(transformer): transformer-based backend Nov 15, 2023
@mudler mudler mentioned this issue Nov 20, 2023
Closed
1 task
@mudler mudler linked a pull request Nov 20, 2023 that will close this issue
feat(transformers): add embeddings with Automodel #1308
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mudler mudler

Labels
enhancement New feature or request roadmap
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(transformers): add embeddings with Automodel mudler/LocalAI
2 participants
@mudler @yhyu13