Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add exllama #881

Merged
merged 1 commit into from Aug 10, 2023
Merged

feat: Add exllama #881

merged 1 commit into from Aug 10, 2023

Conversation

mudler
Copy link
Owner

@mudler mudler commented Aug 10, 2023

Description

This PR fixes #796

LocalAI Setup

This is an extra backend - in the container is already available and there is nothing to do for the setup.

Model setup

Download the model as a folder inside the model directory and create a YAML file specifying the exllama backend. For instance:

$ ls models/                                                                 
.keep                        WizardLM-7B-uncensored-GPTQ/ exllama.yaml                                   
$ cat models/exllama.yaml                                                     
name: exllama                                                                                            
parameters:                                                                                              
  model: WizardLM-7B-uncensored-GPTQ                                                                     
backend: exllama              
# ...

Notes:

  • Token stream is not yet supported, left for an enhancement later

Notes for Reviewers

Signed commits

  • Yes, I signed my commits.

@mudler mudler force-pushed the exllama branch 2 times, most recently from afa71c8 to dc5b724 Compare August 10, 2023 20:51
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler mudler added the enhancement New feature or request label Aug 10, 2023
@mudler mudler merged commit ff3ab5f into master Aug 10, 2023
14 checks passed
@mudler mudler deleted the exllama branch August 10, 2023 22:49
@fblissjr
Copy link

Great news!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

feature: support for exllama and AutoGPTQ
2 participants