Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(speculative-sampling): allow to specify a draft model in the model config #1052

Merged
merged 2 commits into from
Sep 14, 2023

Conversation

mudler
Copy link
Owner

@mudler mudler commented Sep 14, 2023

Description

This PR fixes #1013.

It adds draft_model and n_draft to the model YAML config in order to load models with speculative sampling. This should be compatible as well with grammars.

example:

backend: llama                                                                                                                                                                   
context_size: 1024                                                                                                                                                                        
name: my-model-name
parameters:
  model: foo-bar
n_draft: 16                                                                                                                                                                      
draft_model: model-name

…el config

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler mudler added the enhancement New feature or request label Sep 14, 2023
@mudler mudler merged commit 8ccf5b2 into master Sep 14, 2023
14 checks passed
@mudler mudler deleted the spec_sampling branch September 14, 2023 15:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

feat: speculative sampling
1 participant