Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add /models/apply endpoint to prepare models #286

Merged
merged 11 commits into from
May 18, 2023
Merged

Conversation

mudler
Copy link
Owner

@mudler mudler commented May 17, 2023

This change allow to use the API to programmatically download models from a "gallery'.
It doesn't introduce the concept of the gallery, however, it introduces a way to install models from pre-defined yaml files which would be part of it, so it provides the "base" building block.

It exposes an api endpoint to apply new models from gallery YAML files. gallery files includes information to retrieve the model, templates to use, and its configuration file.

For instance, consider:

name: "groovy"
description: |
    LocalAI GPT4ALL-J Model
license: "Apache 2.0"
urls:
- gpt4all.io

config_file: |
    parameters:
      model: groovy
      top_k: 80
      temperature: 0.2
      top_p: 0.7
    context_size: 1024
    stopwords:
    - "HUMAN:"
    - "GPT:"
    roles:
      user: ""
      system: ""
    template:
      completion: "groovy-completion"
      chat: groovy-chat

files:
    - filename: "groovy"
      sha: ""
      uri: "https://gpt4all.io/models/ggml-gpt4all-j-v1.3-groovy.bin"

prompt_templates:
    - name: "groovy-completion"
      content: |
        Complete the prompt
        ### Prompt:
        {{.Input}}
        ### Response:
    - name: "groovy-chat"
      content: |
        The prompt below is a question to answer, a task to complete, or a conversation to respond to; decide which and write an appropriate response.
        ### Prompt:
        {{.Input}}
        ### Response:

uploaded as a gist, could be used as so to configure LocalAI to download all the necessary files in runtime to start using the model:

$ curl http://localhost:8080/models/apply -H "Content-Type: application/json" -d '{
     "url": "https://gist.githubusercontent.com/mudler/6112666e77061fe35ca3a535d25ccddc/raw/45eabb442f25db5bc991f2d2a616c15d4c26be67/gpt4all-j-localai.yaml"
   }'
{"uid":"1059474d-f4f9-11ed-8d99-c4cbe106d571","status":"http://localhost:8080/models/jobs/1059474d-f4f9-11ed-8d99-c4cbe106d571"}
$ curl http://localhost:8080/models/jobs/1059474d-f4f9-11ed-8d99-c4cbe106d571      
{"error":null,"processed":true,"message":"completed"}

This will:

  • Start a batch job to download the model and the relevant files
  • Reload the models once finished to pick up the model

In a bash script, you can wait on the operation to finish by checking the processed field:

model_url="https://gist.githubusercontent.com/mudler/6112666e77061fe35ca3a535d25ccddc/raw/45eabb442f25db5bc991f2d2a616c15d4c26be67/gpt4all-j-localai.yaml"

response=$(curl -s http://localhost:8080/models/apply -H "Content-Type: application/json" -d '{"url": "$model_url"}')

job_id=$(echo "$response" | jq -r '.uid')

while [ "$(curl -s http://localhost:8080/models/jobs/"$job_id" | jq -r '.processed')" != "true" ]; do 
  sleep 1
done

echo "Job completed"

Related to: #100

@mudler mudler marked this pull request as draft May 17, 2023 17:29
@mudler mudler force-pushed the batch_download branch 2 times, most recently from d095f4d to 945a9c3 Compare May 17, 2023 20:20
@mudler mudler marked this pull request as ready for review May 17, 2023 23:04
@mudler mudler changed the title feat: batch download feat: add /models/apply endpoint to prepare models May 18, 2023
@mudler mudler merged commit cc9aa9e into master May 18, 2023
@mudler mudler deleted the batch_download branch May 18, 2023 13:59
@mudler mudler mentioned this pull request May 26, 2023
6 tasks
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant