feat: nitro multi modal

[tikikun]

No description provided.

[tikikun]

example of loading LLAVA:

curl -X POST 'http://localhost:3928/inferences/llamacpp/loadModel' \
-H 'Content-Type: application/json' \
-d '{
    "llama_model_path": "/Users/alandao/Downloads/ggml-model-q4_k.gguf","mmproj":"/Users/alandao/Downloads/mmproj-model-f16.gguf"
    ,"ctx_len": 4096,
    "ngl": 100, "cont_batching":false, "embedding":false}'

Example of doing inferencing with image

import base64
import requests

# OpenAI API Key
api_key = "YOUR_OPENAI_API_KEY"

# Function to encode the image
def encode_image(image_path):
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode('utf-8')


headers = {
    'Content-Type': 'application/json',
    'Authorization': f'Bearer {api_key}',
}

# Define the image_b64 variable here
image_b64 = encode_image('/Users/alandao/Downloads/download.jpeg')

json_data = {
    "model": "text-davinci-003",
    "messages": [
        {"role": "assistant", "content": "Hello there 👋"},
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe this image"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/jpeg;base64,{image_b64}"
                    },
                },
            ],
        },
    ],
    "stream": False,
}


def print_streaming_response(response):
    for chunk in response.iter_lines():
        if chunk:
            print(chunk.decode('utf-8'))


response = requests.post('http://localhost:3928/v1/chat/completions', headers=headers, json=json_data)
print_streaming_response(response)

[tikikun]

@vuonghoainam

[tikikun]

mode link
https://huggingface.co/mys/ggml_llava-v1.5-7b