Skip to content
This repository was archived by the owner on Jul 4, 2025. It is now read-only.

feat: Support for LVM with openai compatible API with local image path #360

Conversation

hiro-v
Copy link
Contributor

@hiro-v hiro-v commented Jan 18, 2024

Fixes for #303
If it works, can support menloresearch/jan#294

  • Support for remote image URL (contains http)
  • Support for local image path (by default)
  • Support for base64 image (contains data:base64)

Steps to test

  1. Download models at https://huggingface.co/mys/ggml_llava-v1.5-7b/tree/main
  2. Load LVM models
curl --location 'http://127.0.0.1:3928/inferences/llamacpp/loadmodel' \
--header 'Content-Type: application/json' \
--data '{
   "llama_model_path": "/Users/hiro/Downloads/ggml-model-q4_k.gguf",
   "mmproj": "/Users/hiro/Downloads/mmproj-model-f16.gguf",
   "ctx_len": 2048,
   "ngl": 100,
   "cont_batching": false,
   "embedding": false,
   "system_prompt": "",
   "user_prompt": "\n### Instruction:\n",
   "ai_prompt": "\n### Response:\n"
 }'
  1. Infer with chat/completion. The image_url.url can be either base64 or local path. The remote model is not supported yet.
curl http://127.0.0.1:3928/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "gpt-4-vision-preview",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "What’s in this image?"
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "<PAYLOAD>"
            }
          }
        ]
      }
    ],
    "max_tokens": 300
  }'

@hiro-v hiro-v added the P1: important Important feature / fix label Jan 18, 2024
@hiro-v hiro-v requested a review from tikikun January 18, 2024 07:19
@hiro-v hiro-v self-assigned this Jan 18, 2024
@hiro-v hiro-v marked this pull request as draft January 18, 2024 07:21
@hiro-v hiro-v changed the title feat: Support for LVM with openai compatible API with remote and local image path feat: Support for LVM with openai compatible API with local image path Jan 18, 2024
@hiro-v hiro-v marked this pull request as ready for review January 19, 2024 02:17
Copy link
Contributor

@tikikun tikikun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hiro-v hiro-v merged commit cb6c3d8 into main Jan 24, 2024
@hiro-v hiro-v deleted the 303-feat-nitro-chat-completion-with-image-supporting-local-image-path branch January 24, 2024 02:20
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
P1: important Important feature / fix
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

2 participants