Enhance autogptq backend to support VL models

[thiner]

Description

Due to prompt template incompatible with current prompt processing flow, I removed support for internlm/internlm-xcomposer2-vl-7b-4bit in this PR. I may add the support in a separate PR.

This PR fixes #1812, and enhanced AutoGPTQ to support Qwen/Qwen-VL-Chat-Int4 and internlm/internlm-xcomposer2-vl-7b-4bit models.
The AutoGPTQ version is updated to v0.7.1, and enables use_marlin flag for internlm/internlm-xcomposer2-vl-7b-4bit by default. Please make sure you have the correct model in place. Ref: https://github.com/IST-DASLab/marlin

Notes for Reviewers

Signed commits

• Yes, I signed my commits.

[netlify]

✅ Deploy Preview for localai canceled.

Name	Link
🔨 Latest commit	afa16d3
🔍 Latest deploy log	https://app.netlify.com/sites/localai/deploys/6602b29e2761660008b670c6

[thiner]

I've no idea what is the difference between autogptq.yml and transformers-*.yml in common-env folder. If autogptq shares the same dependencies with transformers, does that mean no need to create a separate autogptq conda env?

[thiner]

Just try to remove the "pending" status.

[thiner]

How to retrigger the test build?

[thiner]

@mudler Below code snippet shows that images in payload are converted to base64 StringImage. I think that's the reason why calling the models kept failing.

if pp.Type == "image_url" {
	// Detect if pp.ImageURL is an URL, if it is download the image and encode it in base64:
	base64, err := getBase64Image(pp.ImageURL.URL)
	if err == nil {
		input.Messages[i].StringImages = append(input.Messages[i].StringImages, base64) // TODO: make sure that we only return base64 stuff
		// set a placeholder for each image
		input.Messages[i].StringContent = fmt.Sprintf("[img-%d]", index) + input.Messages[i].StringContent
		index++
}

I found a relevant question in Qwen-VL issues, this answer QwenLM/Qwen-VL#112 (comment) reveals that qwen-vl doesn't support base64 as image input. It supports only URL or image file path.
To solve the problem, I think we have two options:

1. Refactor request.go, allow pass image_url to template directly. This solution requires updating the template evaluation engine, allow reference the image_url and text in the API payload.
2. Keep request.go logic as is, convert the base64 string into image in autogptq.py, then save it to a temporary folder instead. Then we can pass the image file path to the model. But the question is how to get the StringImage value in autogptq.py?

Please share your thoughts.

[mudler]

@mudler Below code snippet shows that images in payload are converted to base64 StringImage. I think that's the reason why calling the models kept failing.

if pp.Type == "image_url" {
	// Detect if pp.ImageURL is an URL, if it is download the image and encode it in base64:
	base64, err := getBase64Image(pp.ImageURL.URL)
	if err == nil {
		input.Messages[i].StringImages = append(input.Messages[i].StringImages, base64) // TODO: make sure that we only return base64 stuff
		// set a placeholder for each image
		input.Messages[i].StringContent = fmt.Sprintf("[img-%d]", index) + input.Messages[i].StringContent
		index++
}

I found a relevant question in Qwen-VL issues, this answer QwenLM/Qwen-VL#112 (comment) reveals that qwen-vl doesn't support base64 as image input. It supports only URL or image file path. To solve the problem, I think we have two options:

1. Refactor `request.go`, allow pass `image_url` to template directly. This solution requires updating the template evaluation engine, allow reference the `image_url` and `text` in the API payload.

2. Keep `request.go` logic as is, convert the base64 string into image in `autogptq.py`, then save it to a temporary folder instead. Then we can pass the image file path to the model. But the question is how to get the StringImage value in `autogptq.py`?

Please share your thoughts.

The option 1 would be what I'd try first, however might be more complex to achieve. You need to be careful to keep a consistent logic between the golang API and the llama.cpp grpc-server. If you are not much into put your hands on C++ that might be more complex to explore

The option 2 sounds more easy to do. In the llama.cpp backend we do:

    //
    for (int i = 0; i < predict->images_size(); i++) {
        data["image_data"].push_back(json
            {
                {"id", i},
                {"data",    predict->images(i)},
            });
    }

because Images inside the request is an array, and each entry contains the base64 encoded images (see here for the golang counter-part). Each [img-0], [img-1] is the index used by the backend to retrieve the image

[thiner]

@mudler I have finally implemented this feature for Qwen-vl model. Please review.

I tested this feature with below configuration.

1. Model: Qwen/Qwen-VL-Chat-Int4
2. The model configuration qwen-vl.yaml

    # Model name.
  # The model name is used to identify the model in the API calls.
- name: gpt-4-vision-preview
  # Default model parameters.
  # These options can also be specified in the API calls
  parameters:
    model: /opt/models/qwen-vl-chat-int4
    temperature: 0.7
    top_k: 85
    top_p: 0.7

  # Default context size
  context_size: 4096
  # Default number of threads
  threads: 16
  backend: autogptq

  # define chat roles
  roles:
    user: "user:"
    assistant: "assistant:"
    system: "system:"
  template:
    chat: &template |
      {{.Input}}
    completion: *template
  # Enable F16 if backend supports it
  f16: true
  embeddings: false
  # Enable debugging
  debug: true

  # GPU Layers (only used when built with cublas)
  gpu_layers: -1

  # Diffusers/transformers
  cuda: true

3. The gpt-vision request:

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d \
'{"model": "gpt-4-vision-preview", "messages": [{"role": "user", \
"content": [{"type":"text", "text": "Describe the image?"}, {"type": "image_url", "image_url": {"url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg" }}], "temperature": 0.2}]}'

4. The output:

{
    "created": 1711430450,
    "object": "chat.completion",
    "id": "23add6fd-009e-43b4-a361-341cb2386672",
    "model": "gpt-4-vision-preview",
    "choices": [
        {
            "index": 0,
            "finish_reason": "stop",
            "message": {
                "role": "assistant",
                "content": "A woman sitting on a beach with a dog and giving it a high five."
            }
        }
    ],
    "usage": {
        "prompt_tokens": 0,
        "completion_tokens": 0,
        "total_tokens": 0
    }
}

[mudler]

Looking good here, thanks @thiner !

[thiner]

@mudler I just found that this feature doesn't work if there are multiple rounds conversation. Below is the error message:

 [10.70.2.129]:50410 200 - GET /readyz

 8:24AM DBG GRPC(/opt/models/llava/qwen-vl-chat-int4-127.0.0.1:37607): stderr generated_text: Instruct: user:g'day

 8:24AM DBG GRPC(/opt/models/llava/qwen-vl-chat-int4-127.0.0.1:37607): stderr assistant:G'day! How can I assist you today?

 8:24AM DBG GRPC(/opt/models/llava/qwen-vl-chat-int4-127.0.0.1:37607): stderr 

 8:24AM DBG GRPC(/opt/models/llava/qwen-vl-chat-int4-127.0.0.1:37607): stderr user:who are you

 8:24AM DBG GRPC(/opt/models/llava/qwen-vl-chat-int4-127.0.0.1:37607): stderr assistant:I am Buddy, your AI assistant, designed to provide assistance, answer questions, and engage in conversations on a wide range of topics. How can I help you today?

 8:24AM DBG GRPC(/opt/models/llava/qwen-vl-chat-int4-127.0.0.1:37607): stderr 

 8:24AM DBG GRPC(/opt/models/llava/qwen-vl-chat-int4-127.0.0.1:37607): stderr user:<img>/tmp/vl-1712132673400.jpg</img>,describe the image

 8:24AM DBG GRPC(/opt/models/llava/qwen-vl-chat-int4-127.0.0.1:37607): stderr Output:

 8:24AM DBG GRPC(/opt/models/llava/qwen-vl-chat-int4-127.0.0.1:37607): stderr A large building with a lot of windows next to a body of water.

 Error rpc error: code = Unknown desc = Exception iterating responses: 'Result' object is not an iterator

This relevant code snippet:

t = pipeline(compiled_prompt)[0]["generated_text"]
        print(f"generated_text: {t}", file=sys.stderr)

        if compiled_prompt in t:
            t = t.replace(compiled_prompt, "")

        # house keeping. Remove the image files from /tmp folder
        for img_path in prompt_images[1]:
            try:
                os.remove(img_path)
            except Exception as e:
                print(f"Error removing image file: {img_path}, {e}", file=sys.stderr)

        return backend_pb2.Result(message=bytes(t, encoding='utf-8'))

According to the code structure and the error message, I suspect the error was thrown from the return statement. Maybe something wrong while pack the result in protobuff?