Skip to content

bug: ai-proxy-multi: Variable $request_llm_model reset in fallback_strategy #13769

Description

@RGZingYang

Current Behavior

开启日志记录:
nginx_config:
http:
access_log_format: "$remote_addr - $remote_user [$time_local] $http_host "$request_line" $status $body_bytes_sent $request_time "$http_referer" "$http_user_agent" $upstream_addr $upstream_status $apisix_upstream_response_time "$upstream_scheme://$upstream_host$upstream_uri" "$apisix_request_id" "$request_type" "$llm_time_to_first_token" "$llm_model" "$request_llm_model" "$llm_prompt_tokens" "$llm_completion_tokens""
使用ai-proxy-multi创建路由 开启fallback_strategy, 发生重试后$request_llm_model不是原始请求的model
创建路由的curl:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${admin_key}" \
  -d '{
    "id": "ai-proxy-multi-route",
    "uri": "/v1/chat/completions",
    "methods": ["POST"],
            "vars": [
            [
                "post_arg.model",
                "==",
                "model_test"
            ]
        ],
    "plugins": {
 "ai-proxy-multi": {
        "instances": [
          {
            "provider": "openai-compatible",
            "options": {
              "model": "upstream-model-A"
            },
            "override": {
              "endpoint": "http://172.0.0.1:8081/test-upstream/v1/chat/completions"
            },
            "priority": 1,
            "name": "test1",
            "auth": {},
            "weight": 1
          },
          {
            "provider": "openai-compatible",
            "options": {
              "model": "upstream-model-B"
            },
            "override": {
              "endpoint": "http://172.0.0.1:8081/test-upstream/v1/chat/completions"
            },
            "priority": 0,
            "name": "test2",
             "auth": {},
            "weight": 1
          }
        ],

        "balancer": {
          "hash_on": "consumer",
          "algorithm": "chash"
        },
        "timeout": 500,
        "fallback_strategy": [
          "http_429",
          "http_5xx"
        ],
        "logging": {
          "summaries": true,
          "payloads": true
        },
        "keepalive": false
      },
    }
  }'

步骤:
发送请求:{
"model": "model_test",
"messages": [
{
"role": "user",
"content": "返回1"
}
]
}

上游正常返回 $request_llm_model 输出的是请求中的model参数 model_test $llm_model输出 upstream-model-A 符合预期
Image
发生重试后 $request_llm_model 输出为 upstream-model-A $llm_model输出 upstream-model-B 不符合预期 $request_llm_model应还是输出 model_test :Image

Expected Behavior

$request_llm_model 应是保持原始的模型名称 重试不应该改变, 即使为了兼容不使用 "vars": [["post_arg.model","==","model_test"],请求参数不带model 也应该在赋值的时候判断下 是否为空 如果不为空的情况下不重新赋值

Error Logs

重试会调用base.before_proxy的方法重试
if request_model then
ctx.var.request_llm_model = request_model
end
赋值的时候需要添加判断是否已经赋过值了Image

Steps to Reproduce

步骤如描述中

Environment

  • APISIX version (run apisix version): 3.17.0
  • Operating system (run uname -a):
  • OpenResty / Nginx version (run openresty -V or nginx -V):
  • etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
  • APISIX Dashboard version, if relevant:
  • Plugin runner version, for issues related to plugin runners:
  • LuaRocks version, for installation issues (run luarocks --version):

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingplugin

    Type

    No type

    Projects

    Status
    📋 Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions