Compatibility Issues with Qwen Series Models (VL, QVQ-max) via LiteLLM Proxy

[Uc207Pr4f57t9-251]

I've been attempting to integrate Alibaba Cloud's Qwen series models (specifically dashscope/qwen-vl-max and dashscope/qvq-max) into Bytebot using the recommended LiteLLM proxy setup (local Docker Compose). While basic connectivity was established after significant debugging (related to agent authentication and build caching), severe compatibility issues remain with these specific models, preventing their effective use.

1. Qwen-VL Models (e.g., qwen-vl-max) - Non-Standard Tool Calling:

• Problem: When bytebot-agent sends a request with tools defined and tool_choice: "auto", qwen-vl-max (via LiteLLM) does not populate the standard tool_calls field in the response. Instead, it returns tool_calls: null and embeds the intended tool calls as JSON code blocks (e.g., ```json { "name": "...", "input": {...} } ```) directly within the message.content field, often mixed with natural language "thinking" text.
• Impact: bytebot-agent's current response parser (formatChatCompletionResponse in proxy.service.ts) only checks the message.tool_calls field. Since it's null, the agent fails to recognize or execute the tool calls, treating the entire content (including the JSON blocks) as plain text output to the user.
• Debugging Done:
  • Confirmed via direct curl/Python requests to litellm-proxy that the issue persists even when bytebot-agent is bypassed, proving it's an incompatibility between Qwen-VL's output format and the standard expected by the agent.
  • Ensured reasoning_effort parameter was removed from bytebot-agent source code (proxy.service.ts) via --no-cache builds, ruling it out as the cause.
  • Configured tool_choice: "auto" via LiteLLM UI ("Default Parameters") for the model, which did not resolve the issue.

2. QVQ/QVQ-max Models - HTTPS Connection Error:

• Problem: Attempts to use dashscope/qvq-max consistently fail with the error litellm.BadRequestError: DashscopeException - current user api does not support http call.
• Debugging Done:
  • Verified multiple times that the api_base configured in the LiteLLM UI for this model is correctly set to HTTPS: https://dashscope.aliyuncs.com/compatible-mode/v1.
  • This error occurs even when the Docker Desktop global proxy and any system-level proxies (like Clash) are completely disabled, ruling out proxy HTTPS-stripping.
  • Other Dashscope models (like qwen-vl-max) connect successfully over HTTPS using the same LiteLLM proxy setup.
• Conclusion: This suggests a specific issue either with the Dashscope endpoint for qvq-max when accessed via the OpenAI compatibility layer, or with how the LiteLLM adapter handles HTTPS requests only for this specific model variant.

Environment:

• Bytebot: Built locally from source (recent edge equivalent).
• LiteLLM: Running via Docker using ghcr.io/berriai/litellm:main-stable image.
• Models Tested: dashscope/qwen-vl-max, dashscope/qvq-max.
• Setup: Local Docker Compose on Windows, using the project's provided postgres container for both bytebot-agent and litellm-proxy databases (bytebotdb and litellm_logs_db respectively). LiteLLM configured with master_key and encryption_key.

Workarounds Attempted:

• Qwen-VL: Manually modified bytebot-agent's formatChatCompletionResponse function in proxy.service.ts to parse ```json ... ``` blocks from message.content when message.tool_calls is null. (Code based on our discussion can be provided if needed). This works but requires modifying agent source.
• QVQ-max: No workaround found. Model remains unusable due to the persistent HTTPS error.

Suggestions:

1. Enhance bytebot-agent Parser: Update formatChatCompletionResponse to include fallback logic that parses JSON code blocks from message.content if message.tool_calls is null/empty. This would provide out-of-the-box compatibility with models like Qwen-VL.
2. Investigate QVQ HTTPS Issue: This seems like a deeper issue, potentially within LiteLLM's Dashscope adapter or the Dashscope endpoint itself. Collaboration with the LiteLLM team might be needed.
3. Document Compatibility: Update Bytebot documentation regarding known compatibility issues with specific Qwen models and potential workarounds (like the agent code modification).
4. Agent Authentication: Address the underlying issue where bytebot-agent ignores standard proxy keys and requires hardcoding or the OPENAI_API_KEY workaround.

Thanks for looking into this. Qwen models are important in certain regions, and improving compatibility would be very beneficial.

[BuesrB]

This is particularly important. To the extent of my knowledge, Qwen VL is also better suited for computer-using agent (with the latest open-source model Qwen3 VL). So, integrating it is definitely very crucial. I am using OpenRouter in my bytebot fork - to use the deployed Qwen3VL. Btw, a quick question, do you think using litellm is better?

[Uc207Pr4f57t9-251]

Hi @BuesrB
Your approach using OpenRouter in a Bytebot fork sounds quite interesting, especially since you mentioned Qwen3 VL seems well-suited for this type of agent work.
I took a look at your GitHub profile hoping to find the fork you mentioned and learn more, but I wasn't able to locate it. Would you be willing to share a link to your fork if it's public?
More specifically, I'm particularly curious about how you handled the Qwen VL tool calling incompatibility we discussed.
Does OpenRouter handle this adaptation automatically for Qwen models, or did you need to implement specific adapter logic (e.g., parsing the content field) or perhaps use targeted prompt engineering within your fork to get the tool calls working reliably? Thanks!

[vxtra1973]

@Uc207Pr4f57t9-251

I made a small change to support the passing of model name when using litellm and was able to get btyebot to work with qwen3-vl. It can open a browser, but fails on other tool calls due to:

#153

So it looks like there is some command mapping to do.

[samkoesnadi]

Hi @BuesrB Your approach using OpenRouter in a Bytebot fork sounds quite interesting, especially since you mentioned Qwen3 VL seems well-suited for this type of agent work. I took a look at your GitHub profile hoping to find the fork you mentioned and learn more, but I wasn't able to locate it. Would you be willing to share a link to your fork if it's public? More specifically, I'm particularly curious about how you handled the Qwen VL tool calling incompatibility we discussed. Does OpenRouter handle this adaptation automatically for Qwen models, or did you need to implement specific adapter logic (e.g., parsing the content field) or perhaps use targeted prompt engineering within your fork to get the tool calls working reliably? Thanks!

The fork is here https://github.com/kira-id/cua.kira . We have improved it - it works now with Qwen3VL, takeover desktop directly on home, also added Blender on Desktop, merged some PR here that actually is useful and more. Could you please take a look and see if it works for you?

The adapter logic is modified accordingly continuing the PR #145 . I do believe that this adapter logic should be possible to be standardized among Grok, Anthropic, OpenAI, etc - just with minimal modification for each providers. This should be the intention of the proxy approach I suppose. At the fork, they are still separated having better clarity and easier development.

The main next step that I see, comes from the fact that no LLM so far that I have tested is able to do CUA that well. Particularly, I see it misclicks buttons a lot, it often misses some pixels a way. Almost there, but not quite there. So, I suppose this is the next step in making CUA actually useful.

[samkoesnadi]

@Uc207Pr4f57t9-251

I made a small change to support the passing of model name when using litellm and was able to get btyebot to work with qwen3-vl. It can open a browser, but fails on other tool calls due to:

#153

So it looks like there is some command mapping to do.

The command mapping is definitely the big thing to do. We have done it here https://github.com/kira-id/cua.kira . I think I have tagged you in another issue haha

[BuesrB]

Hi @BuesrB Your approach using OpenRouter in a Bytebot fork sounds quite interesting, especially since you mentioned Qwen3 VL seems well-suited for this type of agent work. I took a look at your GitHub profile hoping to find the fork you mentioned and learn more, but I wasn't able to locate it. Would you be willing to share a link to your fork if it's public? More specifically, I'm particularly curious about how you handled the Qwen VL tool calling incompatibility we discussed. Does OpenRouter handle this adaptation automatically for Qwen models, or did you need to implement specific adapter logic (e.g., parsing the content field) or perhaps use targeted prompt engineering within your fork to get the tool calls working reliably? Thanks!

The fork is here https://github.com/kira-id/cua.kira . We have improved it - it works now with Qwen3VL, takeover desktop directly on home, also added Blender on Desktop, merged some PR here that actually is useful and more. Could you please take a look and see if it works for you?

The adapter logic is modified accordingly continuing the PR #145 . I do believe that this adapter logic should be possible to be standardized among Grok, Anthropic, OpenAI, etc - just with minimal modification for each providers. This should be the intention of the proxy approach I suppose. At the fork, they are still separated having better clarity and easier development.

The main next step that I see, comes from the fact that no LLM so far that I have tested is able to do CUA that well. Particularly, I see it misclicks buttons a lot, it often misses some pixels a way. Almost there, but not quite there. So, I suppose this is the next step in making CUA actually useful.

Was just about to leave a comment to reply but thank you so much @samkoesnadi , and yes I totally agree with you. Particularly the misclicks and the incorrectness of the coordinates from the cursor are creating lots of errors. But yes, it is almost there.
So anyone who has interest and curious are welcomed to test, try out and leave a comment :) https://github.com/kira-id/cua.kira