open-webui: use a direct color question for the vision smoke - #682
Conversation
The vision smoke sent a 1x1 pixel with max_tokens=256; the reasoning model spent the whole budget deliberating over the degenerate image (finish_reason=length) and returned empty content. Use a small real 8x8 PNG and raise max_tokens to 512. Applies to both Windows device variants and Linux. Refs #669, #670, #673.
There was a problem hiding this comment.
Pull request overview
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Improves reliability of the Open WebUI vision smoke test by avoiding a degenerate 1×1 image that can consume the model’s reasoning token budget without producing visible content.
Changes:
- Replace the 1×1 base64 PNG with a small 8×8 PNG in PowerShell and Python examples.
- Increase
max_tokensfrom 256 to 512 for the vision request across Windows and Linux variants.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
I don’t think this PR actually resolves the issue. Even with your changes, the tests are still failing. I believe that the cause of the failure is as explained in this comment (not 100% definitive though): #460 (comment) I updated the three failing runners (Linux Halo, Linux STX, Linux R9700) based on that comment: lemonade backends install llamacpp:rocm
lemonade config set llamacpp.backend=rocm
lemonade backends install sdcpp:vulkan
lemonade config set sdcpp.backend=vulkanSome useful commands for checking the current state:
Passing workflow runs after I made the above changes:
However, the machine
|
512 still occasionally overran on stx/halo when the reasoning model spiraled; 1024 gives headroom (good-state runs conclude in ~70-100 tokens). Keeps the 8x8 image fix. Refs #670.
The 'if you can see an image input, reply OK' prompt made the reasoning model philosophize about its own vision capability and non-deterministically spiral past the token limit (empty content, finish=length) on stx/halo. Ask 'What color is this image?' and assert 'red' instead. Revert the 1024 stopgap to 512. Refs #670.
lucbruni-amd
changed the title
|
Awesome, thanks for the fix @lucbruni-amd. This was a great catch! |
The
openwebui-lemonade-multimodal-smokevision request asked "if you can see an image input, reply OK". That framing makes the reasoning model debate its own vision capability and non-deterministically spiral past the token limit (finish=length, emptycontent) — the failure on stx/halo. Raising tokens only masked it (halo still overran at 1024).Fix: ask a direct question the model answers by actually looking — "What color is this image? Reply with only the color name." — and assert
red. Keeps the 8x8 image; reverts the 1024 stopgap to 512.Validated on the exact boxes that failed CI: 5/5 deterministic "Red" on stxp-02 and halo-02.
Closes #670.