test(backends): fix vision Ollama tests failing in CI with 400 model does not support vision (#1185)#1188
Merged
ajbozarth merged 2 commits intoJun 2, 2026
Conversation
…does not support vision
Structural payload tests (test_image_block_in_instruction, test_image_block_in_chat)
were failing in CI because the m_session fixture called start_session() with no
model_id, resolving to IBM_GRANITE_4_1_3B (granite4.1:3b) — a text-only model.
Attaching images caused Ollama to reject the request with 400, preventing
post_processing from running and the structural assertions from ever executing.
Fixes are:
1. Add test/backends/conftest.py with a shared mock_ollama_backend fixture that
constructs an OllamaModelBackend entirely offline (patches _check_ollama_server,
_pull_ollama_model, ollama.Client, ollama.AsyncClient). No live server required.
2. Rewrite test_vision_ollama.py into three tiers:
- Tier 1 (construction): pure ImageBlock unit tests, no model or server.
- Tier 2 (structural payload): mocked offline tests that verify images are
embedded correctly in the Ollama conversation payload. The _async_client
property is mocked via PropertyMock at the class level so the mock is
returned regardless of which event loop _run_async_in_thread creates in
its background thread. Runs in CI unconditionally.
- Tier 3 (dormant e2e): skipped until granite-vision-4.1 lands on Ollama;
tracked in generative-computing#1187.
3. Refactor test_ollama_unit.py to use the shared mock_ollama_backend fixture,
removing the duplicated _make_backend() helper.
Closes generative-computing#1185.
Assisted-by: Claude Code
Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
planetf1
force-pushed
the
worktree-issue-1185
branch
from
0a5722f to
862182c
Compare
Remove dead code after unconditional pytest.skip() in vision_session — the availability check is now the sole gate, which auto-activates once granite-vision-4.1 lands on Ollama. Update conftest.py docstring to show the correct PropertyMock class-level patching pattern (instance assignment does not override the event-loop-keyed _async_client property). Compute ImageBlock.from_pil_image() once in test_image_block_in_chat instead of three times. Assisted-by: Claude Code Signed-off-by: Nigel Jones <jonesn@uk.ibm.com>
jakelorocco
approved these changes
Jun 2, 2026
8 tasks
ajbozarth
approved these changes
Jun 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two vision tests —
test_image_block_in_instructionandtest_image_block_in_chat— have been silently broken in CI. The fixture usedstart_session("ollama")with no model, which defaulted togranite4.1:3b. Sending an image to a text-only model gets a400 model does not support vision, which meanspost_processingnever ran and the image-embedding assertions were never reached. The tests appeared to pass (or skip gracefully) while doing nothing useful.The deeper issue: there was no way to construct an
OllamaModelBackendwithout a live server. This PR fixes that, then fixes the tests on top of it.What changed
test/backends/conftest.py(new) — a sharedmock_ollama_backendfixture that patches all four server-touching points in__init__and returns a fully constructed backend offline. Other tests can now reuse this instead of rolling their own patch blocks.test/backends/test_vision_ollama.py(rewrite) — three tiers:ImageBlocklogic, no backend. Module-levele2e/ollamamarkers removed; runs as a plain unit test.images=[...]lands correctly in the Ollama message dict. Uses the offline fixture with_async_clientpatched viaPropertyMockat the class level — required because the property is event-loop-keyed and a simple instance attribute is bypassed when_run_async_in_threadspins up a background thread. Runs in CI with no server, no vision model.granite-vision-4.1, skipped until the model appears in the Ollama library. The skip gate clears automatically onceollama pull granite-vision-4.1works. Activation checklist in test(vision): activate Ollama vision e2e once granite-vision-4.1 lands on Ollama #1187.test/backends/test_ollama_unit.py(refactor) — swapped the private_make_backend()helper for the shared fixture. No behaviour change.Notes
Closes #1185. Dormant e2e tracked in #1187.
test/backends/test_vision_openai.pyhas the same structural gap (assertions hidden behindqualitative+xfail). Worth a follow-up.Testing
CI: quality matrix green on 3.11, 3.12, 3.13.