🤖 ci: improve integration test reliability #1085
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ammar-agent
force-pushed
the
ai-tests-6168
branch
2 times, most recently
from
c5d8be8 to
297563e
Compare
- Use ImageMagick-generated 4x4 pure red (#FF0000) PNG for reliable color detection - Update prompt to explicitly describe solid-color image and request just color name - Tighten assertion to strictly match 'red' instead of accepting wrong colors _Generated with mux_
ammar-agent
force-pushed
the
ai-tests-6168
branch
from
297563e to
5a75dcd
Compare
- Add explicit tool name requirements (MUST use chrome_navigate_page, chrome_take_screenshot) - Add diagnostic logging when screenshot tool call is missing - More directive prompts reduce model non-determinism _Generated with mux_
- Log result object when sendMessage fails to understand CI failures - Will help diagnose why API calls are returning success=false _Generated with mux_
- Previous 1-bit indexed/palette PNGs may not be properly processed by vision APIs - New 4x4 RGB PNGs with explicit color-type=2 ensure proper 8-bit per channel encoding - Affects both RED_PIXEL and BLUE_PIXEL test fixtures _Generated with mux_
ammar-agent
changed the title
ammario
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Fix flaky integration tests for image handling and MCP screenshot functionality.
Changes
Image Test Fix (
sendMessage.images.test.ts)-define png:color-type=2for proper 8-bit per channel RGB encoding.sendMessagefails to help diagnose future CI failuresMCP Screenshot Test Fix (
mcpConfig.test.ts)chrome_navigate_page,chrome_take_screenshot)Root Cause Analysis
The image test was failing because:
success=false(transient API issue)deltas.length === 0)Investigation revealed the PNG images were using 1-bit indexed colormap format instead of proper RGB, which may cause issues with vision API processing.
Generated with
mux