Issue with Inputting Image Data in ChatGPT-vision Block

[chienyutseng]

Hi,

I am experiencing issues with inputting image data into the chatGPT-vision block. I've attempted two different methods to provide image data: directly using an image URL and uploading a file after downloading the image. Unfortunately, both methods failed to execute the model successfully.

Steps to Reproduce

The recipe is simply built with chat input -> chatGPT-vision -> chat output

1. Using an Image URL:

I provided the following image URL as input to the chat-vision block:

https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg

2. Uploading an Image File:
   I downloaded the above image and then uploaded it as a file to the chat-vision block.

In both scenarios, the chat-vision block failed to process the image data.

• upload with image file:

---

• upload with image URL:

Additional Information

I have verified that the image URL works correctly by testing it locally with the OpenAI Python package, ensuring that the issue is not with the image format or accessibility.
This issue seems specific to the chat-vision block's handling of image data inputs.

[manu-sapiens]

Thanks for investigating this with such details. I'm trying to repro but I'm getting [429] errors with all my requests to the vision API... but openAI is experiencing an outage today (Mar 6, 2024) on several end points, so it may not be related to what you were seeing! Will investigate once the outage is over.

[manu-sapiens]

Well, there was a masking issue with 429 errors (due to the switch from postpaid to prepaid).
However, the real issue is that openAI changed the API for the vision module and no longer works if the "logprobs" parameter is passed to it. I've fixed it and will be pushing a new version of Omnitool.ai with this fix soon. I'll then close this bug.

[chienyutseng]

Ok! Thanks for the reply, looking forward to the new version!

[manu-sapiens]

Omnitool 1.03 is live, so this should be fixed.