Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OpenAI and Azure vision support #18

Merged
merged 2 commits into from
Mar 28, 2024

Conversation

moritzfl
Copy link
Contributor

@moritzfl moritzfl commented Mar 26, 2024

This is pretty similar to "Claude vision support" #17

Similar to the Claude PR, this does break API compatibility due to polymorphism in message contents (which can now be either image or text).

API reference: https://platform.openai.com/docs/guides/vision

// Vision will only work with OpenAIChatCompletionModel.GPT_4_VISION_PREVIEW currently

List<OpenAIMessageContent> content = new ArrayList<>();

content.add(new OpenAIMessageImageURLContent(new OpenAIImageUrl("https://picsum.photos/200")));
// Or use a local image: new OpenAIImageUrl("image/jpeg", Files.readAllBytes(Paths.get("/path/to/local/image.jpg")))

content.add(new OpenAIMessageTextContent("What can you see in this picture?"));

OpenAIChatCompletionMessage message = new OpenAIChatCompletionMessage("user", content);

moritz added 2 commits March 26, 2024 21:38
@JsonInclude(JsonInclude.Include.NON_NULL) for OpenAIChatCompletionRequest is required because the vision model does not support "tools" and fails if the attribute is present.
@carlrobertoh
Copy link
Owner

Awesome! Have you already started with the UI? If not, I could take a look sometime next week, if not sooner.

@moritzfl
Copy link
Contributor Author

No - I have not done any work on the UI yet. I could look into it but I am busy during the easter holiday so it might take bit longer. Feel free to work on it.

Btw. after a meeting with the managing director we just informed all developers and technical writers at the company that I work for that they can from now on use CodeGPT with a dedicated API-Key provided through the company.

@carlrobertoh
Copy link
Owner

No - I have not done any work on the UI yet. I could look into it but I am busy during the easter holiday so it might take bit longer. Feel free to work on it.

Sure, I'll take a look.

Btw. after a meeting with the managing director we just informed all developers and technical writers at the company that I work for that they can from now on use CodeGPT with a dedicated API-Key provided through the company.

That's nice to hear! ❤️

@carlrobertoh carlrobertoh merged commit 487d7dd into carlrobertoh:master Mar 28, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants