Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] GetChatCompletionsStreamAsyncSample code not working #40710

Closed
3 tasks done
ventaglio opened this issue Jun 19, 2024 · 3 comments · Fixed by #41102
Closed
3 tasks done

[BUG] GetChatCompletionsStreamAsyncSample code not working #40710

ventaglio opened this issue Jun 19, 2024 · 3 comments · Fixed by #41102
Assignees
Labels
Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. needs-team-triage Workflow: This issue needs the team to triage. OpenAI question The issue doesn't require a change to the product in order to be resolved. Most issues start as that

Comments

@ventaglio
Copy link

Describe the bug
I am not able to get to a working asynchronous chat stream example using GPT-4 through Azure SDK for Java.
I started from the example code provided here
https://github.com/Azure/azure-sdk-for-java/blob/azure-ai-openai_1.0.0-beta.9/sdk/openai/azure-ai-openai/src/samples/java/com/azure/ai/openai/usage/GetChatCompletionsStreamAsyncSample.java
but it does not work out of the box.
When I executed that code, providing the necessary parameters, I got an error.

Exception or Stack Trace
There was an error getting chat completions.java.lang.NullPointerException: The mapper [it.celi.dialog.GetChat$$Lambda$143/0x00000008002a0000] returned a null value.

To Reproduce
Just execute the example code provided here:
https://github.com/Azure/azure-sdk-for-java/blob/azure-ai-openai_1.0.0-beta.9/sdk/openai/azure-ai-openai/src/samples/java/com/azure/ai/openai/usage/GetChatCompletionsStreamAsyncSample.java

Code Snippet
I tried to fix the issue by modifying these lines:
if (CoreUtils.isNullOrEmpty(chatCompletions.getChoices())) { return null; }
this way
if (CoreUtils.isNullOrEmpty(chatCompletions.getChoices())) { return ""; }

This change allowed me to execute the code without NPE errors, but the final behavior is different from what I expected it to be.

Expected behavior
I expected to get the response token by token in an asynchronous way, but I instead received only
Role = assistant

Then I increased the timeout from 10 to 60 seconds
TimeUnit.SECONDS.sleep(60);

This way I got:
Role = assistant immediately
... the full response after more or less 15 seconds.
So the streaming is not working in an asynchronous way.

Screenshots

Setup (please complete the following information):

  • OS: macOS Sonoma 14.4.1
  • IDE: IntelliJ IDEA 2024.1.3
  • Library/Libraries: com.azure:azure-ai-openai:1.0.0-beta.9
  • Java version: 17.0.9 Temurin
  • App Server/Environment: -
  • Frameworks: -

Additional context

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

  • Bug Description Added
  • Repro Steps Added
  • Setup information Added
@github-actions github-actions bot added Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. needs-team-triage Workflow: This issue needs the team to triage. OpenAI question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels Jun 19, 2024
@ventaglio
Copy link
Author

Update: I managed to get the asynchronous streaming working by using another deployment even if I have not yet understood what is the difference between the two deployments.
Is there something I can check to understand why one is not working in asynchronous mode and the other it is, using the same exact Java code?

Anyway the example code contains at least a bug in the return null line.

@ventaglio
Copy link
Author

Update: we finally understood what was wrong with the first deployment. It had a custom filter that was not configured in "async mode" so it was blocking the asynchronous response.
I leave the issue open just to fix the return null line error.

@mssfang
Copy link
Member

mssfang commented Jul 10, 2024

Thank you for reporting this issue. Will be fixed soon

@github-actions github-actions bot locked and limited conversation to collaborators Oct 9, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. needs-team-triage Workflow: This issue needs the team to triage. OpenAI question The issue doesn't require a change to the product in order to be resolved. Most issues start as that
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants