Read Time Out after Batch Testing

[deepakas]

Bug Details

Describe the bug
We are using the backend API to generate response for a set of test questions. When we run for high batches the application fails.

We ran the api for 100 questions with batch size of 10 and time delay of 20 seconds. It worked 2 times. Third time application crashed. API requests to batch endpoint is giving an empty response. Web app returns this error -
Error: HTTPSConnectionPool(host='infoasst-enrichmentweb-clc9v.azurewebsites.net', port=443): Read timed out. (read timeout=60)
Steps To Reproduce

1. Run Batch Testing on backend with 100 requests with a batch of 10 requests at a time.
2. After multiple attempts the application crashes.
3. Backend responds with empty response

What is the expected behavior? Return json response

Screenshots

ErrorsFromAppServiceLogs.txt

If applicable, add screenshots to help explain your problem.

Information Assistant details

Please provide the following details. You can simply include a screenshot of your Info panel as well.

GitHub branch: main

Version or Latest commit:
commit f4b3f3a (HEAD -> main, origin/main, origin/HEAD)
Merge: ce46332 a575426
Author: dayland 48474707+dayland@users.noreply.github.com
What region is your Azure Open AI Service in? US east

What ChatGPT model are you using? gpt-4

model name: gpt-4

model version: 0125-preview

What embeddings model are you using? [text-embedding-ada-002]

Additional context
Errors from App Service Logs :

If the bug is confirmed, would you be willing to submit a PR?

• Yes

[ArpitaisAn0maly]

Hi deepakas. How are you running batch of questions against IA? Are they not dependent on each other?( Conversational style) If you have set of questions and pairs for 100 questions would you please run this through promptflow as flow automates this for testing.

[deepakas]

Thanks @ArpitaisAn0maly. I am making an api call to the backend chat api to get bot response and submitting batches of 10 and having delay of 20 seconds with retries with doubling the delay time. Resolved the issue by scaling the apps and restarting. Thanks George for your help and reference documentation. https://github.com/microsoft/PubSec-Info-Assistant/blob/main/docs/deployment/considerations_production.md

[deepakas]

Resolved the issue with scaling and restarting the apps - https://github.com/microsoft/PubSec-Info-Assistant/blob/main/docs/deployment/considerations_production.md