New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Direct Line] Responses from our bot, back to the user via Direct Line takes 30 seconds to a minute to complete. #3787

Closed
willemodendaal opened this Issue Nov 16, 2017 · 9 comments

Comments

Projects
None yet
8 participants
@willemodendaal

willemodendaal commented Nov 16, 2017

Bot Info

  • App ID: 0883292d-3fe6-41c8-9fda-f2c1adad4592
  • SDK Platform: .NET
  • SDK Version: 4.6.2
  • Active Channels: Direct Line
  • Deployment Environment: Azure App Service

Issue Description

Calls from our bot back to the user (indirectly via DirectLine because that is the channel the user is using) take very long (30 seconds to a minute per message), causing the bot to throw an error after 15 seconds.

The problem started this morning, and impacts all our environments (including two production clients)

eg. Dependency calls as seen in appInsights:
2017-11-16T07:55:16.922 https://directline.botframework.com/v3/conversations/...convId.../activities/...activityid...
Duration: 50,591ms

Code Example

Any post to the user, for example:
await dc.PostAsync("Hello, welcome back");

Expected Behavior

These requests typically take less than 1 second. More than 5 seconds starts to be a problem. 50 seconds is a big problem.

Actual Results

Very slow responses from https://directline.botframework.com/v3

eg. Dependency calls as seen in appInsights:
2017-11-16T07:55:16.922 2017-11-16T07:55:16.922 https://directline.botframework.com/v3/conversations/...convId.../activities/...activityid...
Duration: 50,591ms

@tristanvandam

This comment has been minimized.

Show comment
Hide comment
@tristanvandam

tristanvandam Nov 16, 2017

@willemodendaal We are also experiencing this issue.
AppID_dev: b41c930f-bb27-44cc-aa48-4938d024b293
AppID_qa: f20b587f-3aeb-4b55-bf81-657057cb7aa9
Our bot info is similar to yours:
SDK Platform: .NET
SDK Version: 4.6.2
Active Channels: Direct Line
Deployment Environment: Azure App Service

Our bot is operating at incredibly slow speeds between posting messages to users and we continuously receive 502 errors when using directline with the specific message Retry message, could be 502 error from bot framework.
We noticed the issue about 5 hours ago and has been persisting since.
We can hardly progress past one or two messages without being bombarded with errors.
Any solution or quick fix on your side in order to get your bot stable or any other feedback received? Please let me know.

Appreciate it.
Tristan

tristanvandam commented Nov 16, 2017

@willemodendaal We are also experiencing this issue.
AppID_dev: b41c930f-bb27-44cc-aa48-4938d024b293
AppID_qa: f20b587f-3aeb-4b55-bf81-657057cb7aa9
Our bot info is similar to yours:
SDK Platform: .NET
SDK Version: 4.6.2
Active Channels: Direct Line
Deployment Environment: Azure App Service

Our bot is operating at incredibly slow speeds between posting messages to users and we continuously receive 502 errors when using directline with the specific message Retry message, could be 502 error from bot framework.
We noticed the issue about 5 hours ago and has been persisting since.
We can hardly progress past one or two messages without being bombarded with errors.
Any solution or quick fix on your side in order to get your bot stable or any other feedback received? Please let me know.

Appreciate it.
Tristan

@nwhitmont nwhitmont changed the title from Responses from our bot, back to the user via directline takes 30 seconds to a minute to complete. to [Direct Line] Responses from our bot, back to the user via Direct Line takes 30 seconds to a minute to complete. Nov 16, 2017

@nwhitmont

This comment has been minimized.

Show comment
Hide comment
@nwhitmont

nwhitmont Nov 16, 2017

Contributor

@willemodendaal @tristanvandam Thanks for reporting. I have escalated the issue to our platform team for investigation.

Contributor

nwhitmont commented Nov 16, 2017

@willemodendaal @tristanvandam Thanks for reporting. I have escalated the issue to our platform team for investigation.

@tonyanziano

This comment has been minimized.

Show comment
Hide comment
@tonyanziano

tonyanziano Nov 16, 2017

@tristanvandam Would you mind giving us your Bot name or App ID so we can investigate? Thanks.

tonyanziano commented Nov 16, 2017

@tristanvandam Would you mind giving us your Bot name or App ID so we can investigate? Thanks.

@tristanvandam

This comment has been minimized.

Show comment
Hide comment
@tristanvandam

tristanvandam Nov 17, 2017

@Fignewton5 @nwhitmont

AppID_dev: b41c930f-bb27-44cc-aa48-4938d024b293
AppID_qa: f20b587f-3aeb-4b55-bf81-657057cb7aa9

Thanks so much

tristanvandam commented Nov 17, 2017

@Fignewton5 @nwhitmont

AppID_dev: b41c930f-bb27-44cc-aa48-4938d024b293
AppID_qa: f20b587f-3aeb-4b55-bf81-657057cb7aa9

Thanks so much

@droidpl

This comment has been minimized.

Show comment
Hide comment
@droidpl

droidpl Nov 27, 2017

It is also happening here. The Bot ids are:
Dev: 9aca1667-63d6-440f-aa5a-8397df63e815

From time to time it triggers our timeout limits. I am also attaching an image with the spikes in response time we have in azure. (EU west)
screen shot 2017-11-27 at 5 19 59 pm

droidpl commented Nov 27, 2017

It is also happening here. The Bot ids are:
Dev: 9aca1667-63d6-440f-aa5a-8397df63e815

From time to time it triggers our timeout limits. I am also attaching an image with the spikes in response time we have in azure. (EU west)
screen shot 2017-11-27 at 5 19 59 pm

@droidpl

This comment has been minimized.

Show comment
Hide comment
@droidpl

droidpl Nov 27, 2017

Btw, I am using the Node SDK, so seems more something on the Directline service than the SDKs

droidpl commented Nov 27, 2017

Btw, I am using the Node SDK, so seems more something on the Directline service than the SDKs

@ederbond

This comment has been minimized.

Show comment
Hide comment
@ederbond

ederbond Dec 11, 2017

I'm having the same issues here with my bot build in .net and consuming via DirectLine-JS:
BotId: 0b0170f6-6a74-44a8-889d-2faa73ca62bb

ederbond commented Dec 11, 2017

I'm having the same issues here with my bot build in .net and consuming via DirectLine-JS:
BotId: 0b0170f6-6a74-44a8-889d-2faa73ca62bb

@EricDahlvang

This comment has been minimized.

Show comment
Hide comment
@EricDahlvang

EricDahlvang Jan 3, 2018

Collaborator

@willemodendaal @droidpl Are you still having these issues?

Have you implemented a custom state client in your bots? This blog post walks through adding Azure Table Storage or Document Db (now CosmosDb).

Collaborator

EricDahlvang commented Jan 3, 2018

@willemodendaal @droidpl Are you still having these issues?

Have you implemented a custom state client in your bots? This blog post walks through adding Azure Table Storage or Document Db (now CosmosDb).

@EricDahlvang EricDahlvang self-assigned this Jan 3, 2018

@droidpl

This comment has been minimized.

Show comment
Hide comment
@droidpl

droidpl Jan 3, 2018

Hey @EricDahlvang, thanks for replying. I didn't see the issue again lately. Yes I have the state implemented on document db and the bot migrated. Had some issues with 502 Bad gateway and 500 internal errors while in the migration but seems that it was an issue already fixed on Microsoft bot side. I will keep monitoring.

droidpl commented Jan 3, 2018

Hey @EricDahlvang, thanks for replying. I didn't see the issue again lately. Yes I have the state implemented on document db and the bot migrated. Had some issues with 502 Bad gateway and 500 internal errors while in the migration but seems that it was an issue already fixed on Microsoft bot side. I will keep monitoring.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment