Resume conversation

[billba]

Per microsoft/BotFramework-WebChat#353, it would be useful to resume a conversation in progress. Let's talk about how this could happen.

First of all we'd need the conversationId of the previous conversation. @dandriscoll can we depend on this remaining valid?

Instead of calling /conversation we call /conversations/{conversationId}, which would give us a current streamUrl. I guess if this call fails we know the conversationId is no longer valid.

We could also optionally supply a watermark. If we don't we'd get all the cached messages Direct Line might still be holding, which would sometimes be desirable.

Then business would resume as normal.

From an API perspective we'd just need to add conversationId?: string and watermark?: string to DirectLineOptions.

You might ask, how do we get the conversationId/watermark from the previous conversation? Well it's inside the DirectLine object. So the hosting page could create a DirectLine object, e.g.

const dl = new DirectLine({
    secret: your_direct_line_secret // or token: your_direct_line_token
});

and then save dl.conversationId and dl.watermark to a cookie, or local storage, or its server, etc. Then when reconstituting it, it would do:

const dl = new DirectLine({
    secret: your_direct_line_secret, // or token: your_direct_line_token
    conversationId: from_previous_conversation,
    watermark: from_previous_converation
})

Am I missing anything?

[gilesbutler]

This is pretty much how I do it @billba 👍 I store the token and conversationId in localStorage then grab them when the page reloads and connect with. If the token has expired then I just request a new one and connect with it as well as the old conversationId.

The problem I'm having is when the connection changes to ConnectionStatus.ExpiredToken. I'm refreshing the token using the Reconnect API then passing the result to directLine.reconnect(data) but nothing happens. Is the ConnectionStatus supposed to change to Online?

[billba]

Yes it is supposed to change back to Online. Have you debugged reconnect() to see what's going on?

[baconstrudel]

@gilesbutler which token expires? The conversation token or the DirectLine token?

[gilesbutler]

Thanks for clarifying @billba - lookat reconnect() I see...

DirectLine.prototype.reconnect = function (conversation) {
        this.token = conversation.token; // this is being set correctly
        this.streamUrl = conversation.streamUrl; // this is being set correctly
        if (this.connectionStatus$.getValue() === ConnectionStatus.ExpiredToken)
            // connectionStatus$.getValue() = 4
            // ConnectionStatus.ExpiredToken = 3
            this.connectionStatus$.next(ConnectionStatus.Online);
    };

That doesn't seem right to me 🤔 if the connectionStatus value is supposed to equal ConnectionStatus.ExpiredToken why would the connectionStatus then be set to ConnectionStatus.Online?

@TheTobias Sorry for my ignorance I don't fully understand, I presume the conversation token. I'm connecting to DirectLine on my server and that's using a secret rather than a token. It returns a token though for the client. Is that what you were referring too?

[billba]

@gilesbutler reconnect() is called when the token is expired, to give DirectLineJS a new token. So, once the token has been set, we set the new status to Online. But we only do this if the current status is still ExpiredToken because we don't want to change the status from e.g. ConversationEnded. Now that I look at it, the whole function should really be wrapped in that if.

Just to be clear, this function won't be called in this proposed resume functionality. The two cases are similar but not identical, which may be the cause of your issue, since in your example the current status is 4 i.e. FailedToConnect.

[gilesbutler]

Ah ok thanks @billba, I think I misread the documentation on how to reconnect to a conversation...

Your app can request a new token from its server, which should call the Reconnect API. The resultant Conversation object can then be passed by the app to DirectLine, which will
var conversation = /* a Conversation object obtained from your app's server */; directLine.reconnect(conversation);

So I don't have to call directLine.reconnect() directly, I just have to request a new token from my server and pass it to directLine, something like (sudo code)...

    // refreshToken calls the server and gets a new token from /v3/directline/conversations/conversationId?watermark=watermark
    refreshToken(directLine.watermark)
        .then(data => {
              // Save the data to localStorage
              return setLocalStorageItems(data);
        })
        .then(data => {
            // ??? 
        }

What's the recommended way to pass the results to directLine?

[billba]

@gilesbutler I'm a little lost. What scenario are we talking about here? Resuming a previous conversation (this issue) or dealing with an expired token?

[baconstrudel]

just to be sure would the proposed solution at the beginning also allow to join a conversation by passing the same conversation token and then sending messages from a different username?

[billba]

@TheTobias Yes. The only place user identity exists in the current system is when sending message.

[billba]

Thinking about it a little more, @TheTobias has made a good point. This exact same approach would allow multiple users to join the same conversation, so long as they all have access to the same conversationId and a valid secret/token.

[dandriscoll]

A few questions/observations in order:

• Are conversation IDs valid forever? The ID doesn't have a lifetime (it's just an ID), but the contents of the conversation slowly expire, and the conversation becomes unreachable within Direct Line after a few weeks of inactivity.
• You should indeed retain and replay the watermark. If you include it in the reconnect request, any replay through the socket is guaranteed to start at the watermark, even if you wait a little while to connect.

The approach is sound, and likely what I would have come up with on my own. :)

[gilesbutler]

Sorry @billba you're right, I've gone off topic and mixed the two scenarios. I won't add anymore so I don't confuse the rest of the thread.

[billba]

@gilesbutler feel free to open a new issue if you're still having problems

[baconstrudel]

@dandriscoll, If you rejoin a conversation (after you have closed the tab or something) and you don't include a watermark do you get the whole conversation?

[billba]

@TheTobias Direct Line holds a buffer of activities to help clients bridge temporary connection issues. Without a watermark you'll get whatever buffer it's got, but you shouldn't count on always getting the whole conversation.

[baconstrudel]

Ok

[dandriscoll]

Hi @TheTobias, when you issue the reconnect call, we calculate the watermark at that moment. You have 60 seconds to connect to the stream, and if you connect during this time, we replay everything from that watermark that you may have missed.

This means that this is a safe algorithm to reconnect and retrieve all messages:

1. Call reconnect
2. Connect to the socket
3. If you want messages from the beginning of the conversation, start a GET loop to replay any activities from the beginning of the conversation. You may receive duplicates.

Steps 2 and 3 may be performed in either order.

[billba]

For the record what Dan describes is not functionality that this library currently supports.

[billba]

@dandriscoll you said "This means that this is a safe algorithm to reconnect and retrieve all messages" but my understanding is that Direct Line will only keep messages for a certain amount of time, true? So in an extended resume scenario - which this proposal is meant to implement - there's no guarantee that Direct Line will still have all or even any messages. Correct?

But the point is taken for this scenario that we might want to implement a GET loop if no watermark is provided.

[dandriscoll]

:) Yes, it's correct that this will only retrieve messages which are still on the service. And also yes, I'm speaking in protocol terms, not library terms.

[meulta]

this PR adds support for resuming.

Only thing missing for now is the support of watermarks while using websockets. I am working with @dandriscoll on that.

[nkpatterson]

Is there any progress or a rough ETA on supporting resume/watermarks with WebSockets?

[meulta]

ping @dandriscoll

[dandriscoll]

Hi @meulta, can you be more specific about what you're waiting on? Watermarks have been supported in reconnect from the day we added WebSocket support.

[nkpatterson]

Hey @dandriscoll, you're referring to support in the protocol and not this library though, correct? I'm looking for an update on support for it in the library.

[nicksav]

Have that code in botchat.js. Means watermarks not supported yet?

if (options.conversationId) { this.conversationId = options.conversationId; } if (options.watermark) { if (this.webSocket) console.warn("Watermark was ignored: it is not supported using websockets at the moment"); else this.watermark = options.watermark; }

[nkpatterson]

Bumping this. Is there an ETA on supporting resume/watermarks with WebSockets in the DirectLineJS library? Do we have an idea on what's involved to add support?

[vicenschamorro]

Is there any intention to add some extension point for the cache to connect an external persistence layer like CosmoDB to extend the time / size of it with all the history of the conversations? I understand that the cache has a specific objective but it could be very interesting to add this possibility so who wants more time because they have this necessity adding a database but without the necessity to manage it in parallel of DL.

[zubcomandante]

@dandriscoll I agree with @vicenschamorro : delegating message caching to an external DB would be welcome and make our applications more reliable.
In general, is it possible to have an estimate of how long messages are supposed to be retained in DL's cache? Will this amount of time be affected by the migration to the Azure Bot Service?

[baki32]

Hi, is there any update here?
I've had some exchange with Vince in this thread:
microsoft/BotFramework-WebChat#850

that ended watermark support belongs to DirectLineJS.

[rafaelmaeuer]

Push this, I am looking for a solution to use watermarks with websockets too. It is still not possible to get chat history using websockets.

[sevetseh28]

conversationId inside DirectLine object is private. How do you save it ? Are you editing the compiled .js or the Chat.tsx source code?

[baki32]

You can get it from directline object

var **directLine** = new DirectLine({
    secret: /* put your Direct Line secret here */,
    token: /* or put your Direct Line token here (supply secret OR token, not both) */,
    domain: /* optional: if you are not using the default Direct Line endpoint, e.g. if you are using a region-specific endpoint, put its full URL here */
    webSocket: /* optional: false if you want to use polling GET to receive messages. Defaults to true (use WebSocket). */,
    pollingInterval: /* optional: set polling interval in milliseconds. Default to 1000 */,
});

You then pass it to webchat

BotChat.App({
    botConnection: **directLine**,
    user: user
    ...
}, document.getElementById("BotChatGoesHere"));

and then when connection is established

**directLine**.connectionStatus$
.subscribe(connectionStatus => {
    switch(connectionStatus) {
        case 2:{
             //conversationId
             var id = **directLine**.conversationId
        }       
});

[sevetseh28]

Oh, right! That is in the html file! Thank you! I was trying to edit the Chat.tsx code.

[kosset]

@baki32 @sevetseh28 I still cannot get it. Do you put the lines above into the same script?

[baki32]

That code is used when initializing webchat object in HTML refer to : Advanced Web Chat chapter here https://github.com/Microsoft/BotFramework-WebChat/blob/master/README.md

…

Sent from my Windows 10 device

________________________________ From: Kostas Setzas <notifications@github.com> Sent: Wednesday, October 17, 2018 5:03:20 PM To: Microsoft/BotFramework-DirectLineJS Cc: baki32; Mention Subject: Re: [Microsoft/BotFramework-DirectLineJS] Resume conversation (#9) @baki32<https://github.com/baki32> @sevetseh28<https://github.com/sevetseh28> I still cannot get it. Do you put the lines above into the same script? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#9 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AVyIloHj_Q5a7GEVh4WHFsr-4s65oGx5ks5ul0aegaJpZM4MPwXk>.

[zubcomandante]

Hi all,

any answer to my old question: "is it possible to have an estimate of how long messages are supposed to be retained in DirectLine cache?"

By the way, I found no API to forcibly erase a cached conversation (https://docs.microsoft.com/en-us/azure/bot-service/rest-api/bot-framework-rest-direct-line-3-0-api-reference?view=azure-bot-service-4.0#conversation-operations).
Of course this has impacts on companies and/or customers across the EU, due to GDPR regulation; the document you provide to tackle this issue (https://blog.botframework.com/2018/04/23/general-data-protection-regulation-gdpr/ ) claims:

"Conversation data – data that indicates a user interacted with your bot (excluding any content of the conversation itself)."

and

"it is important to reiterate that the channels can only store information about user participation in conversations, and not the actual message activity of the conversation itself. "

So, what about the actual conversations stored in the DL cache?

Many thanks.

[compulim]

We need to revisit resume/end conversation stories and it is being tracked at #124.

@zubcomandante we have answered this GDPR question in #119. Thanks!

[raghukrishnamoorthy]

So this is the code I have on my page

`

<script src="https://cdn.botframework.com/botframework-webchat/latest/webchat.js"></script> <script>

  const directL = window.WebChat.createDirectLine(
    {
      secret: 'mysecret',
      conversationId: 'convID'
    }
  )
  

  window.WebChat.renderWebChat({
    directLine: directL,
    userID: 'test'
  }, document.getElementById('webchat'));

</script>

`

The ConvID is the conversation ID i got from another page. I've created this page now with the existing convID for directline however the created object has a different conversation ID. What am i doing wrong?

what gives?

[compulim]

Tracking this issue on #124 (uber-bug on "resume conversation story").