Allow passing data from the skill back to the VA

[marawanshalaby]

User Story

As a developer, I would like to be able to pass data back from the skill to the VA.

I would like to be able to pass data back from the skill back to the invoking VA when the skill is complete, in order (for example) to continue the conversation with another bot, with the information generated by the first bot.

My use case is the following:

I have the VA connected to 2 bots, the first bot is used to search for people and the 2nd bot is used to book meetings with those people. One of the functionalities of the search people bot is to enable the user to create a meeting with a searched person. That would require the search people bot to return a reference to the person with whom the meeting will be created, back to the VA. That is currently not possible as far as I can tell, because all that the skill returns to the VA on completion is a boolean value.

It would be great if something like the manifest/SkillContext/slots mechanism is available from skill -> VA instead of just for VA -> skill

[darrenj]

Hi,

We have been working on "action invocation" support for Skills and as part of this formalising how data can be returned back to the calling Bot. Activity has a SemanticAction property which is positioned to enable this scenario.

We have a concept to prove this out in a PR at the moment. Here is an an example of setting this SemanticAction property which you can then pull out on the client side.

Let me know if this helps, happy to help further and get you moving with this scenario which is something we'll be adding more documentation/support for in the next release aligned to the Ignite conference..

[marawanshalaby]

Hey Darren

Thank you for the quick response. Yes, the SemanticAction response would be a great solution for my use case. Pulling the semantic action out from the client (VA) side however would require some changes to the code base there too as far as I can tell, right? Because currently, even if you call sc.EndDialogAsync(semanticAction) on the skill side, the result of this call gets translated by the skillTransport on the VA side to just a boolean true value, indicating the skill is done and nothing else.

[darrenj]

Fair point - in your scenario do you have a desire to "invoke" the skill directly kind of like a remote method call or instead pass an utterance across, let the skill do it's thing but also be able to return data back along with any activity that (currently) gets forwarded to the user without you being able to intercept it.

The action invocation approach we have a pattern for, almost ready for use and the other feels like something we explore as you suggest if that's the scenario your looking at.

[darrenj]

Any thoughts @marawanshalaby ?

[marawanshalaby]

Hey Darren

Sorry for the late reply, been buried by work lately :). So, for my case, I would say that the second option would be more appealing because it's more loosely coupled, the ideal scenario for me would be as follows:

• User sends an utterance, the VA dispatcher sends him to Skill A.
• Skill A handles the user's actions, until the "Create Meeting" intent is identified by Skill A
• Once the "Create meeting" intent is identified by Skill A, it terminates the conversation and sends its local state (Person name and id), possibly through a semantic action, back to the VA, along with an utterance.
• The VA resolves the utterance to Skill B, then passes the semantic action originally from Skill A to Skill B using its slots

This way Skill A knows nothing about Skill B and the only connection between them happens through an utterance handled by the dispatcher, instead of hardcoding the remote method call in the VA/Skill A, so it's also easily extendable to other skills. Do you think this makes sense?

[darrenj]

Yes, totally and SemanticAction as we have today would enable this. In fact this sample shows invocation of three skills to aggregate data together into one daily summary card. It doesn't show the passing data back to then pass data back on to another skill but we've demonstrated that in a few scenarios most notably an experience to retrieve your tasks, triage the types and then offer to route you via a store to resolve (e.g. grocery, pharmacy).

SemanticAction provides the mechanism to perform this exchange, we'll be adding some further documentation in the next few weeks to outline this pattern in a little more detail

[marawanshalaby]

Hey Darren

That looks great, two questions from my side:

• do you have an approximate date when the SemanticAction feature will be released?
• will the typescript support be added at the same time as the C# support?

[Kumar3010]

Hey Darren,

we are just going through the sample for invoking the skills to aggregate the data but didnt see data being passed from skill to VA, i know you have mentioned it in your comment. but would you mind letting us know if there is any mechanism to pass the data from skill to vA. thanks you

[Kumar3010]

@darrenj could you please provide an update to the above question

[darrenj]

Hi

Here is an example of setting the semanticaction within a skill that returns information back to the calling Bot.

[Kumar3010]

we have tried this in the skill but how do we get the data again in the Virtual assistant. couldnt got any sample. appreciate if you can share the vA code tool

[Kumar3010]

@darrenj could you please let us know the VA code too

[marawanshalaby]

@darrenj, I am now trying to implement this, I see that it has been fixed on the VA side in the C# version

botframework-solutions/lib/csharp/microsoft.bot.builder.skills/Microsoft.Bot.Builder.Skills/SkillDialog.cs

Line 326 in 4619333

return await innerDc.EndDialogAsync(handoffActivity.SemanticAction?.Entities);

so any semantic action entities set by the skill are returned, but that's not the case in typescript

botframework-solutions/lib/typescript/botbuilder-skills/src/skillDialog.ts

Line 244 in 4619333

return await innerDc.endDialog();

. Do you have an estimate for when it will be added in typescript too?

This is a blocker for us because in the current flow we are implementing we need to make sure that the dialog ended successfully on the skill side (not a cancel for example) in order to trigger a feedback dialog on the VA side. Any suggestions for how to implement this scenario while we are waiting for TS support?

[marawanshalaby]

@darrenj any update on when this will be fixed in typescript https://github.com/microsoft/botframework-solutions/blob/master/sdk/typescript/libraries/botbuilder-skills/src/skillDialog.ts#L382 ? Is this related to this issue: #2489 ?

[higoorc]

Hello, @darrenj

Can you share how to handle the data again in VA? You shared the sample in CalendarSummaryDialog, but it's just the skill side. How to handle that in VA after the skill send in EndDialogAsync?

Thanks for that

[darrenj]

Issue was closed hence we weren't getting notifications, re-opening. Sorry for the delay, can I just clarify the scenario here?

• VA and SKill is in Javascript/Typescript rather than C#?
• You wish to return "data" back to a parent VA from a child Skill as part of the "End Dialog" pattern?

[higoorc]

Thanks for that.

My question still how to send skill context back to the VA (both in C#). I wanted to do something similar to what @marawanshalaby talked before. "The VA resolves the utterance to Skill B, then passes the semantic action originally from Skill A to Skill B using its slots"

How the best approach to do that? Looking to what was discussed here I still don't know how to handle a semantic action in VA side.

[darrenj]

Hi,

We need a tutorial on this :-) I'll get this added as part of our GA work (end of month) and ideally on the doc site in the next few days. In the meantime hopefully the information below helps.

As part of the Skills GA we simplified data sharing and moved the transfer method from SemanticAction (which was a pain to use due to an old schema) and towards the easier Value property on an activity.

I'll get a code sample too and update this issue.

To pass data to a Skill

To invoke a Skill, an Activity of type Event is sent to the Skill thus triggering Action style invocation rather than the usual utterance.

Here is an example of putting data onto an Activity your going to send

public class BookingDetails
{
        public string Destination { get; set; }
        public string Origin { get; set; }
        public string TravelDate { get; set; }
}

activity.Value = new BookingDetails { Destination = "New York" };

Retrieving data within a Skill

Here and below is an example of a Skill being passed information from a parent (e.g. Virtual Assistant or Power Virtual Agents). In this situation the "data" is passed on the Value property of an incoming Activity. In this situation "data" is passed into a Skill (via an Event Activity) and is then deserialized and put into state - meaning when the dialog is executed it has all the data it needs.

case Events.CreateEvent:
{
    state.IsAction = true;
    EventInfo actionData = null;
    if (ev.Value is JObject info)
    {
        actionData = info.ToObject<EventInfo>();
        // Process data here
    }

    return await stepContext.BeginDialogAsync(_createEventDialog.Id, options);

Passing data back

Within the Dialog (inside your skill) when you finish processing and want to return information back you can do this through an overload on the EndDialogAsync. This data is then marshalled back to the caller as part of the EndOfConversation message that the Skill infrastructure sends.

    return await sc.EndDialogAsync(new TodoListInfo { ActionSuccess = true, ToDoList = todoList });

Getting data on the caller side

On the caller side, to better support this scenario we had to pivot MainDialog to be a Waterfall rather than a ComponentDialog. Component Dialogs didn't provide a way to "get" the result of a Dialog.

As you see here you can reach up and get the result of the skill and process accordingly.

if (stepContext.Result != null)
{
  var message = $"Skill \"{activeSkill.Id}\" invocation complete.";
  message += $" Result: {JsonConvert.SerializeObject(stepContext.Result)}";
}

[darrenj]

I've validated an approach - getting this reviewed by my team before I commit to docs but got this working pretty easily using the Weather Skill as an example.

This assume you have a Skill registered called WeatherSkill - will work with any other skill but need to ensure it's configured.

  var skillSteps = new WaterfallStep[]
            {
                PreSkillStepAsync,
                PostSkillStepAsync,
            };

  AddDialog(new WaterfallDialog("WeatherActionInvoke", skillSteps));

Start the Action Invocation

 return await innerDc.BeginDialogAsync("WeatherActionInvoke");

Skill Action Invocation

stepContext.Result in PostSkillStepAsync will have the return value from the Skill

        private async Task<DialogTurnResult> PreSkillStepAsync(WaterfallStepContext stepContext, CancellationToken cancellationToken)
        {
            var evt = stepContext.Context.Activity.AsEventActivity();
            if (evt != null)
            {
                LocationInfo location = new LocationInfo();
                location.Location = "51.4644018,-2.1177246,14";

                var activity = (Activity)Activity.CreateEventActivity();
                activity.Name = "WeatherForcast";
                activity.Value = location;

                // Create the BeginSkillDialogOptions
                var skillDialogArgs = new BeginSkillDialogOptions { Activity = activity };

                // Start the skillDialog instance with the arguments. 
                return await stepContext.BeginDialogAsync("WeatherSkill", skillDialogArgs, cancellationToken);
            }

            return await stepContext.NextAsync();
        }

        private async Task<DialogTurnResult> PostSkillStepAsync(WaterfallStepContext stepContext, CancellationToken cancellationToken)
        {
            return await stepContext.NextAsync();
        }

[lsoliveira]

Regarding @higoorc question, I suppose the proposed implementation covers the requirements and is doable.

However, there are skill handoff scenarios that would probably benefit from additonal controls in place. Value is a very generic container that VA implementers can leverage to implement any scenario they see fit but does not enforce any particular of behaviour. In a scenario where implementers connect their VA implementations to both 1st party (developed in-house) and 3rd party skills (community skills), it's often not possible to guarantee that Value would be a generic mechanism for skill handoff.

By skill handoff I mean when a skill is designed in a way that its dialog(s) end in such a way that it would be natural to continue the conversation with another skill. Take for example a scenario about booking a trip and, after booking, offer to handoff to a skill that also books rental cars or local tours for that particular destination. I would even imagine skill handoff would not be the only possible thing a skill might want to report as the outcome. Assistant (Google/Alexa/...) handoff could be another one.

One possible way to handle this would be for the skill handing off work to publish the target intent and its known entities/slots as part of the EndOfConversation event (SemanticAction, although strangely named, would be a way to express this, as long as the semantic actions are known ahead of time). Given the disconnected nature of skills this means the skill handing over the intent would have to be updated if the target skill changed their intent/entity/slot processing logic but it's a cleaner architecture. What do you think? Would there be a better way?

[Kumar3010]

@darrenj - could you please provide a sample for this which should have a VA and Skill

[Kumar3010]

@darrenj - could you please let us know when the docs would be updated.
appreciate your help.

[Kumar3010]

@darrenj could you please provide an update on the documents/sample update

[darrenj]

Linked a PR to the issue with an outline of the steps required to exchange data with Skills.

[KoradaRajesh]

@darrenj Passing data from skill to Parent Bot with out using Dialogs can this possible?

I'm using below snippet to call skill from root bot
var response = await _skillClient.PostActivityAsync(_botId, targetSkill, _skillsConfig.SkillHostEndpoint, turnContext.Activity, cancellationToken);

I'm using below snippet to end conversation in skill bot

var endOfConversation = Activity.CreateEndOfConversationActivity();
endOfConversation.Code = EndOfConversationCodes.CompletedSuccessfully;