Missing QnA Maker Insights Telemetry

[igarcia7]

What project is affected?

Virtual Assistant Analytics
currently on latest GA solution - 4.9.1 and Microsoft.Bot.Solutions(1.0.0)

What language is this in?

C#

What happens?

After connecting out of the box Power BI template, and loading app insights key. The data on tab QnA Maker Insights does not display data. After looking at EventTelemetry table, fields such as QnA_OriginalQuestion is empty.

QnA Maker Inisights Screenshot

What are the steps to reproduce this issue?

Following Analytics documentation to read telemetry from Bot located [https://microsoft.github.io/botframework-solutions/solution-accelerators/tutorials/view-analytics/1-intro/]

What were you expecting to happen?

display QnA Maker data to analyse VA information appropriately.

Can you share any logs, error output, etc.?

Any screenshots or additional context?

see above.

[darrenj]

Hi - Could you confirm you have enabled personally identifiable information as per this?

Thanks

[igarcia7]

Yes, the PII was enabled on startup.cs. I am receiving PII information on other tabs such as bot responses and User utterance.

[darrenj]

Hi,

Could you try the following change to opt in to the same personally identifiable information being collected for QnA scenarios? This is through setting the LogPersonalInformation property on the resulting QnADialog.

This section of MainDialog.cs creates and adds the QnADialog when required and is where you can set this property.

               return new QnAMakerDialog(
                    knowledgeBaseId: qnaEndpoint.KnowledgeBaseId,
                    endpointKey: qnaEndpoint.EndpointKey,
                    hostName: qnaEndpoint.Host,
                    noAnswer: _templateManager.GenerateActivityForLocale("UnsupportedMessage"),
                    activeLearningCardTitle: _templateManager.GenerateActivityForLocale("QnaMakerAdaptiveLearningCardTitle").Text,
                    cardNoMatchText: _templateManager.GenerateActivityForLocale("QnaMakerNoMatchText").Text)
                {
                    Id = knowledgebaseId;
                    LogPersonalInformation = true;
                };

[igarcia7]

Thanks Darren, that worked! I am seeing QnA Questions being populated into the BI dashboard. One thing, should the QnA_OriginalQuestion field be empty?

Also, you previously helped me resolve a bug with the Bot response and user utterance in the BI report. After further testing, I am seeing that some of the Bot Responses are not being pulled into the BI report (app insights), specifically for the Transcript tab. We have Feedback enabled in our VA and the Telemetry is only pulling the feedback response instead of the actual QnA reponse. see below.

Bot dialog example

Transcript Conversation:

[igarcia7]

Darren, I tested the 2nd scenario and removed the feedback option out of the VA and the bot response was able to record the correct information. Is there an update that needs to happen in order to incorporate feedback to the VA and not have it display the custom message as a bot response?
Thanks!

[taicchoumsft]

Hi @igarcia7,

A bug was discovered in the PowerBI template that only took one of the responses instead of showing all responses if the activity was multi-turn - we will fix it in a separate PR.

If you need a fix urgently, you can perform these steps manually on your PowerBI report, which will concatenate the texts into a single row:

1. Click on "Transform Data":
   
2. Click on the "Interactions" Queries, then "Advanced Editor":
   
3. In the Interactions codeblock, look for this block of text starting with #Grouped rows:

{"BotResponseText", each List.Max([BotResponseText]), type text}

and replace it with:

{"BotResponseText", each Text.Combine([BotResponseText], " | "), type text}

1. Close and Apply the changes, then look at your Feedback tab, the responses to a multiturn response should now look like this:
   

[pavolum]

@igarcia7 thanks for raising the issue and excuse the delay in getting back to you. We have updated the documentation to add the QNAMakerDialog LogPersonalInformation flag that is needed to ensure proper logging.
Heads up, our next release will contain a single flag that will set all the LogPersonalInformation flags throughout the solution from a single flag in appsettings.json (PR Here: https://github.com/microsoft/botframework-solutions/pull/3475/files).

[igarcia7]

Thank you @taicchoumsft for the resolution, changing the Power Query on interactions resolved the issue!

[igarcia7]

Hi @taicchoumsft , I actually have another Telemetry Dashboard issue regarding the VA. The Bot Responses for Adaptive cards are not being recorded, this is mainly the question being asked by the bot when showing the Adaptive options.

example below: the "Do you want the holiday policy or schedule?" question does not have a text value that will be picked up by the Telemetry as a Bot Response.

this is from the Transcript dialog: It is missing the Bot Responses. Let me know how we can handle these.

[taicchoumsft]

Hi @igarcia7 ,

The adaptive response comes through as a json blob in the telemetry, here's a sample for the help card:

[{"contentType":"application/vnd.microsoft.card.hero","contentUrl":null,"content":{"lgtype":"HeroCard","title":"Help for Virtual Assistant","subtitle":"This can be used to display information to help your user interact with your Virtual Assistant.","speak":"This can be used to display information to help your user interact with your Virtual Assistant."},"name":null,"thumbnailUrl":null}]

Do you want to show this entire blob of information as part of the bot response? Or maybe you want just the titles/subtitles?

[igarcia7]

@taicchoumsft, I would like to grab the question from the Adaptive card as the bot response, see screenshot above. I believe that should be the title. what would be the steps to test with both title and subtitle?

is the Json blob being picked up in the interactions table?

thanks,
Ivan

[taicchoumsft]

Hi @igarcia7 ,

I have a workaround, unfortunately it's not ideal, hopefully you can adapt it to better suit your needs. I'll share the method here, hope it unblocks you for now while I confer with my team to decide what's the best way to properly add support for adaptive cards.

So the approach is roughly this: we add the Card Title (or whatever field you need from the card) via a Kusto projection to the EventTelemetry table. Then we merge it in to the botresponsetext field in the Interactions table. The step-by-step is as follows:

1. Click on Transform Data from the Home tab
2. Click on EventTelemetry, and then click on Advanced Editor
3. Add this line to the query, and click Done:
   

Card_Title = (parse_json(tostring(parse_json(customDimensions.attachments))))[0].content['title']

4. Now go to the Interactions table, and change line 7 from:

#"Added BotMessageText Column" = Table.AddColumn(#"Added OperationID Column", "BotResponseText", each if [EventName] = "BotMessageSend" then [Text] else null),

to:

#"Added BotMessageText Column" = Table.AddColumn(#"Added OperationID Column", "BotResponseText", each if [EventName] = "BotMessageSend" then (if [Card_Title]<>null then [Card_Title] else [Text]) else null),

5) Click Done. The card titles should be interleaved in the right positions as the bot texts as follows:

[igarcia7]

Thanks @taicchoumsft for the thorough explanation, i did run into an error when adding the first query into the EventTelemetry table. Any suggestions?

I tried it with a new template as well to be sure it was not the modified template i have been using.

Error message without app ID -

DataSource.Error: Web.Contents failed to get contents from 'https://api.loganalytics.io/v1/apps/****************************/query?query=customEvents%0A%7C%20project%20%0AActivityId%20%3D%20customDimensions.activityId%2C%20%0AActivityType%20%3D%20customDimensions.activityType%2C%0AChannelId%20%3D%20customDimensions.channelId%20%20%2C%20%0AConversationId%20%3D%20session_Id%20%2C%0AReplyActivityId%20%3D%20customDimensions.replyActivityId%20%2C%20%0AFromId%20%3D%20customDimensions.fromId%20%2C%20%0ALocale%20%3D%20customDimensions.locale%20%20%2C%0ARecipientId%20%3D%20customDimensions.recipientId%2C%20%20%0ASpeak%20%3D%20customDimensions.speak%20%2C%20%20%0AText%20%3D%20%20%20%20customDimensions.text%2C%0AEventName%20%3D%20name%2C%20%0ATime%20%3D%20timestamp%2C%20%0ADate%20%3D%20timestamp%2C%0ADialogInstanceId%20%3D%20customDimensions.InstanceId%2C%0ADialogId%20%3D%20customDimensions.DialogId%20%20%2C%20%0ADialogStepName%20%3D%20customDimensions.StepName%2C%0ALuis_ApplicationId%20%3D%20customDimensions.applicationId%2C%0ALuis_Intent%20%3D%2' (400): Bad Request
Details:
DataSourceKind=Web
DataSourcePath=https://api.loganalytics.io/v1/apps/*******************************************************/query
Url=https://api.loganalytics.io/v1/apps/********************************************/query?query=customEvents%0A%7C%20project%20%0AActivityId%20%3D%20customDimensions.activityId%2C%20%0AActivityType%20%3D%20customDimensions.activityType%2C%0AChannelId%20%3D%20customDimensions.channelId%20%20%2C%20%0AConversationId%20%3D%20session_Id%20%2C%0AReplyActivityId%20%3D%20customDimensions.replyActivityId%20%2C%20%0AFromId%20%3D%20customDimensions.fromId%20%2C%20%0ALocale%20%3D%20customDimensions.locale%20%20%2C%0ARecipientId%20%3D%20customDimensions.recipientId%2C%20%20%0ASpeak%20%3D%20customDimensions.speak%20%2C%20%20%0AText%20%3D%20%20%20%20customDimensions.text%2C%0AEventName%20%3D%20name%2C%20%0ATime%20%3D%20timestamp%2C%20%0ADate%20%3D%20timestamp%2C%0ADialogInstanceId%20%3D%20customDimensions.InstanceId%2C%0ADialogId%20%3D%20customDimensions.DialogId%20%20%2C%20%0ADialogStepName%20%3D%20customDimensions.StepName%2C%0ALuis_ApplicationId%20%3D%20customDimensions.applicationId%2C%0ALuis_Intent%20%3D%20customDimensions.int...

[taicchoumsft]

Hi @igarcia7 ,

Just to triple check - did you add the , to the end of line 51?

If that still doesn't work, then can you post the Kusto portion of the event telemetry script (the code highlighted in red in the advanced editor) here?

Thanks.

[igarcia7]

good catch, I did miss to separate those two commands. After adding the comma, the error disappeared and i was able to save it with both changes. But once i checked the transcript, there was no bot response from the adaptive cards.

I did notice the title and subtitle missing from the Json in the Bot Emulator for the adaptive cards. I modified the query since i did see the text in the body and it pulled that text in the transcript. see screenshot.

I modified it to this: Card_Title = (parse_json(tostring(parse_json(customDimensions.attachments))))[0].content.['body'].[0].['text']

I still need to verify that all Adaptive cards have this body.text value in them to make sure it works for all of them... but I think this would be a feasible solution in the meantime. Let me know if there is anything i need to be aware from using this solution.