QnAMaker responses are said twice when using webchat

[gabog]

Project
Virtual Assistant

Description
When you use speech, QnAMaker responses seem to be read twice when using webchat (this doesn't seem to happen in the emulator)

To Reproduce

1. Deploy VA to webchat with speech enabled.
2. Say "What is your name" using speech
3. You'll hear "My name is the Virtual Assistant! Nice to meet you :)" twice.

Expected behavior
It should only say the response once.

[lauren-mills]

Are you using our webchat harness or the default one? Or a specific demo environment?

[gabog]

the webchat from test harness, will send you internal URL offline.

[gabog]

FYI, i think the issue may be related to the ThumbnailCard created from {Text: "Answer"} json, and it is probably also related to how the new webchat and universal speech handle speech for cards. This is what I observed:

• If the qna answer is plain text everything works well.
• If the qna answer gets deserialized in the thumbnail card and the Speak property is populated, Speech reads the Speak property AND the text on the card.
• If I comment the setting of the Speak property in the BuildQnACard method, then speech reads the card text (only once).

I think this is a bug in webchat or speech, they should stick to the Speak property if present and not attempt to read the card. This is the way it used to work with webchat v3.

If we do not set the Speak property on the response, other channels won't know how to read the card.

[gabog]

• @compulim, hi William, I think there is an issue with how webchat v4 handles speak for ThumbnailCards (not sure if other cards are affected too), WebChat v3 uses the Speak property and doesn't attempt to read the card.

[compulim]

[edit by corinagum]: These repro steps don't address @gabog's problem - his issue is with Speech, not screenreader. Scratching this out for clarity.

Background

All rich cards, including thumbnail card, is converted to Adaptive Cards on render.

Investigation steps

1. Launch Narrator
1. Load Web Chat in Edge, thru https://microsoft.github.io/BotFramework-WebChat/01.a.getting-started-full-bundle/
1. Type "card bingsports"

If the problem persists, we should fix it. If not, we should try a ThumbnailCard and see if it repro.

Expected

The speech engine (Cognitive Services or browser speech) should only speak out "The Seattle Seahawks beat the Carolina Panthers 40-7", but not the content.

[compulim]

@corinagum could you investigate with the repro steps above and see if it repro?

One thing Gabo pointed out "If the qna answer gets deserialized in the thumbnail card and the Speak property is populated, Speech reads the Speak property AND the text on the card."

[corinagum]

@gabog, are you still having this issue? I am not able to reproduce this on Mock Bot with a Thumbnail card. Could you clarify whether you are using Narrator, Cognitive Services Speech Services, or both? If Narrator, I believe this is an Adaptive Cards problem (see below). If Cognitive Services, then I will need more information to reproduce.

Regarding Narrator: I tried the Narrator steps that @compulim applied above, but I think they show a different bug than what @gabog reported. This has to do with how Adaptive Cards handles aria-label, which provides a high level description of the card via the speak property, but doesn't allow the user to hide irrelevant containers or text, or a way to manually provide aria-labels that improve the understanding of the card. If this is the issue you're having, @gabog, please let me know and I will close this issue. I recommend filing this again with the Adaptive Cards team.

If this is an ongoing Cognitive Services issue and Narrator is not involved, please copy-paste our 'New issue' template for reporting bugs and provide the information requested in that template. (You don't have to file a new issue, continuing the conversation in this thread is fine). Thanks!

[edit]
Just confirmed that this is not a Narrator issue. I think Narrator was added to this context incorrectly. We are still working on reproduction steps and getting more detail on this issue.

[gabog]

Hi, below is the repro code:

The minimalistic code to generate the card in C# is:

    public static IMessageActivity BuildThumbnailCard()
    {
        var response = Activity.CreateMessageActivity();
        response.Speak = "This is what the bot should say";
        var card = new ThumbnailCard(text: "This text goes in the card (and SHOULD NOT BE SPOKEN)")
        {
            Images = new List<CardImage>
            {
                new CardImage("https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcQtB3AwMUeNoq4gUBGe6Ocj8kyh3bXa9ZbV7u1fVKQoyKFHdkqU")
            }
        };
        response.Attachments = new List<Attachment>
        {
            card.ToAttachment(),
        };

        return response;
    }

That produces the following json for the activity:

{
  "type": "message",
  "id": null,
  "timestamp": null,
  "localTimestamp": null,
  "localTimezone": null,
  "serviceUrl": null,
  "channelId": null,
  "from": null,
  "conversation": null,
  "recipient": null,
  "textFormat": null,
  "attachmentLayout": null,
  "membersAdded": null,
  "membersRemoved": null,
  "reactionsAdded": null,
  "reactionsRemoved": null,
  "topicName": null,
  "historyDisclosed": null,
  "locale": null,
  "text": null,
  "speak": "This is what the bot should say",
  "inputHint": null,
  "summary": null,
  "suggestedActions": null,
  "attachments": [
    {
      "contentType": "application/vnd.microsoft.card.thumbnail",
      "contentUrl": null,
      "content": {
        "title": null,
        "subtitle": null,
        "text": "This text goes in the card (and SHOULD NOT BE SPOKEN)",
        "images": [
          {
            "url": "https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcQtB3AwMUeNoq4gUBGe6Ocj8kyh3bXa9ZbV7u1fVKQoyKFHdkqU",
            "alt": null,
            "tap": null
          }
        ],
        "buttons": null,
        "tap": null
      },
      "name": null,
      "thumbnailUrl": null
    }
  ],
  "entities": [],
  "channelData": null,
  "action": null,
  "replyToId": null,
  "label": null,
  "valueType": null,
  "value": null,
  "name": null,
  "relatesTo": null,
  "code": null,
  "expiration": null,
  "importance": null,
  "deliveryMode": null,
  "listenFor": null,
  "textHighlights": null,
  "semanticAction": null
}

When using speech, the bot will read the text property in the card and the speak property in the activity.

It should only read aloud the Speak property if set and never read the Text property in the card.