Video/Image Not Showing

[arni5na]

Hi there! I'm trying to send a video/image to the chat UI from Rasa but this is what i'm getting:

This is a snippet of my actions.py code:

I have an actions server and my socket server code is on another Python file. If I use the SocketIOOUtput Class, it will ask me for a recipient ID which I'm not sure where to get. I'm pretty new to Python and Rasa. Any help would be much appreciated!

[arni5na]

Update: I used dispatcher.utter_attachment(message) with message = {
"type":"video",
"payload":{
"title":"Link name",
"src": "https://youtu.be/f3EbDbm8XqY"
}
}

But it still shows as text in the chat window:

[znat]

With Rasa Core it must follow the template schema.
So for an image:

{
  text: ...,
  image: ...
}

See: https://rasa.com/docs/core/domains/#images-and-buttons

[arni5na]

I was able to send an image by adding it in the templates of the domain file, but I can't send through the custom actions. I changed the value of the message variable in my actions file into this:
message = {
"attachment":{
"type":"image",
"payload":{
"text":"Test",
"image": "img.jpg"
}
}
}

Then I used the dispatcher to send it, but it still doesn't show the image. I'm not sure if I'm in the right track here. Is it possible to send image/video through the custom actions or it can only be done through the domain file?

[bing-zhub]

I have encountered the same problem with you, attachment should be part of response. The reality is that it is part of text. I don't know if you have solved this problem.

[arni5na]

Sorry, I got busy... Anyway, it seems that we need the Rasa core output channel of the webchat to send the video. I checked the rasa-addons link where we supposed can get it but I didn't find it there...

[hrqiang]

the socketio.py doesn't seem like it supports video tag, so there is a mismatch.
The other way is send thru custom message.

[arni5na]

It's still not working using custom message. This is the video JSON data I sent to the UI:

message = {
                "attachment":{
                    "type":"video",
                    "title":"Link name",
                     "src": "https://www.youtube.com/watch?v=f3EbDbm8XqY"
                }
            }

I did a debug on the UI and it's acknowledging that it received the data as video but it just doesn't show it. Is there a video JSON format that I should follow?

[arni5na]

Got it finally resolved. These are the steps I did:

In CollectingDispatcher class, I added this function:

    def utter_response(self, message: Dict[Text, Any]) -> None:
        """Send a message to the client."""
        
        self.messages.append(message)

In SocketIOOutput class of the socketio module, I added the following:

    def send_response(self, recipient_id: Text, message: Dict[Text, Any]):
        self._send_message(self.sid, message)

In the Dispatcher class of the dispatcher module, I made the following changes:

    def utter_response(self, message: Dict[Text, Any]) -> None:
        """Send a message to the client."""

        bot_message = BotMessage(text=message.get("text"),
                                 data={"elements": message.get("elements"),
                                       "buttons": message.get("buttons"),
                                       "attachment": message.get("video")})

        self.latest_bot_messages.append(bot_message)
        self.output_channel.send_response(self.sender_id, message)

[arni5na]

Sorry, above fix caused some issue when dispatching messages other than video attachment. I made changes to the send_custom_message function in socketio instead:

def send_custom_message(self, recipient_id: Text, elements: List[Dict[Text, Any]]) -> None: """Sends elements to the output.""" #print(">>>elements:", elements) message = '' try: if not any('attachment' in d for d in elements): message = {"attachment": { "type": "template", "payload": { "template_type": "generic", "elements": elements[0] }}} else: message = elements[0] self._send_message(self.sid, message) except Exception as e: print("Error sending custom message:", e)