Image galleries & captions

[nadonomy]

Summary
Element doesn't support the ability to send or display multiple images together as a gallery, or the ability to add a caption to image(s).

Cross-platform discussion is here

[MurzNN]

There are already several MSC about adding text to images (MSC2529, MSC2530, MSC2881, MSC3382, maybe more? please point to them), but we can split their ideas to two types:

1. Classic behavior: media + caption: this is about adding text description, that comments the media. It suit well for articles, where we already have a long text and media (image, video, file) is just an item of the whole story, and it needs a short text description. In that cases it can be named "media caption" or "media description".

2. Chat behavior: text + media(s): This is a common usage of media in chat style conversations, where we have short text messages and a single media item (or series of several medias), that adds some information to that short text. So actually the text is the main entity, and the media is just an addition to it. The examples can be:

• Here is my photos from last weekend [image1, image2]
• My payment is rejected, here is the file with the rejection message: [file.pdf]
• I've got the "Access denied" error, screenshot is attached. How can I solve this problem?

Also very often we have not a single media, but several ones (two or more photos, several following screenshots, etc) and the "media + caption" behavior doesn't suite for such cases at all!

So I want to discuss the whole idea that the classic media caption implementation does not suit well for Matrix ecosystem (and other chat systems too), and that the text + media(s) behavior suits much better for chat style conversations. I agree that sometimes people really want to post a media and add caption to it (for people with disabilities, or to comment what is displayed on the image), but in most cases the behavior is totally backwards!

Please share your own thoughts about this idea!

[MTRNord]

There would also be https://github.com/MTRNord/matrix-art/blob/main/helpers/event_types.ts#L70-L72 for galleries. Which is ExtEv based. (There is no MSC for it.) Just wanted to mention it as a further thing that was done in this area.

[torrybr]

I think WhatsApp has this figured out quite nicely where if you upload many images, it sends them individually.. then groups them as a +24 images message type once all are uploaded so a user can easily download images and view in a gallery

[MurzNN]

I think WhatsApp has this figured out quite nicely where if you upload many images, it sends them individually.. then groups them as a +24 images message type once all are uploaded so a user can easily download images and view in a gallery

On the WhatsApp side seems this is only a "client-side sugar", on the server side it still represents each image as a separate message with the type "image", that's bad. So on the client side the WhatsApp app just visually groups several successive events with the type "image" to the gallery.

So WhatsApp isn't a good example to mimic the implementation.

[cwmke]

I think WhatsApp has this figured out quite nicely where if you upload many images, it sends them individually.. then groups them as a +24 images message type once all are uploaded so a user can easily download images and view in a gallery

On the WhatsApp side seems this is only a "client-side sugar", on the server side it still represents each image as a separate message with the type "image", that's bad. So on the client side the WhatsApp app just visually groups several successive events with the type "image" to the gallery.

So WhatsApp isn't a good example to mimic the implementation.

How about Signal? Their gallery/caption implementation works really nicely as well. I have no clue how it's implemented though.

[maver2]

I also consider the message with multiple images very useful. When sending some photos one after the other in a quite active family room, the first responses come alreday before I add the last photo and my message.

My workaround is: I write the message but don't send it yet, then I open the gallery app und share some photos targeted to Element (or some other matrix client), send the photos at once, then send the prepared message. This is very cumbersome.

My preferred procedure to send a message with some photos to a matrix room would be:

1. select some photos in gallery app
2. share them with Element/room
3. they are shown in Element, but not sent yet
4. add message text in Element
5. send all to the room.

The same could apply to any client including desktop.

[predictron-cloud]

So I want to discuss the whole idea that the classic media caption implementation does not suit well for Matrix ecosystem (and other chat systems too), and that the text + media(s) behavior suits much better for chat style conversations. I agree that sometimes people really want to post a media and add caption to it (for people with disabilities, or to comment what is displayed on the image), but in most cases the behavior is totally backwards!

Please share your own thoughts about this idea!

I think that these 2 options that you mentioned are basically 2 separate use cases and therefore they could be handled differently on both server and UI:

1. Sometimes user really want to post image (or gallery of images) with small caption. Also there is the use case of "public channels" (e.g. Telegram, WhatsApp etc.), where only moderators can send messages which are like "news". For such use case the image (or even better the gallery of multiple images) message with text in the "body" (MSC2530: Body field as media caption matrix-org/matrix-spec-proposals#2530) field could be the best option. News usually have one image(s) at a top - this is the design standard. I know that Matrix now is not supposed to be a tool for posting the news in public channels (e.g. Telegram), but it is so close to it, why just don't add such functionality at some point.
2. Images like screenshots with stacktraces in the text (e.g. in Microsoft Teams) are definitely a fruitful feature, but it is different use case that could be handled differently. I have the following observation: such behavior is often use with Drag-n-Drop on desktop, and rarely used on mobile devices (as DnD is not available) - that's why it is implemented in Teams (employees usually work on desktop PC) and is not supported in popular messengers (like telegram, whatsapp).

Forcing users to insert images at the top of the message using the inline images could be not extremally UX friendly having in mind that image in such case usually has some visual separation from the main text and appropriate image size adjustments should be applied. Also it is uncertain how to handle galleries with kind of "lightbox".

So, if we want to cover all use cases with inline images, I would propose to:

1. Implement not only inline images, but also inline galleries (with lightbox).
2. On mobile matrix clients, when user wants to send image/gallery with caption, insert it in the top of the message by default.
3. If the message starts with [image] or [gallery], then it should be visually separated from the other text (as text in such case is supposed to be a caption).

CC: @MurzNN , @nadonomy , @t3chguy , @kittykat