Video as additional metadata

[crouthamela]

I believe it would be beneficial to add a video tag for metadata, to enhance the screenshots already part of the specification. Many app stores (such as the Google Play Store, Steam, etc.) provide a means of adding a video, showing the functionality and features of the app. One can provide a quick summary of the app and demo of its use in a short period of time, enhancing the screenshots and description, which can include more detailed information. This makes it easy for potential users to get a feel for an app, and make a better decision on its installation. The Internet is increasingly relying on video for all sorts of needs, I believe this should be no exception.

I would recommend limiting the container and file extension to something such as H.264/MP4, and YouTube links to make it simple to implement for app store developers.

At very least, I hope this opens a discussion on the idea and what may be feasible.

[Pointedstick]

This being the FOSS world, the webm container and VP8 codec are probably going to be less controversial! :)

[crouthamela]

That's fine, you get the idea. :) But whatever is chosen, should be at least common on the web, and easy to implement in distros/applications. Webm/VP8 would qualify :)

[ximion]

Yes, webm/VP8 as container/codec would work, but we would kind of need to have the projects host these files on their own servers, so the appstream generators can download them and cache them for the respective distribution.
I wonder how useful that actually is / how many projects would use that, since many seem to prefer to just host videos on platforms like YouTube.

That being said, I have nothing against adding a new videos tag.

Brainstorming section

Maybe using a style like this:

<videos>
  <video format='webm-vp9'>
    <caption>Showing how to edit a video</caption>
    <url>http://example.org/myvideo.webm</url>
  </video>
</videos>

I don't like this much though, because it uses the url keyword in a different context than usual, also the format property should really be part of the tag containing the link.
E.g. something like this:

<videos>
  <video_group>
    <caption>Showing how to edit a video</caption>
    <video format='webm-vp9'>http://example.org/myvideo-vp9.webm</video>
    <video format='webm-av1'>http://example.org/myvideo.webm</video>
  </video_group>
</video>

In this case, I am not a huge fan of the video_group name though.

[crouthamela]

Why would video have the format tag but screenshot does not? Just to make sure the right codec is used?

I'm not sure what to use other than video_group either. Screenshots has screenshot and image, but I don't have a good synonym for video I can think of that would work better.

[hughsie]

Is a video just another logical kind of screenshot? In gnome-software it'll just be in the existing screenshot carousel. No app is going to have more than one video (video_group?), and we just say that videos have to be playable with gstreamer-plugins-good -- which limits to well known and free software codecs.

[ximion]

@hughsie I am pretty sure that apps will want to show multiple videos - or do you mean they will not have multiple versions of the same video with different encodings?
If we classify a video as just some different kind of screenshot, we could do:

<screenshots>
  <screenshot type="default">
    <image>https://example.com/test.png</image>
    <caption>XYZ</caption>
  </screenshot>
  <screenshot>
    <video format="webm-av1">https://example.org/myvideo.webm</video>
    <caption>Some text</caption>
  </screenshot>
</screenshots>

I don't want to say in the spec "we support stuff that X supports", because different software centers may use very different technology, and by saying "just use GStreamer" we mandate a choice of technology. Also, limits are good (and easy to adjust)

@crouthamela The video tag has a format specifier because just mandating one format does not make much sense. For screenshots AppStream only allows .png and .jpg screenshots (nothing else), which can be distinguished by their file extensions. For videos, with the mix of encoders and containers, the software center can not know in advance whether it supports the specific container/codec combination (but it needs to know that for a good user experience) - with specifying the format explicitly, we will be able to add support for WebM/AV1 as well as soon as it becomes available.

[crouthamela]

My initial thought was to have one introductory video, before the screenshots, in the screenshot carousel that gnome-software and discover use. But it makes sense to allow for multiple videos, just as multiple screenshots are supported, in case that is desired by the developer.

I like the idea of putting <video> under <screenshots>, that makes sense then, as a video performs a similar duty as a screenshot, but is simply a different format, and can live next to <image>.

Gstreamer-plugins-good includes the vpx plugin, which handles VP8 and VP9 (and I'm guessing AV1 in the future), so that would work fine if you restrict to that codec family, as that sounded like the plan anyway.

[ximion]

I am intending to go ahead with implementing videos as described in #175 (comment)
I am planning to only officially support FORMAT_WEBM_AV1, FORMAT_WEBM_VP9 and FORMAT_WEBM_VP8. These are the only free video codecs that are also supported widely by webbrowsers and that we really can expect software centers to have support for.

In future, allowing a dedicated format to embed videos from YouTube and other video platforms could make sense, but I would like to not go that route for now due to privacy concerns.
I intend for tools like appstream-generator to allow people to disable videos entirely in case their CDN can't handle them, and also allow to impose limits on how big these video files can get (especially seeing the GIF monstrosities some people produced).
I hope that makes sense :-)

[crouthamela]

Great! Thanks for continuing this and implementing it.

[ximion]

The specification now contains the details of what was discussed here.
The gist of the commit (6a39d7d) is the following:

• video tags are optionally available in screenshot tags
• A video tag may contain a width/height/container/codec property for metainfo files, but that is not a requirement.
• Only AV1 and VP9 are supported codecs, only WebM and MKV are supported containers
• Having audio in videos is discouraged, but if it is present, Opus is the only audio codec that is permitted. Metadata authors must expect that no audio will be played though.
• Video-containing screenshots must not be the default screenshot
• video and image tags must not be present in one screenshot tag at the same time. They are mutually exclusive.
• AppStream collection metadata generators / data repositories may restrict the use of video in any way they see fit (e.g. by filesize restrictions to 10MB), including by completely removing videos in their output
• Videos may be localized
• Videos may have a (translatable) caption, just like static-image screenshots

I am currently working on the implementation in libappstream which should be done today or tomorrow. Any feedback to the change is welcome, @hughsie please have a look as well!
As soon as this is in a release making further changes is harder.