Allow components to specify preferred display properties for better mobile device support

[ximion]

Hi!
It is useful for software centers and shells to determine whether a (graphical) application is suited for specific screen sizes as well as potentially screen resolutions. One usecase is for software centers to only show applications suitable for a phone device when run on a mobile phone, and only ones suitable for a TV screen when run on a smart TV.

History

In the past I wanted to just encode the chassis type as defined by systemd's machine-info specification. In prior discussions, this idea was shot down by multiple partys though, because it is very inflexible and we don't really have a clear definition of what a "phone" actually is. We could define that, of course, but our definition would always be insufficient for new, emerging device types. Who would have thought that smart watches would be a thing so soon as well? In addition to that, a lot of apps are adaptive now and run on ranges of screen sizes with diferent user input methods, and we need to account for that as well.

To solve this, I proposed a mechanism based on capabilities, which raised no complaints, except for the screen dimension approach, which we deferred at the time as "a problem to solve later". Well, I think the time to address this is now. Not only do we want this feature in Purism as we happen to develop a mobile phone, but there are also multiple other phone and "Linux on other devices than a PC" projects that I am aware of that have asked for it.

Support for the user input control property has already been implemented (see #265) and is widely available now, so the only missing bit for apps to advertise support for different device types is information about the screen (I am aware that some people also would like to specify support for additional peripherals, but that's - again - a problem for later ;-) ).

Prior art

To move this effort forward and not have it get stuck for a while like before, I looked at some prior art. First, CSS apparently had a media-types query which is deprecated now in favor of the device-feture-based media features tests.
This makes me feel very good about not using the chassis typy in AppStream but to go for a feature-based approach as well.

Android has a way to set restrictions on screen properties in a manifest file as well, and this one is pretty close to something we can use in AppStream. Android uses sizes like "small", "large", etc. for screens, which actually resolve to dimensions in dp.

On the matter of DPI/scaling, Pekka Paalanen also forwarded an interesting disccusion to wayland-devel recently: Re-thinking DPI and scaling

Proposal

Here's a quick draft to comment on what I think we can do in AppStream to have applications make recommendations for screen sizes or require certain screen sizes.
It is pretty much a "conversation starter" so we have something to discuss, and in no way a final draft.

If no preferences are specified, software centers have to assume the application supports all screen dimensions, because the reverse would break existing assumptions and also be inconsistent with the rest of the AppStream spec. However, on certain devices the case of "did not specify anything" may cause the application to be hidden, e.g. on a mobile device (as otherwise all desktop apps may show up which didn't have metadata set). LibAppStream would have API for that.

All new feature tests unitile AppStream's requires/recommends system, where "requires" means the app will not work if the condition is not satisfied, and "recommends" means the application will work, but may work poorly.

Units

The situation with units to measure various properties of a screen is an absolute mess. Sometimes (in case of dp & co.) even the same unit may have different interpretations and all are commonly used.
Within AppStream, I would love to not add extra unit conversion, but there is probably no way around that. I would also like to use the metric system consistently everywhere, but given how common units like "dpi" are, having AppStream use "dpcm" could actually be more confusing than just using inches.
This proposal ignores the units problem for now and uses metric units, but I think introducing a unit= property may actually be necessary. Feedback on this is very welcome!

Screen size

The screen size is defined as the screen diagonal in mm. Components can set a preferences like so:

<requires>
  <display_size compare="ge">18</display_size>
</requires>

The compare operator types are the same as defined for version numbers. In the above example, the component requires a screen with a diagonal bigger or equal to 18mm (7in).

Considerations:

• Do we want to define actual screen dimensions using width/height here, or is that too much and the diagonal is enough?
• Do we want a any wildcard to mean "any screensize is supported" explicitly? (IMHO just defining a lower limit is good enough)
• Are there any "odd" (non-rectangular) screen shapes that we'd need to support somehow?

Screen resolution

The screen resolution is defined in dpcm.

<requires>
  <display_resolution compare="ge">118</display_resolution>
</requires>

Considerations:

• Would we want to allow human-readable names like xhdpi here as well, just like Android uses?

Aspect ratio

The aspect ratio is proportional relationship between the width and the height of the display separated by a colon.

<requires>
  <display_aspect_ratio compare="ge">16:9</display_aspect_ratio>
</requires>

Considerations:

• Do we need this at all? (definitely not if we would define the actual screen with/height in screen_size) - is the aspect ratio actually relevant for apps?
• How do comparisons work? Different aspect ratios may be equal, but not the same.
• Represent aspect ration by a float instead of the actual ratio?

Viewing distance

• Should we include this as metadata in some way as well? This may be relevant for TVs mostly.

Other options

We could also do away with defining individual properties and melt things together by using video standard names such as "WXGA". That would be less flexible, but a lot easier to specify.

Discussion

This bug report is intended to be primarily a conversation starter, and to have a dedicated place again to discuss the "screen property metadata" issue again. In the past I tended to get much more feedback via GitHub issues compared to our mailinglist discussions, so let's try GH this time :-)

Since we do need some way to determine "works on mobile" based on device features for the Librem5 phone, I do intend to get this feature specified this time in a reasonable amount of time though (in no way will we have something half-baked in AppStream though that can't reach some agreement and that will later have to be maintained indefinitely).

Cheers,
Matthias

CC: @hughsie @tintou @cassidyjames @PizzaLovingNerd

[ximion]

Thinking about display_size, I think we will definitely need some shorthands like small, medium, large and xlarge defined, so application authors are spared the excessive googleing which devices have which sizes (with "small" meaning "handheld-sized", "medium" meaning "tablet-sized", "medium" meaning approximately workstation screen sizes and "xlarge" meaning common TV-sizes). What these sizes mean in numbers would be defined by libappstream and mentioned in the docs, and may potentially change over time.

[Exalm]

I wonder what's the benefit of using screen physical size and dpcm over screen resolution in logical pixels, e.g. like Android does.

From apps fitting onto a phone or not this is the thing that matters the most, namely the smallest side.

Then an app that fits onto an L5 needs to specify smallest-side=360dp and that's about it.

[hadess]

Thinking about display_size, I think we will definitely need some shorthands like small, medium, large and xlarge defined

What's considered small, medium, large, or xlarge will change over time. There are physically large tablets with small pixel dimensions, there are smaller phones with excessive pixel count.

Does the application work vertically is almost a better filter for whether it's suitable to run on a phone.

[ximion]

@Exalm

I wonder what's the benefit of using screen physical size and dpcm over screen resolution in logical pixels, e.g. like Android does.

Mostly I did this to first be future-proof in case the additional information of a physical screen size is needed at some point, and second so AppStream avoids having to define what "dp" is. According to https://en.wikipedia.org/wiki/Device-independent_pixel there are at least two definitions in use for Android and Windows, and AppStream could add a third or adopt one of them. Qt and GTK on the other hand don't even seem to use dp but instead use logical pixels and scale them to fit the screen (or use "platform defaults"). CSS also appears not to use dp.

[ximion]

@hadess

What's considered small, medium, large, or xlarge will change over time. There are physically large tablets with small pixel dimensions, there are smaller phones with excessive pixel count.

Indeed, that's why display_size is about the display's physical size in some unit of length, independent of pixel density. Having the named placeholders would be for apps which really don't care about the exact display sizes and fine-grained control over metadata but just want to say "I roughly want a desktop-sized display" with the definition of what that is left up to the current AppStream implementation (which explicitly may change over time).

Does the application work vertically is almost a better filter for whether it's suitable to run on a phone.

That's a good point. Would something like

<display_height>11</display_height>

(with length in cm unless unit specified) make sense to you, @hadess @Exalm ? This would do away with the screen diagonal measure being used for screen size, which doesn't actually mean anything, and would also leave the door open for having apps recommend a screen width if we ever need that for some reason.

[Exalm]

Not sure height alone is a good indicator. A 16:9 9" laptop would come very close to 11cm, with about 112mm of height if my calculations are correct. Would be completely different thing for the apps though.

And then, what about landscape?

That said, if it's all hidden behind named placeholders, likely fine.

[ximion]

And then, what about landscape?

Good point - we could either define a screen "length", being the longest side, or just define "height" as the screen height when the device is in its common, non-rotated orientation.

Not sure height alone is a good indicator. A 16:9 9" laptop would come very close to 11cm, with about 112mm of height if my calculations are correct. Would be completely different thing for the apps though.

In the sense of mobile-readiness, the screen dimensions will always be considered together with the input methods. E.g. an app that has a minimum screen height of 11cm will be considered for a laptop in any case (as long as it also can be controlled with a pointing device, maybe), but for a phone or tablet "touch" as input method may be mandatory.
This is a valid point though, in the instance of this particular display, going with the diagonal may be better. Or we could of course encode the screens minimum surface area.
All of this seems less than ideal though...

That said, if it's all hidden behind named placeholders, likely fine.

Developers could do either of this:

<requires>
  <display_height compare="ge">11</display_height>
  <display_height compare="le">large</display_height>
</requires>

so, either specify an explicit size or a placeholder.

[Exalm]

IMO here minimum side makes more sense rather than the maximum one. A phone is certainly going to have much smaller minimum side than any laptop or tablet, meanwhile maximum side is going to be closer considering current phones typically have a lot more extrame aspect ratio than desktops/laptops, and so far adaptive apps have had far more problems with width rather than height so far.

But then we also have ultra-side screens...

[ximion]

A specification draft for how this change may look like now exists at https://github.com/ximion/appstream/pull/286/files

[ximion]

This is fixed now via display_length. See https://www.freedesktop.org/software/appstream/docs/chap-Metadata.html#tag-requires-recommends-display_length for details.