Media overlay semantics

[mattgarrish]

The third bullet in 7.2.1 says:

[A Media Overlay Document] SHOULD use semantic markup where appropriate.

Aside from this being overly vague, I can't find anywhere where semantics could be applied where they are recommended. The skippability and escapability sections don't recommend semantics be set, and these would be the key reasons for them.

Should we remove this statement and determine in the future whether to recommend semantics be set for skippability and escapability, or even leave this for the accessibility specification to require? (It also only has semantics as an optional requirement right now.)

[GeorgeKerscher]

The skipability semantics were thought of for page numbers, where you do not want the narrative interrupted. Extended descriptions are wanted to be skipped automatically by people who can see the images. Escapability was for constructs such as tables where the top left to bottom right reading is not meaningful.

[iherman]

I tried to follow up the details of skippability and escapability right until the SSV document and I indeed did not find anything. These are only vague references to features that have never been defined...

I am not sure whether this is what you propose, but I would go for a radical surgery here. Namely:

1. Remove the statement that you refer to
2. Remove §7.3.3 from the content spec
3. Remove §8.4 from the RS spec
4. Remove references to these from the SSV document, e.g., in Introduction, and from §16, §17, §18, and §19: these all refer to "Media Overlay contexts".

Is this too radical?

[mattgarrish]

Is this too radical?

Ya, that's maybe a bit extreme... 😄

I don't think we need to remove the sections, as skippability and escapability are important for accessibility. What's weird about 7.4 is that it doesn't contain any normative requirements.

I partly understand this, as I believe the intention was always to introduce more requirements through the accessibility specification.

But we can't recommend semantics be applied as a core requirement of MO documents and then not recommend the use of semantics anywhere else where we talk about them. We should be consistent. I think 7.3.3 already says enough about using semantics.

[iherman]

I don't think we need to remove the sections, as skippability and escapability are important for accessibility. What's weird about 7.4 is that it doesn't contain any normative requirements.

Exactly. The current text is very hand wavy compared to the overall spec...

I partly understand this, as I believe the intention was always to introduce more requirements through the accessibility specification.

I forgot to react on your last remark of the opening issue, actually: shouldn't the bulk of these section then go into the A11y spec with possibly more details?

(As an aside: in 7.3.3. shouldn't both links in 2nd paragraph go to the RS spec?)

[mattgarrish]

Exactly. The current text is very hand wavy compared to the overall spec...

I think at a minimum there should be "MAY" statements in these sections for the accessibility specification to bolster up to recommendations/requirements. It's odd to have the core spec talk about them in a completely informative manner.

I don't like the idea of trying and integrate them fully into the accessibility specification, to be honest. That would cause other problems in terms of the accessibility specification then having to define reading system behaviours. Having a may would justify the reading system recommendations for support staying in these docs.

It might help to more fully explain what skipping and escaping mean in the reading system spec, as it is somewhat limited.

So for skipping say to bypass any seq/par element with the skippable semantic, including any nest par/seq, and continue with the next par/seq.

Escaping is a bit better defined in terms of expectations, but doesn't reference any elements. Maybe for that one just bolster what it's saying should happen with reference to par/seq elements. It might be useful to note that there may be multiple levels of escaping, too, like cell, row and table.

Would that make sense @marisademeglio ?

[mattgarrish]

(As an aside: in 7.3.3. shouldn't both links in 2nd paragraph go to the RS spec?)

Ideally, they would both be internal, but table reading mode is only mentioned in the RS spec as it's not specifically an authoring detail.

[marisademeglio]

Well in hindsight I would have left semantics out of MO entirely and left that part up to the content document alone but as things stand now, we can't break any theoretical MO files that may have been made super correctly with epub:type all over them. Right? Because if we can, ignore anything I wrote below.

Authors should understand what skip and escape behaviors are so that they don't overdo or under-do their semantics. And RS need to know which semantics should be associated with skip, and which with escape. Like George wrote above, skipping is for discrete items that repeat, e.g. page numbers or footnotes, whereas escaping is for containers like sidebars or table rows.

We have never said which semantics imply which behaviors, and this probably makes it pretty confusing for both authors and RS developers.

So, is our goal to be more precise or to leave it more open?

If the former, we need a list of SSV terms and the corresponding skip/escape for each, if applicable.

If the latter, we can probably move 7.4 to a note or incorporate it partially in 7.3.3 and partially in the RS spec (the skip escape section is quite light there).

Honestly this concept is a bit of a mess and I know when we started in 2011 (?) we didn't know what the MO/semantics relationship was going to turn out like, but given what we know now, I wonder who would miss it if we removed epub:type from MO entirely?

[mattgarrish]

Well in hindsight I would have left semantics out of MO entirely and left that part up to the content document alone

Ya, it is pretty redundant to have to repeat what can be determined from the markup. The requirement probably should have only been to structure the MO document to match the xhtml markup to enable skipping and escaping.

we can't break any theoretical MO files that may have been made super correctly with epub:type all over them. Right?

That's the requirement of our charter, yes.

We have never said which semantics imply which behaviors, and this probably makes it pretty confusing for both authors and RS developers.

We did try in the last revision, so we have those "non-exhaustive" lists. But there's definitely no explanation of how these work with expected RS behaviours.

I wonder who would miss it if we removed epub:type from MO entirely?

We can't take it out now, but we could make the skippability and escapability sections informative. It might impact the accessibility specification, although that spec only informally recommends following its MO practices.

I'll defer to @avneeshsingh and @GeorgeKerscher on where we want to go. I don't know if there's been any uptake of these semantics among DAISY producers, or if we expect there will be in the future.

Given that these features seem destined for under-implemented labels, maybe there is an opportunity to rethink how we expect reading systems to implement them, even if we don't officially ban use of semantics in the MO document.

[GeorgeKerscher]

With EPUB 3's implementation of MO, and the text centric nature of EPUB, I feel that we should do whatever we can to push the implementation towards using the semantics of the content document.

I am not sure what is allowed by the charter, but we should move in the proper semantic direction for the future. Whatever we can change now, I suggest we do it. We do not want to make things worse by more improper implementations being created.

Could we make significant improvements and see who complains about the changes breaking their existing content?

[mattgarrish]

I am not sure what is allowed by the charter, but we should move in the proper semantic direction for the future.

We can't deprecate the use of epub:type in media overlay documents, but we can remove the suggestion that they're used to enable behaviours like skippability and escapability, and change the section on semantics to say they're allowed but serve no practical purpose anymore. That wouldn't invalidate any existing content.

I don't know if that means completely removing the skippability and escapability sections, given that we don't have a working alternative. Maybe they stay informative but are rewritten to suggest the features be implemented from the xhtml structure? Maybe we take up replacing the features in the CG so we don't stall CR?

Given that there probably aren't implementations of skippability and escapability, and the charter is silent about removing unimplemented reading system features, there's no issue taking out the requirements from the reading systems specification.

If we do this, though, we'll also need to revise the accessibility specification to remove the skippability and escapability features.

[avneeshsingh]

We are at the end of development cycle of these specifications, so I believe that we should not do any radical changes at this time.
If replacing "should" with "may" in the following statement can allow us move forward, it would be a good way
A Media Overlay Document] SHOULD use semantic markup where appropriate.
We can add an informative note along with this statement explaining that semantics are important for accessibility.

[danielweck]

Given that there probably aren't implementations of skippability and escapability,

Hello, support for both features was implemented in the original Readium SDK early on (2014, IIRC), and the newer Readium incarnation also supports skippability (escapability currently not supported). Thorium exposes a user interface checkbox to let the user decide whether to skip page numbers / page breaks, asides, etc. during EPUB3 Media Overlays playback. The authored epub:type attribute values are used to extract the "skippability" semantics of certain content fragments when parsing SMIL XML files. In other words, the playback engine relies entirely on SMIL to orchestrate the text/audio synchronisation. We do not infer semantics from the referenced XHTML documents at runtime, as rendering / highlighting text is just a "side effect" of Media Overlays playback. From a reading system perspective, the transposition of XHTML semantics into SMIL at authoring / production time is a very helpful optimisation, otherwise we would have to perform costly document processing "ahead of time" or "just in time" during playback, either way this would be detrimental to the user experience.

[iherman]

authored epub:type attribute values are used to extract the "skippability" semantics of certain content fragments when parsing SMIL XML files

Thanks @danielweck, and it is great that we do have an implementation. From my editor's point of view: which value of epub:type do you use for skippability? Because, as a said in #2066 (comment), these values aren't defined anywhere, which is a problem for the interoperability of these features. Would we have to add new values to the SSV document?

[danielweck]

Current Readium software implements the non-exhaustive list from the specification (i.e. footnote endnote pagebreak https://www.w3.org/publishing/epub3/epub-mediaoverlays.html#sec-behaviors-skip-escape ) as well as a few other semantic roles, but I must admit not being up to date with the latest SSV (Structural Semantics Vocabulary) https://idpf.github.io/epub-vocabs/structure/
I see that note, rearnote, sidebar and annotation are deprecated but of course Reading Systems based on Readium traditionally maintain support for older specifications, so we'll keep these in for a while.
I must however update our implementation with all the roles marked as "skippable" in the latest revision of the SSV document.
In the real world, the "expectation bar" is low with regards to skippability and escapability support in mainstream EPUB Media Overlays, partly due to lack of consistent Reading Systems implementations, but also because authors probably find the specification lacking in clarity. So historically in Readium we implemented a reasonable baseline such as skipping page breaks, which is a very common use-case / requested feature.
I will review and update our implementation to include other semantics roles, but this will only translate into end-users benefits if/when SMIL XML files are created with epub:type semantic attributes that reflect the targeted XHTML document structures.

[marisademeglio]

Thanks @danielweck for the input! So are there EPUB producers today who are putting epub:type in SMIL files, at least for page numbers and footnotes?

[danielweck]

Hello @marisademeglio , from memory I have indeed come across several "native" EPUB3 Media Overlays publications (outside of the official EPUB test suite) that expose skippability semantics in their SMIL XML, but I do not know if this is common in mainstream publishing workflows. In fact, come to think of it, I strongly suspect that the publications I use for testing Media Overlays skippability are from DAISY folks / organisations. Speaking of which, note that Thorium supports DAISY 2.02 / 3 Digital Talking Books by transforming them into their EPUB3 equivalent ... or to be precise, we convert directly to Readium WebPub Manifest (aka. RWPM which is an internal optimised format), and we preserve the semantic structures commonly-found in DAISY, such as skippable page breaks.

[iherman]

The issue was discussed in a meeting on 2022-03-17

• no resolutions were taken

View the transcript

1. Media overlay semantics (issue epub-specs#2066)

See github issue epub-specs#2066.

Wendy Reid: you may have noticed a flurry of new issues logged due to people reviewing spec.
… we have a normative statement that MO SHOULD use semantics when appropriate, and we do not clarify what we mean by semantics, or what semantics to use.

See github pull request epub-specs#2089.

Wendy Reid: mgarrish has discussed with danielwreck and marissa, and put together a PR.
… to clean up the language in that section, particularly dropping SHOULD to MAY.
… we can't drop the statement altogether, but SHOULD is a little strong in this case.
… and to specify that we are talking about footnote, endnote, page break.
… now we have something that could technically be tested (although it is not a required feature, we are discouraging use of epub:type).
… there is also a small addition to the RS spec to make similar clarifications there.
… questions?.

Murata Makoto: this changes both Core and RS, right?.

Wendy Reid: yes.
… the change to RS is barely a change, it keeps the normative statements but offers a little more information about what the RS should be doing.

Murata Makoto: mgarrish dropped one bullet in EPUB 33 core?.

Wendy Reid: he changed one SHOULD to MAY, and both escapability and skippability are better defined. It should be in the PR.

Murata Makoto: seems sensible, but I haven't reviewed the details.

Wendy Reid: to clarify, he dropped the SHOULD, and made the following statement a MAY.

Murata Makoto: no objections, but 4 reviewers have proof read the proposed change?.

Wendy Reid: yes.
… I didn't know there was a going to be a PR in time for the meeting today, mgarrish was ahead of schedule.
… we will have to close this issue soon (within the next week or so) so if you have an interest in this please review the PR.