Should all Structural Vocabulary Items be normative?

[iherman]

Discussion started on the mailing list thread. The relevant part of the discussion (quoting from my original mail):

We generally have an issue at W3C on how we would define testing and implementation for vocabulary terms […]. The generally accepted approach is that for vocabulary items the term "implementation" is a a misnomer and it should be considered to be an alias to the term "usage" for our CR process. Indeed, there is no RS behavioral requirement for any of the SSV entries, so traditional requirements on implementations would not apply. Instead we may, for example, define "usage" of a specific vocabulary term, for our CR process, that there are at least two publishers out there who use these terms in production (hoping that at least some reading systems do something sensible with it). Such, or similar, ways for "testing" (per W3C process) for vocabularies was used in other specifications in the past.

However: would all SSV terms pass such a requirement? All the answers on John's mail suggest that the answer would be no […], because many terms have been introduced to the spec as part of an aspiration for something. Ie, we may be creating a problem for ourselves. Shouldn't we mark the SSV vocabulary as non-normative overall with, possibly, explicitly mark a few terms as normative because we know they are accepted by the community (e.g., landmarks)? Note that if we keep all the terms as normative and then we fail on the CR tests, the usual expectation would be to remove them altogether, which we do not want (I presume).

[iherman]

Additional relevant mails from that thread (to ease the discussion):

• https://lists.w3.org/Archives/Public/public-epub-wg/2021Jul/0053.html (from @mattgarrish)
• https://lists.w3.org/Archives/Public/public-epub-wg/2021Jul/0055.html (from @mattgarrish)
• https://lists.w3.org/Archives/Public/public-epub-wg/2021Jul/0056.html (from @johnfoliot)
• https://lists.w3.org/Archives/Public/public-epub-wg/2021Jul/0057.html (from @mattgarrish)

[iherman]

The issue was discussed in a meeting on 2021-08-13

List of resolutions:

• Resolution No. 1: Explore removing the epub:type vocabulary from the specification and into it's own note

View the transcript

2. Structural Vocabularies

See github issue #1763.

Dave Cramer: this all started on the mailing list with a chain about SSV. About the nuance between the different terms. Some are common in the publishing world, but not used outside. Over the years we've had questions about the SSV. And as part of the formal spec process, how do we show usage or implementation of these terms?

Ivan Herman: There are 2 different things here.
… at the moment, the way the document is written is such that the entire vocab is normative
… all the terms are normative. What this means in practice is proving that these terms are used by the community (e.g. epub authors). In analogy to traditional testing, the criteria that we had was that there should be at least two independent significant epub publishers that use that term in production.
… if there are two of these, then it is normative
… question is whether there is any probability that this test would show that the entire vocab is really normative
… my feeling is no. There are specific terms that would pass this definition of normative, but not all
… just a note that epubcheck is not a concern here as it has been changed to not flag terms in epub:type even if they are outside the SSV
… one thing we can do is make the SSV non-normative
… then it can come out of the spec, go into a registry
… but we need to determine which terms should be normative vs non-normative before we go to CR
… the other issue is whether some of the terms are vaguely defined
… but this later is an editorial job, whereas the first issue is more about process

John Foliot: i can speak to the perspective of a11y concerns. One of the issues we have to deal with in terms of WCAG is that our spec has been taken up by regulators and adopted into law
… for that reason we have to make sure that definitions are not changed on the fly, which could put people out of spec and cause legal consequences
… e.g. for micro-formats, the definitions existed on a public wiki that anyone could change. Presented legal risk for organizations relying on it.

Dave Cramer: To ivan's point on usage of these terms, I'm optimistic. Hachette uses 80% of these terms regularly.
… problem is that very few of the terms have any effect on many UAs
… some stuff about footnotes, endnotes, and the toc
… so it doesn't support a11y, and it doesn't do much outside of a11y

Avneesh Singh: +1 Dave, epubtype is not for accessibility

Avneesh Singh: aria roles have taken over. Media Overlays uses some EPUB-type.

Dave Cramer: how to maintain? We've ported the main terms over to DPUB-ARIA
… in terms of pure epub:type, not sure

Avneesh Singh: the definite list that JF was referring to should be ARIA roles, which are fairly stable
… epub:type is not really for RS, but more for production systems
… so I think we should pull this out and put it in a registry or a WG note

John Foliot: what about a must not clause in the spec? "SSV must not be used for a11y considerations"
… I was told that SSV should not be used to replicate nav functions, but it seems that this is something that Thorium does
… so this change would deter people from misusing

John Foliot: SHOULD NOT (Ref RFC 2119)

Dave Cramer: one thing is not sure how I would test that must not, would seem to require knowing why a SSV terms was put into an epub
… I see this as a matter of educating the community

Ivan Herman: agree that MUST NOT is not appropriate here as we couldn't test it
… but we could make the purpose of SSV much clearer in the spec, and think this should be done
… back to epubcheck thing, it seems it would do no harm to take SSV out of spec and turned into a separate note, and then we see what happens
… but there might be 1 or 2 terms that are used by RS (e.g. 'cover')
… so these we'd keep as normative

Avneesh Singh: we can also consider W3C registries for this

Wendy Reid: as far as i know, the only epub:type that is categorically relied upon by RS is footnote
… there are other ways for the cover to be identified

Dave Cramer: over the course of the last few revisions of epub we've made a concerted effort to bring in all the registries that were floating around because having them separate made spec hard to read
… i think some of those registries just ended up being not needed
… in this case its maybe not a bad idea to move the SSV away from the core spec because of its lack of utility

Ivan Herman: at present the SSV is subsection 8 of appendix D. While we are at it, we may want to take another look at all the subsections there
… try to be consistent in how we treat these

Dave Cramer: i'm wary of having some of these terms be normative, and the rest non-normative. I think this will confuse people.
… although I see the distinction

Avneesh Singh: in the pub manifest WG we're relying a lot on Schema.org vocabs

Ivan Herman: but references to Schema.org were considered to be as strong as normative, but yes, formally its not normative
… Schema.org is on the borderline of whether something is normative by virtue of being used by so many sites

George Kerscher: we have footnote in DPUB-ARIA so i would suggest moving all of this vocab into a note, and then just recommend using DPUB-ARIA for semantics
… and not have epubcheck report on this

Avneesh Singh: just flagging that we need to maintain backwards compatibility

Dave Cramer: right, using epub:type won't invalidate an epub. Consistent with how HTML is supposed to ignore attributes it doesn't understand
… so little harm to allowing epub:type to continue existing in epubs
… but want to move away from using it to convey information about a11y, semantics

George Kerscher: agree

John Foliot: where would we move the SSV?

Ivan Herman: into a WG note

Brady Duga: so there would be no normative way of doing footnotes (even though they would still exist)

John Foliot: what about the proposal to have registries that would be normative even though they aren't full spec

Ivan Herman: if we can't test these terms then they can't be normative

Dave Cramer: does Google Play do pop-ups?

Brady Duga: yes

Dave Cramer: based on epub:type?

Brady Duga: yes, but we also look for numbers that have <sup> around them, but excepting certain cases, etc. etc. It's involved.

Wendy Reid: Kobo has the exact same thing

Dave Cramer: does anyone look at the DPUB-ARIA vocab for this?

Wendy Reid: no

Brady Duga: no?

Wendy Reid: but it would be trivial to change over to looking for DPUB-ARIA, since we already have the logic of checking for epub:type

Dave Cramer: so could we migrate the things that rely on epub:type to refer to DPUB-ARIA?

Charles LaPierre: our Global Certified Accessible program is pushing for use of DPUB-ARIA vocab, especially for footnotes
… and we're de-emphasizing the use of epub:type
… we're fine if there's duplication between EPUB and DPUB-ARIA, but we don't require use of both vocabs

Dave Cramer: not quite ready to resolve here. Could we put the question of how to separate out the SSV to the group?

Ivan Herman: we could resolve that this is the direction we want to take with the SSV, with language to decided upon

Avneesh Singh: in epub a11y 1.0, we required both epub:type and DPUB-ARIA. But in recent revision we de-emphasized epub:type. In MO we do use epub:type, but this could be changed over time
… one thing about DPUB-ARIA is difficult to modify. Good for a11y, but might not work for publishers who rely on it for process

Proposed resolution: Explore removing the epub:type vocabulary from the specification and into it's own note (Wendy Reid)

John Foliot: +1

Ben Schroeter: +1

Ivan Herman: +1

Matthew Chan: +1

Charles LaPierre: +1

Wendy Reid: +1

Brady Duga: +1

Murata Makoto: +1

Masakazu Kitahara: +1

Dave Cramer: +1

Aimee Ubbink: +1

Avneesh Singh: +1

Resolution #1: Explore removing the epub:type vocabulary from the specification and into it's own note

Dan Lazin: +1

[iherman]

The issue was discussed in a meeting on 2021-08-19

List of resolutions:

• Resolution No. 1: Do a research study into the usage of the vocabularies in EPUB authoring

View the transcript

1. Vocabularies

Dave Cramer: See Appendix on vocabularies

See github issue #1763.

Dave Cramer: Main topic for today is the vocabularies
… we had a substantial discussion in the last meeting about what to do with the structural semantics vocabulary (SSV)
… there are some concerns about asserting some of the vocabularies as substantive
… some of these values don't have behaviours attached, some do
… given our overarching goal of reaching W3C recommendation status and interoperability
… what we have to show here is that a particular vocabulary item is used by document authors, not necessarily used by reading systems
… i.e., epub:type="bodymatter"
… even though it doesn't impact the rendering of EPUBs
… it seems to me that we need to collect some information about these values
… especially from anyone creating EPUBs and using these terms
… Ivan was also interested in moving the vocabularies into a separate note
… I'm unsure
… we just embarked on a long process of moving things here from their various homes
… to codify it for anyone looking to learn about EPUB
… or what properties to use with the item element
… are the terms normative
… is the appendix normative
… in the last meeting we explored moving the SSV vocabulary to a non-normative note
… that's what I'm doing here, exploring the other side
… I think it would be good as a group to note what terms are used
… and to get that information from other parts of the publishing world
… we do want to demonstrate wide use of these terms to defend retaining them
… any thoughts?

Murata Makoto: I always try to avoid these things

Dave Cramer: Same
… do we think it's enough to gather information on them for now

Shinya Takami (高見真也): I think some prefixes for fxl or epub:type are used in many cases
… it's necessary to maintain some of these for interoperability
… but may not be required to be normative
… I'm in support of moving them out

Murata Makoto: Even if we specify these values, do reading systems care?

Dave Cramer: Some are important, like nav or cover-image
… the FXL properties are important
… and have an impact on rendering
… one of my goals is to have this spec match reality
… there's already deprecated values amongst these vocabs
… if our research shows that more are aspirational than realistic
… I do want to inform an author of that fact
… Are we good to collect some information on these terms to help us make a decision on this?

Wendy Reid: +1

Murata Makoto: +1

Shinya Takami (高見真也): +1

Proposed resolution: Do a research study into the usage of the vocabularies in EPUB authoring (Wendy Reid)

Dave Cramer: +1

Wendy Reid: +1

Shinya Takami (高見真也): +1

Masakazu Kitahara: +1

Resolution #1: Do a research study into the usage of the vocabularies in EPUB authoring

Dave Cramer: I will figure out a methodology for this

[mattgarrish]

FYI, here are the results we got when we polled publishers on their use of dpub-aria roles, which was based on converting from the equivalent epub:type values: https://w3c.github.io/test-results/dpub-aria/

[iherman]

Thanks, @mattgarrish, we may have the answer were looking for! (In case we decide to keep things normative, we may reuse the reference as part of the CR report!)

[iherman]

To help the discussion I have compared the dpub-aria report with the terms in the spec. The list below contains terms that are NOT covered by dpub-aria, i.e., whose normative role in terms of the spec is still in question:

• §D.8.1 Document Partition
  • frontmatter, bodymatter, backmatter
• §D.8.2 Document Divisons
  • volume, division
• §D.8.3 Document Sections and Components
  • preamble
• §D.8.4 Document Navigation
  • toc-brief [draft], landmarks, loa, loi, lot, lov
• §D.8.5 Document Reference Sections
  • keywords [draft]
  • §D.8.5.1 Indexes
    • index-headnotes, index-legend, index-group, index-entry-list, index-term, index-editor-note, index-locator, index-locator-range, index-xref-preferred, index-xref-related, index-term-category, index-term-categories
  • §D.8.5.2 Glossaries
    • glossterm, glossdef
  • §D.8.5.3 Bibliographies
    • (none)
• §D.8.6 Preliminary Sections and Components
  • titlepage, halftitlepage, copyright-page, seriespage, imprint, imprimatur, contributors, other-credits, revision-history
• §D.8.7 Complementary Content
  • case-study
• §D.8.8 Titles and Headings
  • (All terms)
• §D.8.9 Educational Content
  • (All terms)
• §D.8.10 Comics
  • (All terms)
• §D.8.11 Notes and Annotations
  • (none)
• §D.8.13 Document Text
  • keyword, topic-sentence, concluding-sentence
• §D.8.14 Pagination
  • (none)
• §D.8.15 Tables
  • (All Terms)
• §D.8.16 Lists
  • (All Terms)
• §D.8.17 Figures
  • figure (this is the only term in the section
• §D.8.18 Asides
  • aside (this is the only term in the section)

Some notes:

1. "(none)" means that all the terms listed in that specific subsection are present in the report, ie, the corresponding section may be considered as normative without further ado (if we consider the report as an implementation report for epub).
2. "(All Terms)" means that all the terms listed in that specific subsection are missing from the dpub-aria report. Unless there are good reason not to, I would think we can simply label the corresponding section as non-normative.
3. A number of entries are labeled as "deprecated". They should be considered as "non-normative" anyway. These not appear in the list above.
4. There are a number of entries labeled as "draft" (some of them are labeled as such in the list above). I do not know whether they should be considered as non-normative; note that some of the draft entries are actually listed in the dpub-aria report.
5. There are full sections (e.g., lists, tables, asides, figures) that are missing from the dpub-aria report because they correspond to semantically clear elements in HTML and, therefore, are not labeled with ARIA.

[iherman]

The issue was discussed in a meeting on 2021-10-08

List of resolutions:

• Resolution No. 1: Move the epub:type vocabulary into it's own note

View the transcript

1. Should all Structural Vocabulary Items be normative? (issue epub-specs#1763)

See github issue epub-specs#1763.

Wendy Reid: as we approach CR we have to clear out our issue list (or defer them). I prefer finding resolution.
… this one is regarding SSV. We had previously decided to do research about moving epub:type out of spec and into its own document.
… so what are publishers using? Are they using DPUB-ARIA? And also, what are RS doing with epub:type (if anything)?
… I was still working on the RS research side of things
… but mgarrish shared a survey of publishers who had reported use of DPUB-ARIA

Ivan Herman:

Ivan Herman: i did some work on this topic. Posted my results on the issue recently.
… I looked at what the DPUB-ARIA group did about collecting information, and compared it against what is today in the spec
… per W3C rules, we have to prove that each term is used at least by 2 publishers (in this case)

Ivan Herman:

Ivan Herman: this is what DPUB-ARIA group did for certain of their terms, namely those that have an equivalent as epub:type values.
… so I compared this list with what is in our document, to see which are the terms that may have issues under this W3C process (i.e., which terms might we have to specify as non-normative)
… good thing is that a lot of terms are already okay, and can be kept as normative by referring to the DPUB-ARIA report
… but some terms (or even sections) might have to be labelled as non-normative
… you can go to my comment, in which I went through the SSV, and listed terms that are not covered in DPUB-ARIA report
… for these we have to do our own research to show that they are used by at least 2 publishers, or we have to clarify that these terms are non-normative
… some of the affected terms comprise entire sections
… when I say non-normative, an alternative approach is to label as "deprecated"
… if we did this we would still have to clarify that "deprecated" means "non-normative"
… there are a few sections towards the end of my list in the comment that are slightly different
… as far as I know ARIA does not define labels for those HTML elements that have clear semantics attached to them
… so all our SSV terms for tables, asides, etc., are not DPUB-ARIA terms
… I wonder whether we can do the same as what ARIA does. We could deprecate those completely.

Wendy Reid: very helpful, Ivan

Brady Duga: for marking something as a table it might be used in the case of CSS tables

Ivan Herman: but are there cases of publishers using that?

Brady Duga: its typically used where publisher wants to emulate the styling of a table, but for non-tabular data

Wendy Reid: for most of these we could find weird edge cases like that
… with DPUB-ARIA testing has already been done, but we don't have that same data for epub:type
… and getting similar data for epub:type could be difficult

Ivan Herman: if I understand well, what they did in that report is to look at use of epub:type by publishers to show that use of the same terms in ARIA makes sense
… but we'd need someone who was there at the time, e.g. mgarrish, to say for sure

Brady Duga: we had discussed doing searches in our epubs for these, but we left it off at asking for a list of search terms
… but I can't tell you which publishers use which terms (that's private user data)
… but I might be able to 1) go to publisher to ask permission to share, or 2) share data in the aggregate

Wendy Reid: yeah, that at least 2 publishers are using a term, for example

Brady Duga: right, as long as you will take my word for it

Avneesh Singh: ARIA roles becomes significant for a11y, but epub:type became used for production processes
… from principle of backwards compatibility, I would be concerned about removing it
… but also, historically, epub:type was intended to be extensible
… so I would prefer moving the SSV into WG note

Ivan Herman: re duga comment, I understand some of that information is confidential, and I seem to remember that the W3C process might allow that sort of evidence to be accepted
… but please do the search first, and if the result shows at least 2 publishers, then I will talk to W3C internal about using it

George Kerscher: all this stuff about SSV came from old work about a11y, but once DPUB-ARIA got going, a lot of these things got replicated
… because SSV is outside the norm of HTML processing, I see no reason not to move it to a note, preserving the ability of publishers to use it for production if they wish
… but it would also encourage more extensive use of DPUB-ARIA

Gregorio Pellegrino: from epub author perspective, InDesign only allows users to put epub:type, but not ARIA role semantics

Brady Duga: re. Ivan, I'm greatly constrained in what I can do with user data, regardless of what the W3C process allows

Ivan Herman: right, so in that case, I might be able to look into how we could use that research without need to show any user data
… mgarrish, was it the case in your report that you looked at each epub:type, and then looked at which DPUB-ARIA role analog they could/would use?

Matt Garrish: yes, it basically confirms that those epub:type values were in use by those publishers

Ivan Herman: so for those terms that appear on your list, we could decided to make them normative, if we wanted?

Matt Garrish: yes
… the question I have is whether there is a need to make them normative

Ivan Herman: that's a somewhat separate question. We were talking about what we could ascertain about publisher usage from that DPUB-ARIA report

Matt Garrish: if epub:type isn't really a major focus of the spec anymore, then do we need it in the spec at all?
… it takes a lot of space in the spec for something that we're really de-emphasizing at this point
… there are some terms that the spec depends on, but do we need the entirely SSV just to call on a couple terms?

Hadrien Gardeur: i don't think we should have it
… we're giving a lot of emphasis to something that isn't widely implemented
… we have too much in the SSV, and I wouldn't miss anything if we just had the DPUB-ARIA terms

Ivan Herman: based on all that, I propose that we formally say we will move SSV into a separate note, leaving the follow up as editorial work
… but we also have other vocabs in the spec for which the same question (normative or not) arises

Wendy Reid: let's take one at a time

Gregorio Pellegrino: it would be fine for me to remove SSV from spec, but epub:type is important for MO, right?

Matt Garrish: it is meant to influence that, but its not a requirement
… the navigation relies on epub:type too, but I don't know that you need the entire SSV in the spec to enable that
… don't know if its a procedural issue to reference terms from a note, but assuming its not an issue, this isn't a problem

Ben Schroeter: i thought that UA relied on epub:type for certain affordances like notes

Matt Garrish: yes but that was never part of the spec
… epub:type was loosely defined on purpose to encourage experimentation

Wendy Reid: RS do use epub:type to identify footnotes, but most RS also have to have 2 or 3 fallback ways of identifying footnotes because its so inconsistently used

Avneesh Singh: is there any benefit of putting this SSV into W3C registry instead of WG note?

Ivan Herman: I wonder whether this is a vocab that will be used in coming years, or more for archival of something that will not evolve much
… I feel it is probably more the latter than the former
… if that is correct, then turning it into a registry is not for us

Avneesh Singh: thank you

Bill Kasdorf: SSV is used a lot in production workflows, both by publisher and vendors
… and there's a value for everyone to be using the same terms for the same things

Bill Kasdorf: so having it live somewhere and be referenceable is valuable

Matt Garrish: I don't think we need to worry that people will think the SSV has disappeared or anything
… I'm for not making it into a formal registry

Bill Kasdorf: what I was saying was the terms themselves, not so much the epub:type prefix

Proposed resolution: Move the epub:type vocabulary into it's own note (Wendy Reid)

Ben Schroeter: +1

Matt Garrish: +1

Matthew Chan: +1

Ivan Herman: +1

Brady Duga: +1

Wendy Reid: +1

Murata Makoto: +1

Bill Kasdorf: +1

Masakazu Kitahara: +1

Avneesh Singh: +1

Dan Lazin: +1

Toshiaki Koike: +1

Gregorio Pellegrino: +1

George Kerscher: +1

Resolution #1: Move the epub:type vocabulary into it's own note

Dan Lazin: why do we want to move it into its own note? What does this accomplish?

Ivan Herman: the question was not raised this way, but rather whether the SSV is normative or not
… if yes, then we have to have a mechanism to prove that each of those terms are used by two publishers
… and because there are so many terms, it would present a problem for testing

Dan Lazin: that makes sense, thank you

[iherman]

The issue was discussed in a meeting on 2021-10-14

List of resolutions:

• Resolution No. 1: Publish the SSV as a working group note, using ECHIDNA, using the short name "epub-ssv"

View the transcript

1. Publication of Structural Semantics Vocabulary as Working Group Note

See github pull request #1847.

See github issue #1763.

Dave Cramer: last week we resolved on separating SSV into a wg note
… we just need a formal resolution to publish it, and to use echidna

Proposed resolution: Publish the SSV as a working group note, using ECHIDNA, using the short name "epub-ssv" (Wendy Reid)

Murata Makoto: the link in Ivan's message doesn't work, so I can't see the note

Dave Cramer: https://github.com/w3c/epub-specs/blob/main/epub33/ssv/index.html

Wendy Reid: See The editor's draft for the note

Dave Cramer: +1

Wendy Reid: +1

Matthew Chan: +1

Matt Garrish: +1

Toshiaki Koike: +1

Masakazu Kitahara: +1

Shinya Takami (高見真也): +1

Murata Makoto: does this document have to do with the IMS Global Consortium? Do they need to see this?

Matt Garrish: no, i think this originally came from DAISY

Murata Makoto: what about the content under sec. 10?

Matt Garrish: this is all stuff that came from us, we don't have dependence on any other party

Resolution #1: Publish the SSV as a working group note, using ECHIDNA, using the short name "epub-ssv"

Dave Cramer: