text/vtt as core media type?

[VeryGoodErotica]

I did not see text/vtt listed as a core media type but it is the required subtitle format for the html5 video tag if subtitles are provided, as they should be for video content.

[VeryGoodErotica]

Ouch - with a validating .vtt file, epubcheck doesn't complain but it crashes iBooks on a 3rd gen iPad - 3rd gen iPad doesn't get updates, so it looks like audio subtitles will never be usable on that device.

Not a W3C issue, I know, but that is a bad bug.

[VeryGoodErotica]

Maybe if text/vtt is part of the specified core media types, even though there isn't a specified video type, future ePub viewers will have proper code to handle it and not crash.

[VeryGoodErotica]

Okay it seems iBooks only crashes if the track element has the default attribute set, so those who want to include VTT (works beautifully in Marvin on same device) - don't set the default attribute on any track elements.

Too bad Apple won't fix bugs for that iPad version, but that's capitalism.

[mattgarrish]

The track element is exempt from fallbacks:

The following [HTML] elements can refer to Foreign Resources [EPUB32] without having to provide a fallback Core Media Type Resource:

• link — when its rel attribute has the value "pronunciation"

• track

• video — including any child source elements

https://www.w3.org/publishing/epub3/epub-contentdocs.html#sec-xhtml-fallbacks

[VeryGoodErotica]

Okay I was looking at this section which references video and not needing a fallback but makes no reference to the track element:

https://www.w3.org/publishing/epub3/epub-spec.html#sec-publication-resources

That's within the section on creating a manifest, where a vtt file obviously needs to be specified. So perhaps that section needs to be updated to match the contentdocs page.

ePub viewers that do handle video though really should be required to handle both VTT and probably TTML so they still should (imho) be core media types so that crashing viewers, like iBooks, can be declared non-conforming. Otherwise authors will be discouraged from using the track element.

[iherman]

This issue was discussed in a meeting.

• RESOLVED: Add WebP as a core media type in EPUB 3.3

View the transcript

WebP and VTT as media types
Wendy Reid: #1344
Wendy Reid: #1299
Wendy Reid: caniuse looks good for WebP
… supported in Big Sur, which is out soon (?)
… the argument is a more efficient image format
… DeMarque feels it significantly reduces the overall size of their EPUBs
… which is good
… the other possible CMT is VTT,
… used for video captions
Marisa DeMeglio: if we had web vtt as a core type
Wendy Reid: it’s text vtt
Marisa DeMeglio: does it accompany the video?
Wendy Reid: the mystery is how we refer to videos; we don’t have a CMT for video
… it would accompany whatever video format you were using
Shinya Takami (高見真也): (speaks in Japanese)
… I discussed both of these with Mr. Koike… we don’t know what impact they would have on RSs
Brady Duga: Matt seems to think a fallback isn’t needed in this case?
… since VTT is itself a fallback for the track element?
… do we really need to do anything other than clarifying the manifest section
Wendy Reid: that’s a task on it’s own
Brady Duga: that’s why we have matt :)
… you don’t need a fallback for a foreign resource in this case
… so it might make it confusing to have it as a CMT
Dave Cramer: This is the perfect opportunity to mention that CMT doesn’t mean that something can be a spine item
… Content Docs can be spine items, but CMTs mean they don’t need a fallback
… a RS would be assured of finding something to display
Brady Duga: that’s a good point, but you could put vtt in an object tag :)
… the real problem is, if we required a fallback for VTT, there isn’t one.
… so you couldn’t use VTT. There’s not a format that would work instead. You can’t replace it with HTML.
… I’m assuming it has time-based features.
… if it does require a fallback, we should make it a CMT. If it doesn’t require a fallback, there’s no point.
… thumbs up to WebP
Shinya Takami (高見真也): how about adding 2nd-priority CMT
… maybe JPEG/PNG is already popular
… webp and opus are new technologies, maybe there should be fallbacks
Zheng Xu (徐征): If a CMT can be a spine item…
… what is the core media type we wanted to define to have benefit for end user
… we can use it as content doc without fallback
… if CMT does not have this purpose, even some mimetype can be used as CMT but can’t be put in spine item
… how do we use this scope?
… like web browser we have caniuse
… if we have something like caniuse for reading systems
… but if some reading system can’t support webp, it’s a reading system problem
… we need to draw a line for CMT to ask creator to add fallback
… it’s still a question for me
Wendy Reid: it’s a fair question
… having tests will help
… as we discussed earlier, do we throw away the idea of CMT, and the consensus was we didn’t want to do that.
… this puts a lot of pressure on content creators, and leads to interop problems.
Brady Duga: re: fallbacks for webp and opus
… defeats the whole purpose, which is to make your EPUB smaller
… so that’s why you want to extend CMTs
… a size benefit
… we can add to our list of reasons: Is there a benefit you cannot achieve with existing core media types
… the answer for both WebP and Opus is that they are smaller.
… we have opened things up, it was called FXL
… we have insane hacks that downloads Times New Roman so that it can match fruit company’s rendering
… free-form content makes it hard to support lots of things
… we also don’t have enough people to make caniuse for epub work
Zheng Xu (徐征): caniuse for RS… I know it would take a while
… they have more content creators
… we have smaller scope of user
… say user wanted to use some type of… do whatever you want
… use any font, but it might break FXL… sure
… if we can define certain type… with web browsers people only test in 3 browsers
… for reading systems it’s not the same
… we have too many types of reading systems
… as RS creator, if we cannot support it, we can put our support status in caniuse
… it’s kind of more use-case driven
… if we draw a line here,
… how can create a fallback for it?
… this fallback blocks creator from using this feature
… it’s not possible to create fallback
… I think purpose of creator and spec and RS
… expand it, let it be more easy to use
Dave Cramer: I would just remind everyone we tried to do caniuse for EPUB and it failed epically
… there’s a handful of major browsers, and hundreds of reading systems
… which all behave differently on different platforms
… the volunteers could never keep up
… we failed at trying to keep up with it and maintain
… we need more marisas
Marisa DeMeglio: you took the words out of my mouth
… I developed the site
Marisa DeMeglio: https://daisy.github.io/old-epub3-support-grid/features/
Marisa DeMeglio: keeping it up to date was not possible
… we relied on volunteer testers; the tests were not automated
… even though we had some provisions to make it easier
… the rate at which reading systems were changing required too much retesting
… a couple of months ago, BISG asked me to put the static site back
… i’ve put it back
… link ^^^
… this only scratches the surface
… but this is what we tried to do for caniuse for epub
Zheng Xu (徐征): I understand; I respect the amount of work required
… two things
… one, in terms of caniuse or test
… if it needed to be maintained by each RS
… publishers would know how to create content for a RS
… we might have more than a hundred reading systems. No 3rd party can do that.
… webp… based on the current track, it’s hard to add more to CMT
… what is benefit of having this CMT?
… we have spec, we have 3.3 that defines that webp must be CMT
… but we have a RS that does not support this
… that is a problem
Wendy Reid: we have to get back to the core of the core of the talk about core media types
… it sounds like we have consensus on criteria
… they need overwhelming support by browsers and operating systems, with a path to completion
… and they need a direct benefit to content creators and/or reading systems
… based on those criteria, let’s start with WebP
… I’ll put in a proposal
Proposed resolution: Add WebP as a core media type in EPUB 3.3 (Wendy Reid)
Brady Duga: +2 (1 for me, 1 for Garth)
Teenya Franklin: 0
Laura Brady: 0
Shinya Takami (高見真也): +1
Matthew Chan: +1
Marisa DeMeglio: +1
Zheng Xu (徐征): +1
Yu-Wei Chang (Yanni): +1
Ben Schroeter: 0
Wendy Reid: +1
Resolution #1: Add WebP as a core media type in EPUB 3.3
Wendy Reid: I can’t make a strong case for VTT
Dave Cramer: I don’t think we should vote on this right now
… it might not need to be a CMT
… it should be available in EPUB
… let’s figure out how to do that

[iherman]

The issue was discussed in a meeting on 2020-12-04)

List of resolutions:

• Resolution Disallow empty content for the four required "dc:" metadata items in OPF #1: EPUB 3.3 will keep the concept of core media types as it is today.

View the transcript

1. Core Media Types

See github issue #1344, #1299, #645.

Wendy Reid: we had discussed this back in sep. that we would vet them one at a time
… we've tried to add some ones
… but run into issues with things that are implemented, but not standardized

Dave Cramer: the general question here: We have 2 competing principles. 1 is supporting what the web supports, and we hope that epub will maintain compatibility with the web.
… 2. but epubs also must work. e.g. someone who buys a book should be able to read it on their device
… an issue esp. because older RSes are still out there
… e.g. I made a little test of webp inside an epub, and it only worked on 50% of the RSes it was tested on
… we could give up on the idea of core media types and just leave the decision to content authors, but that could result in a bad experience

Ivan Herman: we have already made the similar decision in terms of css
… e.g. whatever css can do, it is fair game for authors

Dave Cramer: i think css is different. CSS has very well defined fallback behaviour. Not true with, e.g., new media type in epub

Tzviya Siegman: +1 to dauwhe about fallbacks

Dave Cramer: with CSS, the reading experience may be degraded, but not entirely lost
… there's also lots of experience out there about writing CSS that works even if certain features aren't supported
… not exactly the same with EPUB

Ivan Herman: +1 to dave's response

Brady Duga: agreed
… in terms of modelling epub after CSS, 90% of the issues I fix are to do with CSS not working. So not enthused about using the same model for media types

George Kerscher: where are we in terms of video formats?

Wendy Reid: with video, we've also taken an "open approach" i.e. just use what the RS accepts
… h264 encoded is probably the standard?

George Kerscher: i think the lack of clarity around the video is holding us back. People would love to see video in epubs

Dave Cramer: video is tricky because there are even brand new devices that won't support video - eInk!

Wendy Reid: also, some platforms have upload sizes that make video incompatible

Avneesh Singh: this is a problem i see with both video and audio in epub. The larger the file size of the zipped file, the greater the chances of corruption after download
… going back, we said that epub 3.3 would not be a major revision. And we only have 1 year to finish the spec.
… recall our experience with epub 3.1. The publishing industry, unlike the web, is slow to move to new standards

Dave Cramer: a lot of the trouble with epub 3.1 was with epubcheck not being available (although the spec also had its own issues)
… i think we should keep core media types
… and maybe periodically consider new media types as the underlying technology evolves
… and i think this discussion shows that that decision comes down to the specific media type under consideration

Ivan Herman: i'm okay with what Dave said, but what are the criteria when we decide that something becomes a core media type?
… earlier there was a request that webp to become a core media type, which we accepted without lots of discussion, but now (I believe) there's still an open PR about it

Bill Kasdorf: when we originally created core media types, we still allowed the use of other media types, just with the caveat that the author must provide a fallback, true?
… yes, okay

Hadrien Gardeur: but we shouldn't always assume that there will be a working group to oversee the question of core media types, especially with newer media like video/audio
… perhaps we can have a normative document where we would retain the capability to update it when we need to (i.e. between working groups)

Garth Conboy: we currently have a list of types for image, a list for audio, and a vague suggestion for video
… but I don't think the current state of support for video is broken right now
… about what Hadrien said, maybe it could be an external vocabulary document?

Garth Conboy: See the relevant section in the 3.2 spec

Ivan Herman: about Hadrien's point, 1. The new process at W3C will make these types of updates much much easier. Under the new process we can update the spec if there is committee agreement.
… 2. But we also have the option to separate the media type into a separate registry. The W3C will have a more formal way to update registries in the future.

Brady Duga: I like the idea of pulling out the media types to a separate location rather than having them buried inside the main spec
… re. Bill's Q about fallbacks. The specific issue with webp is that webp makes images smaller. If authors had to provide fallbacks when using webp, that would kind of defeat the point (by expanding the epub size)
… there seems to be some disagreement about how to add new core media types

Avneesh Singh: we already tried what Hadrien suggested in epub 3.1, but then with epub 3.2, we put it back in
… externally incorporated documents created additional issues when it came time to take 3.2 to ISO
… maybe we could ask Makoto whether whatever we decide here will create an issue for ISO

Dave Cramer: yes, external registries have definitely been an issue for specs in the past
… and audio and video formats are more amenable to being remediated by fallbacks than images (the fallback can even just be text)

Ivan Herman: to wrap up, we seem to be converging towards the point of not changing anything for now
… we keep core media types as they are today and move on (and perhaps changes in the W3C processes will make these easier to maintain going forward)

Wendy Reid: yes, we want to keep core media types, and we will wait and see about the idea of using external registries
… esp. given that how registries will work in the future is still being sorted out
… and the Process 2020 will allow us to make piecemeal modifications to the spec without revisiting everything
… we're still in favor of adding webp, we just need to work out the implementation issues around webp

Proposed resolution: EPUB 3.3 will keep the concept of core media types as it is today. (Ivan Herman)

Ivan Herman: i think webp is a separate discussion

Tzviya Siegman: +1

Ivan Herman: +1

Matthew Chan: +1

Brady Duga: +1

Hadrien Gardeur: 0

Wendy Reid: +1

Garth Conboy: +1

Charles LaPierre: +1

Juliette McShane: +1

Avneesh Singh: +1

Toshiaki Koike: +1

Bill Kasdorf: +1

Masakazu Kitahara: +1

Garth Conboy: and for video, we're going to keep it as it is for now - i.e. no specific type, just a suggestion

Hadrien Gardeur: isn't that an inconsistency? There's core media types for some types of content, but not for video?

Dave Cramer: In the past that was because video types were evolving so quickly

Hadrien Gardeur: Images seem to be evolving quickly today

Dave Cramer: In case of conflict, consider users over authors over implementors over specifiers over theoretical purity.

Resolution #1: EPUB 3.3 will keep the concept of core media types as it is today.

George Kerscher: +1

Wendy Reid: Well with image elements, there are a robust assortment of image fallbacks

Gregorio Pellegrino: +1

Brady Duga: Let's please try to keep substantive discussion out of the irc chat. Let's keep the irc chat just for metadata about the meeting only please!
… everyone may not be closely watching the chat log