Add Opus codec as core media type

[mattgarrish]

A feature request for 3.1 is to add the Opus audio codec as a core media type:

Publishers that rely heavily on Media Overlays to provide synchronized text-audio publications would, in order to achieve reduced file sizes and increased audio quality, benefit from a speech-optimized format as a Core Media Type.

Opus would also represent a royalty-free option to MP3.

Discussions to date have been favourable, as Opus is supported, or indicated to be support, in all major browsers. The working group is not likely to come to a resolution prior to the first draft, however.

A note will be added to the core media type section that Opus is still under consideration and will be resolved in a future draft.

[pettarin]

Why not including also OGG/Vorbis, Speex, SILK, FLAC then?

At least OGG/Vorbis and FLAC are widely supported.

[caraya]

Matt,

Is there a technical reason why you chose Opus as the alternative to MP3 for audio? Has there been a comparison of formats that led to that conclusion?

[mattgarrish]

We haven't actually added opus as a core media type at this point; I opened this issue so that everything under discussion is in the tracker.

A call was put out to the epub community to make feature requests prior to the start of the revision, and a speech-optimized audio format was one of the requests.[1] So far, the only interest expressed has been for opus.

A survey of DAISY consortium producers was done prior to the holidays and there's also outreach to mainstream audio publishers to determine what their needs are. What those efforts reveal is going to be important in making a decision on what to add, if anything. But we haven't returned to this issue in some time, as there are bigger items on the list of todos.

[1] https://docs.google.com/document/d/1QINuKE5Hsemll6G6tSQe6BaRuCHerFuwfXIRTgdaO7o/edit#heading=h.ar9g25b43eyd

[pettarin]

For general-purpose, speech-optimized tasks, Opus has become the de-facto standard, merging and superseding e.g. SILK and Speex, so it is a reasonable choice.

On the other hand, for e.g. music files, having OGG/Vorbis and FLAC would be nice too.

[lloydrasmussen]

Some organizations wish to provide audio at the lowest reasonable bitrate for talking books. At the US National Library Service we have been using AMR-WB+ at 24 kbits/sec at very good quality for the past 10 years. We may not change codecs any time soon. However, Opus appears to have a quality level per bitrate approaching that of AMR Wideband Extended. We are currently conducting some MUSHRA subjective quality tests comparing AAC, AMR-WB+ and constant-bit-rate Opus in this range. Hope to have our results available in a few weeks.
It is interesting to note that although both MP3 and LC-AAC are listed as core media types for audio in Epub 3.01, MP3 is a MUST while LC-AAC is only a SHOULD.
The Opus codec itself is described in IETF 6716. Another internet draft describes how Opus can be encapsulated in OGG-type files and bitstreams; it should be ratified by IETF soon. This second standard would be helpful in providing an ability to seek within a file or bitstream in cases where variable bit rate is used.

[mattgarrish]

This bullet came up on the wg call yesterday:

Unless specified as conditional behavior in this section, it must support all Core Media Type Resources.

I'd read this initially as leaving wiggle room for exemptions like the animation attributes in SVG, but re-reading the specification in more depth it's not the case that core media types have to be supported -- specifically, MP4 audio and PLS lexicons are not requirements.

I went back to the definitions from 3.0.1 and they only indicate that being a core media type means that no fallbacks have to be provided. (The new wording of a CMT needs to be readjusted as it makes a stronger statement.)

In this light, I don't see why we couldn't add Opus as allowed but not definitely required to be supported - the same as MP4.

We're already drawing a distinction between Core Media Types and Core Media Types that have to be supported, in other words.

Otherwise, we need to look at why MP4 and PLS are only a should to support even when the behaviour they're associated with is (I'm thinking patent encumbrance on the former and lack of support on the latter).

/cc @murata0204 @TzviyaSiegman @GarthConboy

[TzviyaSiegman]

Thanks for the research. How do we clarify this in RS requirements?

[mattgarrish]

Assuming we keep the current (3.0.1) definition that core media type only means no fallback, maybe to be absolutely clear:

It must support all Core Media Type Resources unless it does not support an optional behavior (e.g., audio and text-to-speech playback) or support is not required for conformance per this section (e.g., MP4 audio).

I could live with this, but it would be nice if we didn't selectively make a few core media types not required. In other words, we really would be better off without the "or".

Is there a reason we can't bump MP4 audio up to a must to correct?

If we can, the audio bullet can be generalized to:

If it has the capability to render pre-recorded audio, it must support the audio Core Media Types, and should support Media Overlays [MediaOverlays31].

Whenever Opus gets enough support then adding it to the CMT list automatically makes it recognized.

Assuming we can't, the audio bullet would then have to become:

If it has the capability to render pre-recorded audio, it must support the MP3 audio Core Media Type, should support the MP4 and Opus audio Core Media Types, and should support Media Overlays [MediaOverlays31].

(The problem, of course, is that Opus and MP4 become locked into "should" barring another revision.)

PLS is just a headache. Realistically, it never should have made it in without any support and it's somewhat misleading to claim its available when nothing still supports it (just like the SSML attributes). It's great to claim these features enhance accessibility, but they're also something of a headache to have to explain.

[mattgarrish]

It was resolved to defer adding Opus on the July 13 call due to concerns about support on mobile and ios.

When support improves this issue will be revisited. By separating out the core media type list, the codec can be added later without the need for a full revision.

[dauwhe]

It seems that support for OPUS has increased

Opus will officially be supported in some form by every major operating system, and every major web browser

which includes iOS and Android. Should we revisit this now?

[GarthConboy]

I would be leery of this change at this time. Still seems to be "coming", and lots of client running on OSes that don't offer said support.

[avneeshsingh]

It looks that all major operating systems support codec now, although the file name extension is not supported everywhere.
https://en.wikipedia.org/wiki/Opus_(audio_format)#Support

The only argument that I found in favour of including it in core mediatypes is that core mediatypes is now part of main specifications and if we miss including OPUS this time, then it will have to wait for another EPUB version update.

So, the question is how often do we plan to update versions? If EPUB 3.2 will be the latest version for next 2 years, then it is better to consider inclusion of OPUS in current version.

[murata2makoto]

In the last PBG meeting, we discussed maintenance beyond 3.2. As long as we commit to strong backwards-compatibilities, I think that frequent updates have no problems. So, I propose to close this issue for now.

[ghost]

I have a side question here. Why we need to distinguish "Core Media Types" and "Foreign Resource" in epub? Would that actually be "Recommended Media Types"? So would it be better to stay in best practice instead of epub spec?

[GarthConboy]

Core Media Types are those that must be supported, and those to which any non-Core Media Type must fallback. For 3.2, I do not think we should abandon that approach.

[avneeshsingh]

Interesting blog for OPUS in context of Web RTC
https://www.wowza.com/blog/opus-codec-the-audio-format-explained

Support:
https://caniuse.com/opus

In EPUB 3 context, it will significantly reduce the size of Media Overlays publications and publications with audio files.

[GeorgeKerscher]

Indeed, very interesting! What about the status of Video?

[HadrienGardeur]

On a related note, do we have issues for other formats that have been suggested as well? Is there any way that we can aggregate these issues together?

I know for instance that WebP is now finally available on iOS thanks to iOS 14: https://caniuse.com/webp

[wareid]

Resolved in the meeting on 10-09-2020: We will add the media type for OPUS/OGG to the specification with a note explaining the status of support for Safari.

[mattgarrish]

I assume you meant to assign this to me, not close it... :)

[iherman]

The issue was discussed in a meeting on 2020-12-04)

List of resolutions:

• Resolution Disallow empty content for the four required "dc:" metadata items in OPF #1: EPUB 3.3 will keep the concept of core media types as it is today.

View the transcript

1. Core Media Types

See github issue #1344, #1299, #645.

Wendy Reid: we had discussed this back in sep. that we would vet them one at a time
… we've tried to add some ones
… but run into issues with things that are implemented, but not standardized

Dave Cramer: the general question here: We have 2 competing principles. 1 is supporting what the web supports, and we hope that epub will maintain compatibility with the web.
… 2. but epubs also must work. e.g. someone who buys a book should be able to read it on their device
… an issue esp. because older RSes are still out there
… e.g. I made a little test of webp inside an epub, and it only worked on 50% of the RSes it was tested on
… we could give up on the idea of core media types and just leave the decision to content authors, but that could result in a bad experience

Ivan Herman: we have already made the similar decision in terms of css
… e.g. whatever css can do, it is fair game for authors

Dave Cramer: i think css is different. CSS has very well defined fallback behaviour. Not true with, e.g., new media type in epub

Tzviya Siegman: +1 to dauwhe about fallbacks

Dave Cramer: with CSS, the reading experience may be degraded, but not entirely lost
… there's also lots of experience out there about writing CSS that works even if certain features aren't supported
… not exactly the same with EPUB

Ivan Herman: +1 to dave's response

Brady Duga: agreed
… in terms of modelling epub after CSS, 90% of the issues I fix are to do with CSS not working. So not enthused about using the same model for media types

George Kerscher: where are we in terms of video formats?

Wendy Reid: with video, we've also taken an "open approach" i.e. just use what the RS accepts
… h264 encoded is probably the standard?

George Kerscher: i think the lack of clarity around the video is holding us back. People would love to see video in epubs

Dave Cramer: video is tricky because there are even brand new devices that won't support video - eInk!

Wendy Reid: also, some platforms have upload sizes that make video incompatible

Avneesh Singh: this is a problem i see with both video and audio in epub. The larger the file size of the zipped file, the greater the chances of corruption after download
… going back, we said that epub 3.3 would not be a major revision. And we only have 1 year to finish the spec.
… recall our experience with epub 3.1. The publishing industry, unlike the web, is slow to move to new standards

Dave Cramer: a lot of the trouble with epub 3.1 was with epubcheck not being available (although the spec also had its own issues)
… i think we should keep core media types
… and maybe periodically consider new media types as the underlying technology evolves
… and i think this discussion shows that that decision comes down to the specific media type under consideration

Ivan Herman: i'm okay with what Dave said, but what are the criteria when we decide that something becomes a core media type?
… earlier there was a request that webp to become a core media type, which we accepted without lots of discussion, but now (I believe) there's still an open PR about it

Bill Kasdorf: when we originally created core media types, we still allowed the use of other media types, just with the caveat that the author must provide a fallback, true?
… yes, okay

Hadrien Gardeur: but we shouldn't always assume that there will be a working group to oversee the question of core media types, especially with newer media like video/audio
… perhaps we can have a normative document where we would retain the capability to update it when we need to (i.e. between working groups)

Garth Conboy: we currently have a list of types for image, a list for audio, and a vague suggestion for video
… but I don't think the current state of support for video is broken right now
… about what Hadrien said, maybe it could be an external vocabulary document?

Garth Conboy: See the relevant section in the 3.2 spec

Ivan Herman: about Hadrien's point, 1. The new process at W3C will make these types of updates much much easier. Under the new process we can update the spec if there is committee agreement.
… 2. But we also have the option to separate the media type into a separate registry. The W3C will have a more formal way to update registries in the future.

Brady Duga: I like the idea of pulling out the media types to a separate location rather than having them buried inside the main spec
… re. Bill's Q about fallbacks. The specific issue with webp is that webp makes images smaller. If authors had to provide fallbacks when using webp, that would kind of defeat the point (by expanding the epub size)
… there seems to be some disagreement about how to add new core media types

Avneesh Singh: we already tried what Hadrien suggested in epub 3.1, but then with epub 3.2, we put it back in
… externally incorporated documents created additional issues when it came time to take 3.2 to ISO
… maybe we could ask Makoto whether whatever we decide here will create an issue for ISO

Dave Cramer: yes, external registries have definitely been an issue for specs in the past
… and audio and video formats are more amenable to being remediated by fallbacks than images (the fallback can even just be text)

Ivan Herman: to wrap up, we seem to be converging towards the point of not changing anything for now
… we keep core media types as they are today and move on (and perhaps changes in the W3C processes will make these easier to maintain going forward)

Wendy Reid: yes, we want to keep core media types, and we will wait and see about the idea of using external registries
… esp. given that how registries will work in the future is still being sorted out
… and the Process 2020 will allow us to make piecemeal modifications to the spec without revisiting everything
… we're still in favor of adding webp, we just need to work out the implementation issues around webp

Proposed resolution: EPUB 3.3 will keep the concept of core media types as it is today. (Ivan Herman)

Ivan Herman: i think webp is a separate discussion

Tzviya Siegman: +1

Ivan Herman: +1

Matthew Chan: +1

Brady Duga: +1

Hadrien Gardeur: 0

Wendy Reid: +1

Garth Conboy: +1

Charles LaPierre: +1

Juliette McShane: +1

Avneesh Singh: +1

Toshiaki Koike: +1

Bill Kasdorf: +1

Masakazu Kitahara: +1

Garth Conboy: and for video, we're going to keep it as it is for now - i.e. no specific type, just a suggestion

Hadrien Gardeur: isn't that an inconsistency? There's core media types for some types of content, but not for video?

Dave Cramer: In the past that was because video types were evolving so quickly

Hadrien Gardeur: Images seem to be evolving quickly today

Dave Cramer: In case of conflict, consider users over authors over implementors over specifiers over theoretical purity.

Resolution #1: EPUB 3.3 will keep the concept of core media types as it is today.

George Kerscher: +1

Wendy Reid: Well with image elements, there are a robust assortment of image fallbacks

Gregorio Pellegrino: +1

Brady Duga: Let's please try to keep substantive discussion out of the irc chat. Let's keep the irc chat just for metadata about the meeting only please!
… everyone may not be closely watching the chat log

[dlazin]

In w3c/epub-tests#12, I added tests for Opus (and other formats). In my limited testing, none of the major RSes on Mac or iOS passed the Opus requirement.

All of these failed:

• Apple Books (OS X) (error)
• Apple Books (iOS) (error)
• Play Books (web) (no audio UI; same issue for MP3/MP4)
• Play Books (iOS) (no audio UI; same issue for MP3/MP4)
• Kobo (iOS) (error)
• Thorium (OS X) (no audio UI; same issue for MP3/MP4)
• Safari (OS X) (error) (testing the XHTML, not the EPUB)

The XHTML does work in Chrome and Firefox, so I think the test is correct (although I could certainly be wrong).

[mattgarrish]

Similar to WebP, I'm going to close this issue but leave it with the "to review" label. We have a note that it's at risk in the spec.