manifest JSON type property with AudioBook from Schema.org does NOT mean the "audio book" "kind" (same issue for differentiating comics/mangas)

[danielweck]

The manifest JSON type property with the AudioBook value (from Schema.org) does NOT mean the "audio book" "kind" (withreadingOrder audio-only files)
...consequently, as a reading system / user agent I cannot infer the "audio book" "kind" of Web Publication (which logically / semantically should consist only in audio PublicationLinks in the readingOrder).
For example, a "sync media" Web Publication (or any other type that needs to use properties from the Schema.org AudioBook, such as readBy or duration) would have type = AudioBook! (yet the readingOrder would potentially consists of HTML files)

{
    "@context": [
        "https://schema.org",
        "https://www.w3.org/ns/wp-context"
    ],
    "type": [
        "Book",
        "Audiobook"
    ],
    "...": "..."
}

or in short (because Audiobook is Book in Schema.org):

{
    "@context": [
        "https://schema.org",
        "https://www.w3.org/ns/wp-context"
    ],
    "type": "Audiobook",
    "...": "..."
}

[danielweck]

So, as a reading system that needs to display UX / affordances that are audio (or image, for comics) -centric, how do I infer the "kind" "audio book" (or "comic/BD/manga book") from a Web Publication manifest?
I cannot rely on type = AudioBook, so do I check that every PublicationLink in readingOrder is encodingFormat = audio/xxx (or image/xxx)?
Problem: encodingFormat is optional. Also, on the web URLs resolve to HTTP response Content-Type which can be 404 text/html (for instance), not necessarily the expected 20x audio/mp3 (for example).

[danielweck]

Related issue (mixed media types in readingOrder):
#322

[danielweck]

Alternative option: we might reach a stage when the "packaged" form of a Web Publication with "kind" = "audio book" (or "comic book") will have a registered media type (e.g. application/vnd.wp-audiobook+zip) and a recognized file extension (e.g. .wp-audiobook) which MAY be how reading systems identify the "kind" of Web Publication in order to launch the appropriate user experience / affordances.

[iherman]

It is correct, that the default schema.org behavior is such that the presence of an audiobook type only results in a control whether certain type of properties are allowed or not (readBy is the typical case). However, it does not preclude that our spec would specify a specific behavior for the user agents we define. We have to make that explicit, yes, but I do not see this is as a problem.

[danielweck]

So, let's consider this hypothetical instance of a "sync media" Web Publication:
(type = AudioBook, yet the readingOrder contains HTML resources)

{
    "@context": [
        "https://schema.org",
        "https://www.w3.org/ns/wp-context"
    ],
    "type": "Audiobook",
    "url": "https://publisher.example.org/sync-media-demo",
    "name": "WP sync-media demo",
    "author": "Somebody",
    "readBy": "Somebody Else",
    "duration": 12345.67,
    "readingOrder": [
        "...",
        "preface.html",
        "...",
        {
            "type": "PublicationLink",
            "url": "chapter1.html",
            "encodingFormat": "text/html",
            "sync-media": {
                "type": "PublicationLink",
                "url": "sync-media/chapter1.json",
                "encodingFormat": "application/vnd.wp-sync-media+json",
                "duration": 123.45
            }
        },
        {
            "type": "PublicationLink",
            "url": "chapter2.html",
            "encodingFormat": "text/html",
            "sync-media": {
                "type": "PublicationLink",
                "url": "sync-media/chapter2.json",
                "encodingFormat": "application/vnd.wp-sync-media+json",
                "duration": 67.89
            }
        },
        "...",
        "postface.html",
        "...",
    ],
    "resources": [
        {
            "type": "PublicationLink",
            "url": "table-of-contents.html",
            "encodingFormat": "text/html",
            "sync-media": {
                "type": "PublicationLink",
                "url": "sync-media/table-of-contents.json",
                "encodingFormat": "application/vnd.wp-sync-media+json",
                "duration": 432.1
            }
        },
        "...",
        {
            "type": "PublicationLink",
            "url": "sync-media/chapter1.mp3",
            "encodingFormat": "audio/mpeg",
            "duration": 123.4
        },
        {
            "type": "PublicationLink",
            "url": "sync-media/chapter2.ogg",
            "encodingFormat": "audio/ogg",
            "duration": 567.8
        },
        {
            "type": "PublicationLink",
            "url": "sync-media/table-of-contents.mp4",
            "encodingFormat": "audio/mp4",
            "duration": 87.65
        },
        "...",
        "cover.jpg",
        "styles.css",
        "...",
    ],
    "...": "..."
}

Here is a possible solution to this problem (please let me know if this is feasible): we could define our own SyncMediaBook type that would "extend" Schema.org's Book (just like AudioBook does), and that would support the existing Schema.org definitions of duration and readBy. In fact, we already plan to use this extension mechanism to attach the existing Schema.org duration property to the WP-specific PublicationLink type, right?

https://pending.schema.org/duration
The section "Used on types defined in extensions" would include not only AudioBook but also PublicationLink and SyncMediaBook (hypothetically).

https://bib.schema.org/readBy
The section "Used on these types" would include not only AudioBook but also SyncMediaBook (hypothetically).

So, assuming that ; in the context of Web Publications ; we are indeed allowed to do the following:

• refine the semantics of Schema.org's AudioBook type (in order to set expectations about audio resources in the readingOrder)
• create a new (for example) SyncMediaBook type (in order to differentiate from AudioBook, whilst inheriting similar/related Schema.org semantics)
• create a new (for example) ComicBook type (in order to set expectations about image resources in the readingOrder)

...then reading system implementors will be able to detect the "kind" of Web Publication in order to instantiate the appropriate ; potentially-specialized ; user experience (i.e. audio-centric vs. image-centric vs. generic Web Publication). Note that the hypothetical "sync media" kind of WP does not actually require significant UX adaptations, as this is based on the principles of "progressive enhancement" / feature overlay (the readingOrder of a "sync media" WP contains regular HTML resources, the TOC processing model remains generic, etc.).

[iherman]

So, assuming that ; in the context of Web Publications ; we are indeed allowed to do the following:

• refine the semantics of Schema.org's AudioBook type (in order to set expectations about audio > resources in the readingOrder)
• create a new (for example) SyncMediaBook type (in order to differentiate from AudioBook, whilst inheriting similar/related Schema.org semantics)
• create a new (for example) ComicBook type (in order to set expectations about image resources in the readingOrder)

These are all doable. If you remember, this is one of the issues we discussed with DanBri at the F2F: we can add our own, say, SyncMediaBook to the type hierarchy (and add our own semantics/behavior to it).

DanBri's advice was to use wikidata for this, ie, a place where we could add our own terminology. We used the example of a Proceedings: the idea was that Wikidata already has a term for this: https://www.wikidata.org/wiki/Q1143604. This could be used as a URL for a class (we would put an alias in our context file to use the string 'Proceedings'). SyncMediaBook does not yet exist in Wikidata but, I guess, the idea was that we could add that item ourselves (Wikidata is usable for everyone, just like wikipedia) and take it from there.

[wareid]

As discussed in the AudioTF call on Nov 16th, I've added this to the queue to be discussed in the PWG call in coming weeks.

[wareid]

Explainer for WG Call Dec 10
Schema.org currently has type values for "Book" and "Audiobook" with clearly defined semantics, but they do not fully address the possibilities of syncMedia or Comics, or that user agents might rely on type for generating an appropriate reading experience.

Should we add new type hierarchies to the spec to allow user agents to clearly identify what content they are receiving and respond accordingly? This is an appealing proposal as can define it within wikidata and use it within the spec.

[nruffilo]

Based on the discussions we had in our weekly call, I'd like to propose some wording we can vote on. First, I want to lay some items. I am slicing the world into generic User Agents (a default web browser, or random in-app webview), and dedicated reading systems. At a minimum, I want to have generic User Agents able to open and play content. So, as long as the UA can determine that the content is a WP, has a logical reading order, and can serve up that content (HTML, audio files, etc) in an accessible way, that meets the minimum requirements.

For dedicated reading systems, they would want/need additional information that can let them set up additional functionality. For example, panel-by-panel flow with comics/manga, additional audio features for an audiobook, etc. This sort of functionality should not be in spec as it's the secret sauce that makes these apps worth downloading/buying, and the spec should focus on ensuring content is accessible in a logical way. That said, I think it's reasonable to have optional fields in spec that allows authors to identify their content more specifically allowing reading systems to create enhanced functionality.

The proposed language (to be wordsmithed):
"Type remains unchanged. A User Agent should be able to serve up all content, in the order specified in the reading order, to a user, whether that content be a list of HTML, images, or audio.

Authors & User Agents can use the BookFormatType https://schema.org/BookFormatType to specify a more specific type of book. An authoring guide should be created that lists common types, but if an author/publisher wishes to work with a reading system to create additional custom types to allow for additional functionality within the reading system, that is fine."

[iherman]

I must admit I did not realize the existence of that BookFormatType. I think this may be indeed a good alternative to strict typing for things that we are talking about.

Thanks @nruffilo

[llemeurfr]

@nruffilo using BookFormatType seems interesting but it does not free authors to type an audio publication structure as Audiobook because of the use for properties such as readBy or duration in the jsonld structure.
If we go this way, what is the fate of a SyncMedia book? Is it of type Audiobook (because of requested properties) and BookFormatType SyncMedia?
Won't authors be totally lost in such subtleties?

[nruffilo]

@llemeurfr Thats a great comment. I'm not sure the right answer, but it sounds like having "SyncMedia" as a BookFormatType seems logical enough. I'm not sure where readBy and duration would fit, but I have a feeling others have suggestions about that.

Doing it in this way will require us to educate authors about the subtleties, although it would be good to make sure that we're not being redundant or offering multiple ways of doing things. If we can clearly state that "for all synced media, you want type 'Audiobook' BookFormatType 'SyncMedia'" I think that's easy enough. The question then comes - is there ever a type "Audiobook" that is NOT a SyncMedia. Unless that is the case, then why would we need to separate or use BookFormatType? If there is a case between the two, then I think it's fair and reasonable to expect that authors could utilize both.

[wareid]

PROPOSAL: Audiobook profile would use the type audiobook to declare to user agents the contents are primarily audio. For other types like SyncMedia or Comic, use custom classifications.

[iherman]

@wareid by "custom classification" I presume you mean that we would define additional types for cases that are not covered by schema.org. If that is indeed the intention, then 👍 to your proposal.

[wareid]

@iherman That is indeed what I am suggesting :).

[css-meeting-bot]

The Publishing Working Group just discussed https://github.com/w3c/wpub/issues/351.

The full IRC log of that discussion

<dauwhe> topic: https://github.com//issues/351
<bigbluehat> wendyreid: if there are no strong oppositions, then I'm going to close this as resolved
<dauwhe> Github: ttps://github.com//issues/351
<dauwhe> Github: https://github.com//issues/351
<wendyreid> Resolution: Audiobook profile would use the type audiobook to declare to user agents the contents are primarily audio. For other types like SyncMedia or Comic, use custom classifications.
<bigbluehat> q+
<bigbluehat> q-
<bigbluehat> wendyreid: next issue
<wendyreid> https://github.com//issues/322

[wareid]

RESOLVED. See notes above.

[iherman]

This issue was discussed in a meeting.

• RESOLVED: Audiobook profile would use the type audiobook to declare to user agents the contents are primarily audio. For other types like SyncMedia or Comic, use custom classifications.

View the transcript

1.1. Issue #351: manifest JSON type property with AudioBook from Schema.org does NOT mean the “audio book” “kind”
Wendy Reid: #351
Wendy Reid: this was originally raised about synced media
… the profile would use the type of “audiobook”
… and other profiles would get their own classification (via this type)
… we would define new types when not already defined via schema.org
… and someone pointed out we can use wikidata for this
Wendy Reid: if there are no strong oppositions, then I’m going to close this as resolved
Resolution #2: Audiobook profile would use the type audiobook to declare to user agents the contents are primarily audio. For other types like SyncMedia or Comic, use custom classifications.

[danielweck]

The resolution makes sense. Thank you :)