Should the packaging spec include processor conformance requirements?

[llemeurfr]

The OCF spec contains high level conformance requirements for reading systems.
Should we expressed something alike in the lightweight packaging spec?

For instance would it be a good place to state that if the PEP is not present in the Package, a Processor MUST create one on the fly if its unzips the archive, so that the end result is a propre Web Publication?

[iherman]

On the specific example: yes, in principle, but only if the processor implements some sort of a 'service' to unpack an LWP. I am not sure many reading systems would do that; maybe some browser implementations may, akin to the archival function some of them provide.

The issue with the extra PEP generation is that I would think most of the unpacking cases are when a user 'just' uses an off-the-shelf unzip application...

[llemeurfr]

Re. User Agent Conformance, an LPF processor is not necessarily an audiobook player. If can be an "audiobook server" exposing Web Publications, or it can simply be a converter to another format. On the reverse, a web browser, even with the help of javascript, will not be an LPF processor: web browsers will consume web publications of type Audiobook, which is largely different (and complementary).

We could simply formalize this as ...

A compliant User Agent MUST meet at least one of the following criteria:

• It is able to import the Package and fulfill the requirements of an audiobook player as defined in [[wpub-ucr]] ;
• It is able to expose the Package as a W3C Web Publication, as defined in [[wpub]];
• It is able to convert a Package to an alternative audiobook format suitable for electronic distribution.

Note: [[wpub-ucr]] will point to https://w3c.github.io/dpub-pwp-ucr/#audiobook-player

What do you think?

[iherman]

@llemeurfr

• per (1) is fine if we are talking about an audiobook. But the LPF is not only about audiobooks; maybe we should be a bit more general about user agent behaviour as is/will be described in the ucr, and refer to audiobook as a specific, and important, example.
• per (2) What does "expose the Package as a W3C Web Publication" mean? Does it involve the on-the-fly generation of a PEP in case there isn't any? I think this should be mentioned.

[llemeurfr]

I just added a proposal in the draft, taking @iherman 's comments into account.

[llemeurfr]

There may be something else to add to the conformance requirements: a rule about missing resources (indicated in the manifest but missing in the zip) and another about extra resources (present in the zip but missing in the manifest).
In the first case, the safe side is to consider the Package invalid -> stop its processing.
In the second case, it is to consider that the extra resources are not harmful and consider the Package valid.

[iherman]

This issue was discussed in a meeting.

• No actions or resolutions

View the transcript

Should the packaging spec include processor conformance requirements?
Laurent Le Meur: #36
Laurent Le Meur: https://w3c.github.io/publ-epub-revision/epub32/spec/epub-ocf.html#ocf-conformance-rs
Laurent Le Meur: next is issue #36 a little more complex. There were high-level requirements that are there…
A compliant User Agent MUST meet at least one of the following criteria:
It is able to import the Package and fulfill the requirements of an audiobook player as defined in [[wpub-ucr]] ;
It is able to expose the Package as a W3C Web Publication, as defined in [[wpub]];
It is able to convert a Package to an alternative audiobook format suitable for electronic distribution.
Laurent Le Meur: We want to keep something about conformance of reading systems/user agents but we don’t want the words that are in the OCF document. I tried to express compliance with what is pasted in IRC now…
Dave Cramer: These kinds of - we have lots of statements of these kinds across the specs. With the eye of writing tests, it makes it really hard to write a test for these sort of things. If I was building an implementation, what would I have to do. I don’t have answers but I have a general concern that we need testable assertions
Laurent Le Meur: I agree
Laurent Le Meur: a rule about missing resources (indicated in the manifest but missing in the zip) and another about extra resources (present in the zip but missing in the manifest). In the first case, the safe side is to consider the Package invalid -> stop its processing. In the second case, it is to consider that the extra resources are not harmful and consider the Package valid.
Laurent Le Meur: This sort of definitions of packaging - it’s not a conformance. There is an alternative proposal that I’m writing now. (pasted in IRC)
… we could consider an UA as a parsers - and if it see missing items, it should raise an error and consider the package damaged.
… not damaged - invalid
… For me it’s better for it to be boolean - it is either good or bad.
Dave Cramer: Once again we’re faced with the collision of two worlds. The Web tries to route around any damage from network to HTML to CSS. If there is something missing, it tries to keep going the best it can. In the XML world, if a file isn’t well formed, the parser must refuse…
… epub has formal validation - but we’re relatively silent as to how reading systems should conform to this. I would lean towards not requiring user agents to give up if they see an error.
Nick Ruffilo: What if we think in terms of the opposite, if an error is found, we focus on the resource being available and playable. If a valid audio file is provided, it has to be played in the reading order.
… we provide the files in order, can they be played ,that is a test case
Garth Conboy: I think what Nick may make sense at the audiobook layer, but for the lightweight packaging format, i’m not sure that it fits in the packaging spec. Listening to both dave and Laurent’s comments on different sides of requirements./
… for packaging, I can see being stricter - maybe lean more towards Dave take at the audiobook level being more permissive…
Bill Kasdorf: Since the audiobooks is basically a profile of the master WP constellation of specs - my concern is having a profile that undoes something that it is a profile of. I think of a profile that adds something not that makes the master invalid.
Wendy Reid: The packaging spec isn’t a profile of the main spec, it’s a companion
Bill Kasdorf: I wanted to make sure we weren’t creating a conflict there.
Wendy Reid: in this case, the packaging spec is a companion, but it may be replaced if web packaging becomes a thing for example.
Laurent Le Meur: I agree with Nick to think positively about the user agent where it has to do something if resources exist. My only worry is that we have to address user agent conformance.
… We could end up with a user agent processing model. That it has to open the manifest and process the reading order. Do we want to go there?
… especially since we already have a processing model in the WP spec.
Wendy Reid: Should the packaging model not differ to the WP spec? Wouldn’t need to be in the packaging spec…
Laurent Le Meur: What is the processing model in the main spec. I’m not sure we have spec’d anything about that yet.
Wendy Reid: It would fall into the web publications.
… I think the packaging spec should just reference the processing model of web publications and use that. It seems the clearest way to proceed
Laurent Le Meur: I can try - it’ll be two lines.
Wendy Reid: There are lots of items in the audiobook spec that feel the same

[llemeurfr]

Taking the last discussion of the group as base for modification of the conformance section:

WP user agent conformance criteria are found in https://w3c.github.io/wpub/#conformance-classes.
It seems useful to add to our LPF conformance criteria the capability to process a Package (seems obvious but specs still express such things clearly).

The resulting proposal is (re-using the style of the WP conformance section):

A User Agent conforms to this specification if it meets the following criteria:

• It is capable of processing a conforming Package.
• It fulfills the conformance criteria for Web Publications as specified in Web Publication Conformance Classes.

I also removed mentions of "LPF Processor" in favor of "user agent".

-> see https://raw.githack.com/w3c/pwpub/master/spec/ocf-lite.html#sec-ua-conformance

[iherman]

This issue was discussed in a meeting.

• No actions or resolutions

View the transcript

2.1. ua conformance
Laurent Le Meur: #36
Laurent Le Meur: Link to the draft spec in minutes. First item #36 - followup on discussion about conformance requirements for user agents. What should we include inside the packaging specification. What I did was to specify that - simply - a user agent is conformant if it’s capable of processing a conformant package…
… and if it fulfills the criteria specification. And then it links to the W3C specification.
… That way we could rely just on the WP specification. We added a line to ensure that the User Agents conform to the LPF spec.
Laurent Le Meur: https://raw.githack.com/w3c/pwpub/master/spec/ocf-lite.html#sec-ua-conformance
Laurent Le Meur: This is something you find in section 3. (link in chat) If you agree, we can move forward and close this issue
Ivan Herman: +1
Luc Audrain: +1
Nick Ruffilo: +1
Nellie McKesson: +1
Wendy Reid: +1
Ric Wright: +1
Benjamin Young: I was just looking where processing is defined.
… Is it just unzipping - or is it other things
… ‘it is capable of processing a conformant package’ what do you mean by processing
Laurent Le Meur: there is no strict definition - just following the rules of section 2 of the same spec.
… But I don’t see a big semantic difference between handling and processing
Benjamin Young: Is an LPF compliant processor going to turn this stuff into something else, and what is that?
Laurent Le Meur: we decided not to define an LPF processor that turns the package into something else. Most would turn it into a WP, but we won’t exclusively add it. If we only make a reference to the user agent conformance, there would be no mention of the file name (index.html or publication.json)
… we can come back to that later - by making a reference to the conformance of WP, we will have to go through - step by step retrieval of the manifest…
Tzviya Siegman: I hear Benjamin’s concern, but I’m not sure what we can put in there to address it.
Benjamin Young: Zip has a clear intention, but is this meant to be like zip, where it’s just a transfer format, and you explode it, then process it… but if it’s going to a mobile app, there are expectations. I realize the scope here is mixed because it’s tied to web publications.
… I’m just - since these are conformance requirements, these are not a testable phrase.
Luc Audrain: This document defines a file format and processing model. I suppose we should have something to explain the processing model?
Laurent Le Meur: the previous decision was to rely on the processing model.
Luc Audrain: so maybe we add a line pointing to the web publication spec? It will help the concern perhaps?
Tzviya Siegman: We’re then defining two different processing models in one document. In 2.1 - a web publication lightweight package - we’re already referencing the zip spec.
Ivan Herman: There is an LPF processing model - all it is is the zip processing format that is defined elsewhere. The result is processed as defined. In a sense, this does not define it’s OWN processing model separate, just the combination
… We might have to be a little more clear about the two different processing models, but that is the intention
… Maybe the user agent conformance is the first bullet item - we can make it more clear that the user agent must follow whatever is described in the zip - must unpack the content, and the content must follow web package conformance.
Laurent Le Meur: this is only one type of processing - unzip, explode, use after that, but there might be other types of processes.
Ivan Herman: I think what we have to be careful is that we define a conceptual processing engine, but can it be implemented. Conceptually speaking - per spec, the result should be a web publication
… if someone does something different inside, that’s not for us to decide
Laurent Le Meur: I would say that reading or exploiting the zip is possible.
Benjamin Young: I was going to suggest that we use the same phrasing that the zip spec uses when it refers to accessing the contents of files. Laurent is right - unzipping is a specific step, but there are other ways to access files.
Tzviya Siegman: https://raw.githack.com/w3c/pwpub/master/spec/ocf-lite.html#sec-zip already exists in the document
Benjamin Young: but we have to make sure you can go from that knowledge to whatever is in the publication. The discovery methods seem different - so some sort of processing description determining the difference between the zip and the web publication - that would be really helpful
Laurent Le Meur: Benjamin, do you think we should write the processing model in this document?
Benjamin Young: in so much as it differs. In the case where there is a missing primary entry page - that would be different from standard zip
Laurent Le Meur: there are also differences in the discovery of the manifest. In the web publication document - we use the word ‘fetch’ which is an HTTP word, but in zip you don’t do that, it’s a different concept, so there is something to discuss there.
… the action item i propose is to state that we agree on adding wording that will be modified from the original. Some wording that links to the web publication specification
… we will see in another issue the exact issues we have with the web publication processing model and try to solve this there. I propose we table this issue and see the next issue