MUST the manifest include information about secondary resources or not? #23
Comments
|
See #22 (comment) in a separate issue that is relevant here |
|
Do we mean secondary resources as currently defined -- needed for the rendering of a primary resource -- or as modified by issue #16 -- essential for the rendering of a publication? That's why my tongue was getting tied in #22. Maybe we should just think of primary resource as any top-level resource for the sake of this issue, and avoid correlating them as required in the default reading order. As I said previously, I'd love to share the pain of manifests and have the author only responsible for listing all the primary resources (as defined in the previous paragraph). To me, that's the minimum we need to establish the bounds of the publication. The user agent can compile (if perhaps imperfectly in some known cases) the dependencies of those resources if it has a need for that level of information (caching, building a pwp, etc.). I'm not against listing secondary resources, and depending on how your WP is created it may not even be an issue to list them, but for simplicity of authoring I suspect it would help to bend a little on an EPUB-like hard requirement to list everything. As others have suggested, maybe not every author wants their content taken offline, or turned into a pwp, or used in a way that a full manifest would help simplify. We still need the bounds, but not every dependency. But I'm repeating myself, so I'll leave my comments at that. |
|
@iherman wrote:
And I (and others) disagree. I am fine with this as a should - where an author/publisher can do that if they see fit, but I see absolutely no reason to mandate it. In fact, were we to connect this particular requirement with the offline one (as some have done) - then I even more strongly oppose the mandating. Why? Because the ability to not include a secondary resource would be the WP equivalent of "Cache-Control: no-cache" (aka do not cache/offline).
Why? I don't understand this requirement |
|
@lrosenthol, my problem is: how would you ensure that a WP (as a collection of resources) would not, via for example a reference to a wikipedia page, essentially pull 'in' the whole Web? Ie, although not mathematically but practically be infinite? What would be of any practical use of such a WP? But I see your point of the "no-cache", so the issue is how we reconcile these two things, and I am not sure exactly how. At the moment, secondary resources are those that are necessary to render the publication, wherever those resources are. Maybe we should restrict that definition to those resources that are to be used offline, or something like that; after all, the resources that are not to be "cached" are "just" Web resources that are not specific to the WP. My experience comes from the program I wrote to turn W3C specs into EPUB. I had to collect all the resources (primary or secondary) and, of course, the secondary resources were the problematic ones. I had to use a heuristic (which works for that specific application) to list only those resources that are in the same directory tree than the document itself, plus some common W3C CSS files. Thereby I could keep things finite. Otherwise I simply do not know how I could have created an EPUB3 file, with some specs referring to (indirectly) zillions of other pages. What I am looking for is a framework for that type of heuristics which, I believe, is necessary. |
|
On Wed, Aug 9, 2017 at 9:55 AM, Ivan Herman ***@***.***> wrote:
@lrosenthol <https://github.com/lrosenthol>, my problem is: how would you
ensure that a WP (as a collection of resources) would not, via for example
a reference to a wikipedia page, essentially pull 'in' the whole Web? Ie,
although not mathematically but practically be infinite? What would be of
any practical use of such a WP?
I don't understand the context where one would "pull in" anything? Are you
talking about the case where a WP is going offline? Or when it wants to
make a PWP? Or something else? It would help to understand the use
case(s) where this need is arising.
At the moment, secondary resources are those that are necessary to render
the publication, wherever those resources are. Maybe we should restrict
that definition to those resources that *are* to be used offline, or
something like that; after all, the resources that are not to be "cached"
are "just" Web resources that are not specific to the WP.
Where does it mandate that a resource "required for rendering" also have to
be cached? As we have said, there is no requirement that *every* WP can be
taken offline *or* that it can be packaged. What we want, however, is to
ensure that a publication that wants to be used offline *or* be packaged
can be. But I also know that this is also controversial, in that some
people believe that the user should control this not the
author/publisher/publication.
|
I do not really believe in the feasibility of such compilation without some sort of a hint...
You have a point insofar as not every WP is meant to be offline (although I do not have a good example from the top of my head). So I am happy to mellow the requirement:-) But then we need some sort of a clear definition what the UA has to do if there is no information. What I could imagine:
(In practice the "information" for secondary resources may either be an explicit list or some sort of a scoping mechanism through scope URL-s or URL patterns, or something more clever.) Would that direction (possibly refined) work? (this is relevant to issue #22 :-) |
|
Sorry @lrosenthol
Yes, I meant going offline and/or packaged. |
|
On Wed, Aug 9, 2017 at 10:09 AM, Ivan Herman ***@***.***> wrote:
What I could imagine:
1. The Manifest MUST include info on all primary resources
2. The Manifest MAY (or SHOULD?) include information on secondary
resources (not necessarily all). Those should be taken into account by the
UA when going offline and for any other purpose (eg, as subjects of
metadata, if applicable). That set of information is not necessarily
exhaustive v.a.v. secondary resources
3. Any secondary resource not referred to explicitly is ignored for
offline or other usages.
Would that direction (possibly refined) work?
Works great for me!! I think it addresses both positions.
Thanks!
|
This is where the browser's existing
Also, I'd be careful to not open the door to a WP that can't be taken offline. If that's the case, then it's a broken WP...or "just" a web page/site. So. Riffing of @iherman's list:
The result being that any resource not descriptively referenced someplace (the manifest or a primary resource) will not be available offline. |
|
I prefer @iherman take on this than what @BigBlueHat just proposed. |
|
On Wed, Aug 9, 2017 at 10:23 AM, BigBlueHat ***@***.***> wrote:
Also, I'd be careful to not open the door to a WP that can't be taken
offline. If that's the case, then it's a broken WP...or "just" a web
page/site.
Sorry, @BigBlueHat, but I have to strongly disagree with you.
I believe that the author/publisher of a WP has every right to determine
that they don't wish to have some/all of their publication take offline (or
packaged).
|
|
@HadrienGardeur because? @lrosenthol because? If they don't want those WP features, then they should build a Web Page, Web Site, or Web App (sans-offline-ability).
|
|
My take on this is that it's entirely up to the author to decide what's listed as primary and secondary resources, that's how they define the boundary of a publication. We shouldn't force authors to list resources used by a primary resource in a secondary resource list. Only the author can know what should or shouldn't be listed and no one else. |
|
@BigBlueHat, this example has been used a lot in the DPUB IG and we might all be sick of it:-), but let us suppose a WP online uses some sophisticated font which cannot or shouldn't be put offline (either because it is way too large or whether there are some legal restrictions). The WP uses that font of line, ie, it is, strictly speaking, a secondary resource, but it is not declared in the manifest to avoid being downloaded. |
|
This may be a silly question if I simply missed another discussion: What is the list of secondary resource for? Some people seem to expect that the list of primary and secondary resources feeds into some kind of serviceworker cache (though it sounds more like appcache manifests). Was there agreement on that? If not, what else is the list for? |
|
@pkra in the context of Readium-2 we've used the list of secondary resources to do the following things so far:
You're correct that how these resources will be cached is more similar to AppCache Manifest than a Service Worker + CacheStorage. That said, the list of primary and secondary resources is purely a hint for a UA. How the content is actually cached is entirely up to the UA to decide. |
|
Is not listing resources really a viable approach to no-cache? The user agent can determine cache-ability from the headers if it attempts to offline the publication, and should respect those. It shouldn't also have to inspect the manifest. And what stops someone with a manifest from turning your web publication into a pwp whether you want them to or not? All they need is a program that will use the manifest to find and inspect the listed resources and they'll pull it down whether you like it or not, and regardless of what is listed. A directive not to pwp-ize a publication for conforming user agents might be a start, but it's not going to stop someone who is determined to steal your publication. |
|
@mattgarrish stealing is a strong word, but you're right that there are multiple approaches to generate a PWP from a WP. The "simple" approach is to simply package resources listed in the manifest, but you could also take the approach that Chrome is taking to handle |
|
@HadrienGardeur thanks for describing Readium2. I've finally caught up on #22 completely so this makes a bit more sense (not sure how I got here first -- apologies for adding noise). In particular @baldurbjarnason's comment at #22 (comment) covers a lot of relevant issues. |
Fair enough. I'm just thinking of the intellectual property theft concern of prohibiting a pwp. Telling publishers that if they don't list a resource is not the security they'll want. But, yes, someone might, and can, still try to pwp-ize a publication the author doesn't want made into a pwp without it being theft. |
|
On Wed, Aug 9, 2017 at 11:31 AM, Matt Garrish ***@***.***> wrote:
stealing is a strong word
Fair enough. I'm just thinking of the intellectual property theft concern
of prohibiting a pwp. Telling publishers that if they don't list a resource
is not the security they'll want.
I don't think any suggested that the (main?) reason for not allowing
offline versions of a WP had anything to do with theft or security. I
simply pointed out that it needs to be something that the author/publisher
can control.
|
|
Since nobody has presented a pre-existing real world publishing industry use case for leaving resources that are clearly a part of the publication out of the manifest: One of the frequent issues I encountered back when I was still churning out ePubs as a living was the sheer, mind-boggling size that the 'include everything' requirement induced. This had serious consequences on a lot of product decisions with real-life economic impact because the size could literally prevent customers from fetching, loading, or storing the ePub. Not to mention the fact that for the files we were distributing via KDP, Amazon was still charging for downloads at an exorbitant rate (they might still be doing this, thankfully I don't care any more). So we were faced with questions like:
In the end the decision was always: no videos, reduce the images even if it makes them look bad in some contexts, don't risk rejection. Web Publications are going to have the same economic constraints when it comes to offline as device space is not increasing that fast and large parts of the world still have slow networks. If we relax the 'must include everything' restriction then we could solve 1 by only listing the resources linked to from the default src attribute in images and leave the higher resolution versions external and linked to via srcset. It would solve 2 by making it clear from the start that UAs need to support this. And, hopefully, being on the open web platform will go a long way to solve 3. 😊 |
|
@iherman for the font license scenario you mentioned, I'd put the onus of enforcing usage of the font on the font's licensor not on the author/licensee. Meaning, Content Security Policy (CSP), CORS, etc, would be the technological system(s) to prevent misuse of the font...not it's reference by the author/creator of a WP. @HadrienGardeur it's true that the CacheStorage system is scripted, but in the case of Readium-2 it's being populated from a descriptive document. Everything mentioned so far is focused on descriptive definition of the "binding" for a publication. The question (back to the issue title) is more about where the dependencies (secondary resources) are declared. @pkra ...
They're dependencies of primary resources...some of which MAY be content related (i.e. quiz answer keys, pop-up something or others, etc).
To make sure it's part of the publication as considered by the reading system. The question is less "would an author list something" as it is "where would an author list something" as a secondary resource.
The frequent mention of ServiceWorker's is mostly about shimming/poly-filling these ideas today. In the future (one would hope) such a shim/poly-fill would go away. We should certainly look to where AppCache went wrong--which seems mostly centered around whether it or the referencing documents were "in charge"...and all the related issues caused when that was unclear. If there's a singular document that defines the "binding" of the publication, then my hope is that we'd avoid the split-brain situation of AppCache. Hopefully. 😃 |
|
@lrosenthol still curious as to why "readable offline" would be the domain of the author/publisher to decide and not the reader. |
And there was the the sheer, mind-boggling labour of documenting & listing 'everything', and coming up with an ID for everything, where the ID wasn't used for anything, and I could have produced the list with If I make a website, I don't have to list all the resources anywhere. If I write a book, I don't have to make a separate list of all the words I used. I dream that WP authors won't have to make such a list, hence my interest in other ways of obtaining such admittedly useful information. |
I can't speak for others but when I mention ServiceWorkers it's because many browser vendors prefer functionality of new features be defined in terms of how lower level features work. E.g. features involving network requests are now generally being specified in terms of the fetch API even if the feature would be implemented entirely in C++. Defining a higher level feature in terms of how a lower level feature works adds clarity and aids implementation even if that implementation does not use the lower level feature. E.g. discussing the specifics of how a publication be made offline using the manifest and a service worker clarifies things like how you'd handle security updates and cache strategies, even if you don't use a service worker to implement it. And we need a clear answer on the security update question. That can't be up to the implementation. If one popular implementation gets it wrong, the results could be wide-ranging and devastating. |
AppCache was a security nightmare. Its updating mechanism resulted in stale and insecure versions of code often being persisted forever. One-time exploits could be made permanent and irreversible by a hacker by updating the appcache to cache itself (no more updates, current versions of assets are what you're stuck with). And given that appcache worked on non-secure origins, some websites could be hacked just by virtue of accessing them via Café wifi. There's very little about AppCache that wasn't awful in some way. It also highlights why implicit offline storage of resources via scope is a potential security nightmare. It resulted in a lot of resources being unintentionally stored and then never updated. |
|
On Wed, Aug 9, 2017 at 12:11 PM, BigBlueHat ***@***.***> wrote:
@lrosenthol <https://github.com/lrosenthol> still curious as to why
"readable offline" would be the domain of the author/publisher to decide
and not the reader.
I think it's the domain of both.
If an author wishes to have complete control over the "offlining" of their
content - they should be able to. If an author doesn't care and is happy
to rely entirely on the UA, they should have that option.
I am just making sure that our designs cover both cases.
|
|
On Wed, Aug 9, 2017 at 11:53 AM, Baldur Bjarnason ***@***.***> wrote:
Since nobody has presented a pre-existing real world publishing industry
use case for leaving resources that are clearly a part of the publication
out of the manifest:
I thought that I have presented two of them in previous messages - but let
me restate them. (and remember that I am viewing this from the lens of
both _ad-hoc_ as well as professional publishing
1) Authors want to be able to control which items can be cached and/or when
they can be cached. On the web, this is done with a Cache-Control header,
but most content lives on servers where users don't necessary have access
to that value. One (possible) way to provide some hinting to that would be
not having it in the manifest (eg. not present == no cache).
2) Combining publications together (such as the quite common Combine/Merge
PDFs) is a huge nightmare if you have to reconcile naming conflicts on
secondary resources.
|
|
And there was the the sheer, mind-boggling labour of documenting & listing 'everything', and coming up with an ID for everything, where the ID wasn't used for anything, and I could have produced the list with unzip -1.
If I make a website, I don't have to list all the resources anywhere. If I write a book, I don't have to make a separate list of all the words I used. I dream that WP authors won't have to make such a list, hence my interest in other ways of obtaining such admittedly useful information.
I agree. Hence my proposal that *when we get down to the way the abstract manifest is expressed* we allow for, say, URL patterns. Ie, I can simply say, via the manifest, that "all resources in this directory are secondary resources to be listed in the manifest".
|
Let us not bog down in the specific example. The reason why an author would want to exclude a font may also be because the font is huge, and unnecessary complex for offlining on a slow device. Similar considerations may come from huge images, or complex javascripts that should not be used offline (or cannot be used because the author knows that there are extensive external references in the script). What my proposal in #23 (comment) ensures that the author/publisher has control, if needed. |
Correct. And we should be very clear in our prose when writing the spec that any restriction on the secondary resource listings should not be viewed as an efficient security measure. It is certainly not. |
I'm not sure how this could work on the Web, there's no way you can list all resources in a "directory". Are you talking specifically about PWP? If we want UAs to use secondary resources for something, we should IMO work with URLs not templates/patterns. |
I am sorry, I did not use the right terms. What I meant was to use URL templates[1], which is an existing IETF Proposed Standard (which already has Javascript implementations[2]). We will have to examine whether it really works for us but, as far as I could see, it is a regexp-like formalism tailored to the URL syntax. |
|
We use URI templates in Readium-2 to discover and interact with various APIs (search, locator resolver and media overlay). But URI templates are not suited for listing resources. To do that you'd need to have some sort of convention in how the URIs are built (I don't think that's reasonable). When a UA decides to download a resource (for prefetching, caching or packaging) it needs to know exactly what the URL will be. |
|
As I've been out, I'm gonna briefly riff back to later in this thread -- specifically Ivan's proposal which has seemed to gain some consensus. I too think that's good progress, and can certainly accept it for now. I think we can defer to post-FPWD the argument on whether the #2 is a MAY, SHOULD, or MUST. Though, we need to keep in mind that a WP has a requirement (in our charter) of "A Web Publication must be available and functional while the user is offline", and PWP is described (in our charter) as: "This specification defines a way to combine the resources of a Web Publication into a distributable file using a packaging format." To move away from MUST we'll need to consider changing our definitions (which could be done via consensus) from "all WPs must be offline-able and must be transformable to PWPs" to "WPs may/should be constructed such that they can be made available offline and be transformable to PWPs". Again, for where we are now, I think we can defer a decision here. |
Hm. Why? My model was that the user/publisher would use a URL template as a regex-like specification for which secondary resources are "listed" in the manifest (eg, for offlining). The UA can then take any specific URL it meets and can check whether it resolves to (one of the) pattern(s). But we are getting down to the weeds. It is too early. |
So you're expecting the UA to prerender the primary resources or parse them to:
I don't find this easier to author (at least not much) and it clearly makes the job much more difficult for the UA. |
|
On 10 Aug 2017, at 18:25, Hadrien Gardeur ***@***.*** ***@***.***>> wrote:
@iherman <https://github.com/iherman>
The UA can then take any specific URL it meets and can check whether it resolves to (one of the) pattern(s).
So you're expecting the UA to prerender the primary resources or parse them to:
find URLs
then check such URLs against a number of URI templates that would pretty much act as a white list?
I don't find this easier to author (at least not much) and it clearly makes the job much more difficult for the UA.
True for the UA. But if the author has the choice between using a template or list 1000 images then I believe it is a no brainer.
As for the extra work on the UA: there is a hierarchy, and we should optimize first on the user and only then to the implementers...
|
|
I wouldn't call a regular-expression like syntax a "no brainer" for anyone (authors or implementers). With such a syntax, there's also no way to provide important information about a resource (dimension of an image, duration of an audio track). Saying "it's up to the implementers to figure it out" is IMO not a good strategy, you have to consider everyone when designing such documents. Putting my Readium hat on, I can tell you that if we have to support prerendering to handle caching resources, this will take years to figure it out (the Chrome team has been working on this problem for >5 years now). |
|
@HadrienGardeur, I would propose to postpone this discussion for a later time. This issue is whether we should/may/must have a list of secondary resources in some way or other in the manifest (abstract and concrete). What syntax we would use, whether we would use other tools like URL templates, etc, is a detail. Let us concentrate on the original issue for now. I shouldn't have brought up these details for now. My mistake. |
|
Sorry for jumping in .. The manifest will help determine what is inside or outside the WP - which of course is also useful for attribution purposes. Without the manifest I am not sure what is the point as you could just have I think preservation should also be taken into consideration - not just being able to go offline as a client. A web resource you depend can also disappear - for example when using interactive figures. A secondary entry in the manifest means it is made explicit (that is - detectable by a simple Python script) that somewhere in your WP you have used sloppy and fragile Cloud hosting like An UA can warn "This page requires 'external content'" if a resource outside the manifest is attempted loaded and probably should be allowed to disregard it. |
|
Propose closing: the infoset in the draft has now a number of entries, and this issue itself became extremely long an a bit lost focus. We may be better off closing it and, if necessary, open new, more focused issues when the time comes. |
This has been discussed in different other issues, and it is probably better to separate this as an explicit question to be followed.
The text was updated successfully, but these errors were encountered: