URL Scope to which the manifest applies #114
See also, a thread on Mozilla's dev-b2g mailing list https://groups.google.com/d/msg/mozilla.dev.b2g/r6zAZfEBAI8/qvSmGpAPAJMJ
This is also important for navigational purposes (see #142).
The spec defines properties like "display" which define characteristics of the window in which the bookmarked app is opened, but under what circumstances should those options be applied?
Once a browsing context has been created it can navigate to any URL. Does the "standalone" or "fullscreen" display mode still apply if I navigate outside the app to an unrelated web page from another origin? What defines "outside the app" exactly?
My conclusions so far from that thread are that by default the scope of an app is the origin of the manifest. But there are cases (like multiple apps per origin) where you want to be able to carve the origin up into multiple apps, or into app content and general web content.
In the thread I suggest a manifest property called "scope" for this purpose, which contains a list of strings (URLs with wildcards or regex, but relative to the origin of the manifest URL like the start_url) which define the scope of the app. Two apps with overlapping scope can not be installed.
Whenever a URL is navigated to it is matched against the list of "bookmarked" origins. If there is a match then it is matched against the scope property of the corresponding manifest(s) if present. If there is still a match then the URL is opened in the app defined by that manifest, with the properties defined in the manifest. If there is no match then the URL is opened inside a normal browser window.
This is similar to the "urls" property in the manifest of Chrome Hosted Apps, except that it is a path relative to the origin rather than an absolute URL which can be from any origin. This is because allowing multiple origins may be open to abuse where an app registers to handle URLs from another origin. To prevent that I believe Chrome Hosted Apps require developers to prove that they own a given domain name, which is a centralised tricky process that requires the manifest be hosted by the app store. By restricting the scope at maximum to the origin of the manifest we can already deduce that the author of the manifest has a certain level of ownership over or access to that origin.
If no scope is defined, should the scope be the full origin of the manifest, or restricted to just the start_url?
I had not considered how this might interact with scope of ServiceWorkers so I'm interested to find out more about how that works.
Ok, so my plan is to pilfer "URL scope" from Service Workers and use it to set the boundary of the web application.
The boundary is the extent to which members declared in the manifest apply to the top-level browsing context* (display, orientation, navigation, etc.). The
The default behavior will be to open anything outside the
The URL scope does not affect where resources are loaded from within HTML. That is controlled via CSP.
I like this approach, particularly how it mirrors the feature in Service Workers.
I have some questions...
"The default behavior will be to open anything outside the scope in the browser. "
When I said "open in the browser" I mean leave it up to the UA. Definitely, if you have an app installed that own some URL space, then it should just open in that app with the manifest applied.
UA specific - the UA may progressively apply the manifest at that point or ask the user.
No - it does not influence
We can't stop the overlap. It's the same problem service workers have. We should behave the same - and that's still not fully clear how it will work.
Probably. I'm on the fence ATM. It sucks that the only choice someone has is to either "/*" or to put everything inside a single directory "/myapp/". OTOH, it does make everything much easier to deal with - but might be too limiting. I need to play with a few apps to understand more how they are laid-out.
The scope is bound to the origin of the document doing the installation (which is not necessarily the origin of the manifest). "path expressions" force this by default.
CSP 1.2 will include a
Ok, discussed with @mounirlamouri.
The default manifest scope is that URL of form which the page is bookmarked.
If the page is bookmarked from:
Then the following are in scope:
Use case: allows bookmarking http://www.w3.org/Mobile/IG/ without needing to add a scope attribute. Allows navigation of
If the manifest is present and has a
So, bookmarking: http://www.bbc.com/news/world-us-canada-29842472 is in scope.
The following is not in scope:
If the Document is http://www.bbc.com/weather/3413829, but the scope is "/news/" and there is no start_url, the scope is "http://www.bbc.com/weather/3413829". That is, the start_url must be in scope, or else the scope gets dropped on the floor and replaced Document's URL.
If for some reason a weather forecast did point to a manifest for a BBC News app with a scope of "/news" as in your example, why would you want to install an app with the name "BBC News" and a BBC news icon, with a scope of http://www.bbc.com/weather/3413829?
Why be inconsistent with the way scope works for Service Workers? Surely a better default scope would be "/" as it is with Service Workers? At least that way the start url will always be in scope by default.
The solution described above seems to be optimising for single page apps being the most common, but your research indicates that isn't the case.
Here are some other examples:
If you're going to use a path other than "/" as the default scope then a better choice would seem to be the provided "start_url". However, for the very common case of http://foo.com/index.html as a start url, http://foo.com/bar would be out of scope, which seems wrong.
This leads me to the conclusion that "/" is the most sensible default so that by default the whole origin is in scope, as with Service Workers. If the developer wishes to carve a single origin up into multiple apps (e.g. http://google.com/calendar and http://google.com/maps), then the scope property allows them to do that.
If you want to bookmark http://www.bbc.com/news/world-us-canada-29842472 then just bookmark it. Why would I want a standalone app which can only view one news story? The user agent could optionally choose to open that bookmark inside the BBC News app if the BBC News app is also installed with a scope of http://www.bbc.co.uk/news.
On October 31, 2014 at 3:14:43 AM, Ben Francis (email@example.com) wrote:
Start URL becomes the scope.
Rationale for author/developer decisions is not really in scope for the discussion. I.e., we can't really correct for developer error - we can only work within the authors intent.
Service workers are outside the scope of this discussion. This is to do with navigation scopes, and not related to service workers.
Actually, that's a matter of perspective. We start with the default case where nothing is defined and work our way from there. The most logical thing to do in that situation is to work with the information the UA has.
Correct. That's Facebook's choice. We don't make decisions for Facebook.
Correct. That's Twitter's choice. We don't make decisions for Twitter.
Correct. That's Yelp's choice. We don't make decisions for Yelp.
In all cases above. the service in question could just add:
The counter example is:
Where defaulting to "/" makes no sense - defaulting to "/Mobile/IG/" does.
There is no guarantee of a start_url. But yes, then start_url can override the scope iff it's outside the "scope".
That would again be developer error. It's easy to fix with just adding a scope property.
And so on...
You are confusing the manifest with installation. It's just metadata. What the UA does with it is something different. The manifest can be used in bookmarking just as much as it can be used in installation.
Linking/link handling is currently outside the scope of this discussion - this is to do with navigation. We will deal with that in a separate bug.
@annevk that's kinda exactly the point, we are not initially providing anything and not making any security guarantees. We are deferring to the user agent to provide the UI to inform the user when a boundary is crossed (as Chrome does today, by showing the URL bar if the user leaves the origin from which they started navigating from). Again, like this:
It is, of course, totally valid to argue that showing the URL bar might be completely inadequate and insufficient. However, it gives a starting point on which to build a security model from, and can serve as a "stick" that breaks the "app" experience (e.g., throwing the application out of fullscreen mode on crossing and origin boundary). "Good" developers will want to avoid that, for instance, by not crossing domain boundaries.
By default, the spec assumes the page is opened in the "browser" (using system convention for URL handling). However, the spec allows web apps to be opened in one of four display modes:
This is controlled by the
I'm part way through a study  of URL scope for web properties from the Alexa Top 500, Chrome Web Store and Firefox Marketplace. So far I've looked at the top 20 from Alexa, 10 from Chrome Web Store and 30 from the Firefox Marketplace.
I haven't reached any final conclusions about how scope should work, but I thought it might be helpful to share my findings so far.
Some conclusions so far:
Regarding the previous proposal of using the start_url or Document URL as a default origin:
Regarding the proposal to set an unbounded scope for apps by default:
Regarding the proposal to use the root of the web app's origin as the default scope:
Great summary, @benfrancis. Just some minor things: the "open pages in new tabs" pattern for Asian sites seems to be a desktop only thing (which I guess makes sense). For their mobile/small-device equivalent, when they have one, sites work similarly to what we experience in US/UE sites.
Generally, I think I'm in agreement with your initial conclusions. At the very least, it seems that "unbounded" by default (with UX security behavior like Chrome's) poses the least number of hurdles to adoption of this technology.
Question is: should we prioritize adoption over security? As long as installation is left in control of the browser, then I think it's for us to do so as we continue to work out the heuristic and UX for web app installation. We can then also work out if we need white/black lists to define actual navigation scope.
I don't think that the point should be to turn any site into an app with as little effort as possible. Quite the opposite.
If people want to create an app like experience we should make it possible for them to do so, ie. give them the right tools.
Default should be what gives the best security and a real app experience. People should invest in creating nice app experiences and not do the bare minimum. When some show what is possible, others will follow. If everyone just does the bare minimum, people will keep saying that the web isnt compelling in comparison with native offerings.
If some links etc ends up opening the browser (and closing it again for logging in) that is not the worst thing in the world, but we should make it possible for the web app author to mitigate these issues, even if that has to be done in the manifest or on their servers.
It seems there are a couple of directions this could be taken in. One is to try to accommodate all existing web content with a complex, flexible solution. The other is to go for a simpler solution which is not backwards compatible with some of the web.
A complex solution might be an unbounded scope by default and a manifest which looks like:
The scope has both a whitelist and a blacklist and there's an additional list of URLs which are not part of the app but should stay in the context of the app (e.g. third party authentication). This is very flexible but I think the danger is that the matching rules start to get quite complex for the user agent to carry out on every top level navigation and we risk creating a complex declarative list like AppCache which is unwieldy to maintain.
At the other extreme, the default scope is the root of the domain and the manifest looks like:
Scope is a single URL prefix. This would be simple for new web apps to use, but may provide a bad experience for a lot of existing content where web sites and web apps share URL spaces in complex ways. For example https://evernote.com/products/ would be forced to open inside the Evernote app, and users of GMail might be kicked out of the app into the browser when they want to log in.
If we were starting with a clean slate then my preference would be that the default scope is the root of the origin and the scope of the app is defined by a single URL prefix (or perhaps a whitelist array).
Unfortunately if we turned this feature on overnight in Firefox OS and Chrome OS for existing apps it would create a bad user experience for many existing apps which were not built to this pattern. In fact in the case of Firefox OS which has separate data jars for each app it might actually break apps which navigate to a separate origin in the top level browsing context for authentication purposes. It would also create friction for a lot of big name web properties to adopt Web Manifest because it would either require them to add in workarounds in their apps or to make big infrastructure changes in order to use a manifest.
So, should Web Manifest be backwards compatible with all web content, should it be a clean slate for new web content built with a clean URL scope, or should it take a middle ground?
An example of a middle ground might be:
which incidentally is basically what Jonas proposed in https://groups.google.com/forum/#!topic/mozilla.dev.webapi/yqLffXgokuI
I think we could also make the root of the origin the default scope as in Jonas' proposal above, but to do this for existing apps we would need to give developers advance warning.
I've added examples of this "middle ground" manifest format to the document at https://docs.google.com/a/tola.me.uk/document/d/1fOsQWOOVuKyqO7cXZoKmxZGQ9FLgLMwmCRw3OEqIKrQ/edit#heading=h.854z5nn814fy
It's kind of more complicated. Why wouldn't a scope be needed for external domains? Maybe
which would be the same as
We should probably enforce some CSP rule if we allow other domains , or ignore them otherwise.
I still think it would be good to show some sign when moving to another domain (maybe ignoring subdomains). It would still more or less feel like one app and still it would discourage the use of multiple domains somewhat.
One potential use case for scope is that an installed app could capture navigations to URLs within the scope of the app and load them in the app instead of in a browser tab. This would allow deep linking inside a web app.
You don't want evilapp.com/manifest.json to claim that facebook.com is part of its scope and then capture all of the user's navigations to Facebook. How do you safely prove that another origin is part of the same app? As I understand it the Chrome Web Store uses a centralised process using CNAME records to do this, but that doesn't scale very well to the rest of the web. In my opinion, the maximum scope of a web app should be a single origin.
However, you still might want goodapp.com to use facebook.com/login for authentication. The idea of the "stay_in_app" property is that it can be used to enumerate third party URLs which are not part of the app itself but are used by the app for something like third party authentication. The app will not capture navigations to those URLs, but if the user is already in the context of the app and then navigate to one those URLs they will stay in the app rather than being kicked out to the browser.
Jonas explains this more thoroughly in the thread linked to above.
I agree that if an unbounded scope or a stay_in_app URL causes an app window to be navigated away from its original origin, that the user should be made aware of the new origin they just entered. I don't think subdomains can be assumed to be part of the same app though because many shared hosting services used subdomains to separate sites.
On November 17, 2014 at 10:09:22 AM, Ben Francis (firstname.lastname@example.org) wrote:
True. But we've not experimented with this enough to know if scopes are the right thing here.
Exactly. Hence the same origin restriction.
Why is window.open() not sufficient? What am I missing?
Agree. For example, GitHub pages.
A lot of existing web content navigates the top level browsing context to another origin for authentication. There are web apps which span multiple origins for other reasons too (e.g. both http:// and https:// URLs or different origins for different parts of the app). The stay-in-app property is a proposal of how to be backwards compatible with this behaviour in-app, whilst not allowing malicious apps to capture outside navigations from other origins (I think capturing/deep linking is a key use case we should be designing for).
The Manifest spec could enforce that all top level navigations for an app should come from a single origin, and therefore intentionally not be backwards compatible with some content, but that should be a conscious decision.
I agree that window.open() could offer an alternative to the stay-in-app property if app developers author their apps that way. But I actually think the default behaviour for window.open() or a target=_blank hyperlink to a URL outside of the app's defined scope should be to open outside the context of the app (e.g. in the browser). The stay-in-app property would be one way to override this default, another way would be to pass a special argument to window.open().
To throw something else into the mix...
The scope of Service Workers is "restricted by the path of the Service Worker script unless the Service-Worker-Scope: ... header is set." 
We could follow the example of Service Workers and make the maximum scope of a manifest be the path of the manifest file (i.e. the same directory). This would prevent an app on a shared hosting site (e.g. http://example.com/~jack/manifest.json) from registering to manage URLs of another app (e.g. http://example.com/~jill/index.html). We could even go as far as to make this the default scope, though that would again not be backwards compatible with some content. I believe this is basically how Amazon web app manifests handle scope  and it's a pretty neat solution.
@benfrancis, I'm not a fan of binding scope to the location of a resource (because resources can, and should be able to, move and redirect independently of the navigation scope).
I think keeping Chrome's current behavior + and scoping to a single URL will work ok for now. I'm preparing a PR based on that.