Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
URL Scope to which the manifest applies #114
See also, a thread on Mozilla's dev-b2g mailing list https://groups.google.com/d/msg/mozilla.dev.b2g/r6zAZfEBAI8/qvSmGpAPAJMJ
This is also important for navigational purposes (see #142).
The spec defines properties like "display" which define characteristics of the window in which the bookmarked app is opened, but under what circumstances should those options be applied?
Once a browsing context has been created it can navigate to any URL. Does the "standalone" or "fullscreen" display mode still apply if I navigate outside the app to an unrelated web page from another origin? What defines "outside the app" exactly?
My conclusions so far from that thread are that by default the scope of an app is the origin of the manifest. But there are cases (like multiple apps per origin) where you want to be able to carve the origin up into multiple apps, or into app content and general web content.
In the thread I suggest a manifest property called "scope" for this purpose, which contains a list of strings (URLs with wildcards or regex, but relative to the origin of the manifest URL like the start_url) which define the scope of the app. Two apps with overlapping scope can not be installed.
Whenever a URL is navigated to it is matched against the list of "bookmarked" origins. If there is a match then it is matched against the scope property of the corresponding manifest(s) if present. If there is still a match then the URL is opened in the app defined by that manifest, with the properties defined in the manifest. If there is no match then the URL is opened inside a normal browser window.
This is similar to the "urls" property in the manifest of Chrome Hosted Apps, except that it is a path relative to the origin rather than an absolute URL which can be from any origin. This is because allowing multiple origins may be open to abuse where an app registers to handle URLs from another origin. To prevent that I believe Chrome Hosted Apps require developers to prove that they own a given domain name, which is a centralised tricky process that requires the manifest be hosted by the app store. By restricting the scope at maximum to the origin of the manifest we can already deduce that the author of the manifest has a certain level of ownership over or access to that origin.
If no scope is defined, should the scope be the full origin of the manifest, or restricted to just the start_url?
I had not considered how this might interact with scope of ServiceWorkers so I'm interested to find out more about how that works.
This was referenced
May 6, 2014
referenced this issue
May 30, 2014
Ok, so my plan is to pilfer "URL scope" from Service Workers and use it to set the boundary of the web application.
The boundary is the extent to which members declared in the manifest apply to the top-level browsing context* (display, orientation, navigation, etc.). The
The default behavior will be to open anything outside the
The URL scope does not affect where resources are loaded from within HTML. That is controlled via CSP.
I like this approach, particularly how it mirrors the feature in Service Workers.
I have some questions...
"The default behavior will be to open anything outside the scope in the browser. "
When I said "open in the browser" I mean leave it up to the UA. Definitely, if you have an app installed that own some URL space, then it should just open in that app with the manifest applied.
UA specific - the UA may progressively apply the manifest at that point or ask the user.
No - it does not influence
We can't stop the overlap. It's the same problem service workers have. We should behave the same - and that's still not fully clear how it will work.
Probably. I'm on the fence ATM. It sucks that the only choice someone has is to either "/*" or to put everything inside a single directory "/myapp/". OTOH, it does make everything much easier to deal with - but might be too limiting. I need to play with a few apps to understand more how they are laid-out.
The scope is bound to the origin of the document doing the installation (which is not necessarily the origin of the manifest). "path expressions" force this by default.
CSP 1.2 will include a
Ok, discussed with @mounirlamouri.
The default manifest scope is that URL of form which the page is bookmarked.
If the page is bookmarked from:
Then the following are in scope:
Use case: allows bookmarking http://www.w3.org/Mobile/IG/ without needing to add a scope attribute. Allows navigation of
If the manifest is present and has a
So, bookmarking: http://www.bbc.com/news/world-us-canada-29842472 is in scope.
The following is not in scope:
If the Document is http://www.bbc.com/weather/3413829, but the scope is "/news/" and there is no start_url, the scope is "http://www.bbc.com/weather/3413829". That is, the start_url must be in scope, or else the scope gets dropped on the floor and replaced Document's URL.
If for some reason a weather forecast did point to a manifest for a BBC News app with a scope of "/news" as in your example, why would you want to install an app with the name "BBC News" and a BBC news icon, with a scope of http://www.bbc.com/weather/3413829?
Why be inconsistent with the way scope works for Service Workers? Surely a better default scope would be "/" as it is with Service Workers? At least that way the start url will always be in scope by default.
The solution described above seems to be optimising for single page apps being the most common, but your research indicates that isn't the case.
Here are some other examples:
If you're going to use a path other than "/" as the default scope then a better choice would seem to be the provided "start_url". However, for the very common case of http://foo.com/index.html as a start url, http://foo.com/bar would be out of scope, which seems wrong.
This leads me to the conclusion that "/" is the most sensible default so that by default the whole origin is in scope, as with Service Workers. If the developer wishes to carve a single origin up into multiple apps (e.g. http://google.com/calendar and http://google.com/maps), then the scope property allows them to do that.
If you want to bookmark http://www.bbc.com/news/world-us-canada-29842472 then just bookmark it. Why would I want a standalone app which can only view one news story? The user agent could optionally choose to open that bookmark inside the BBC News app if the BBC News app is also installed with a scope of http://www.bbc.co.uk/news.
On October 31, 2014 at 3:14:43 AM, Ben Francis (email@example.com) wrote:
Start URL becomes the scope.
Rationale for author/developer decisions is not really in scope for the discussion. I.e., we can't really correct for developer error - we can only work within the authors intent.
Service workers are outside the scope of this discussion. This is to do with navigation scopes, and not related to service workers.
Actually, that's a matter of perspective. We start with the default case where nothing is defined and work our way from there. The most logical thing to do in that situation is to work with the information the UA has.
Correct. That's Facebook's choice. We don't make decisions for Facebook.
Correct. That's Twitter's choice. We don't make decisions for Twitter.
Correct. That's Yelp's choice. We don't make decisions for Yelp.
In all cases above. the service in question could just add:
The counter example is:
Where defaulting to "/" makes no sense - defaulting to "/Mobile/IG/" does.
There is no guarantee of a start_url. But yes, then start_url can override the scope iff it's outside the "scope".
That would again be developer error. It's easy to fix with just adding a scope property.
And so on...
You are confusing the manifest with installation. It's just metadata. What the UA does with it is something different. The manifest can be used in bookmarking just as much as it can be used in installation.
Linking/link handling is currently outside the scope of this discussion - this is to do with navigation. We will deal with that in a separate bug.
So, I spoke with @annevk and he pointed out a bit of an issue with the path based scope model: essentially,
We have a complicated issue where some want sites to use one origin per application, but the situation is that many don't or can't - and motivating sites to change their infrastructure/setup to accommodate manifests will be extremely challenging.
@mounirlamouri can you check what the Chrome security folks think?
Is this something which the Web Manifest specification needs to solve now, or a wider issue around the definition of "origin" on the web?
The current properties of Web Manifest are more related to presentation than to a permissions model. Perhaps the "scope" property of Web Manifest should focus on navigation scope rather than permissions scope, until such a point in the future when Web Manifests are used to define permissions.
Chrome Hosted Apps have a path based permission scope which is something I think Mozilla Open Web Apps lack, but this does not effect the fact that multiple apps on the same origin have access to the same IndexedDB databases. Mozilla Open Web Apps have separate "data jars" per installed app and key some permissions on origin + manifest URL, but have a restriction of one app per origin and don't currently enforce a navigation scope for app windows so a permission granted to an app is effectively granted to its whole origin.
Permissions scope is a thorny issue. Defining a navigation scope to which a certain display mode and default orientation applies seems a lot less contentious.
Started looking at site structures here in Alexa's top 500:
I'm hopeful we can do the top 500 over time, but it will require a coordinated effort with multiple people. Ben and I will try to do at least the top 100... my eyes are already bleeding from the yahoo homepage: ask me about which celebrity is trending this week! ASK ME! GO ON!!! I DARE YOU
A thing that we haven't added to the discussion is that, because the default display mode is "browser", if the developer doesn't actually specify a display mode, the scope doesn't matter because the application is just opened in the browser.
Hence, a "scope" (and its default) only comes into play with the display mode is explicitly not "browser".
So, looking a various sites, I find that one cannot reconcile the same origin policy with navigation (or site structures in general). That is, if we try to enforce a structure or scope by default, we are breaking the Web.
tl;dr: it seems, to me, that Chrome's unbounded (or infinite?) default scope is the most sensible given how sites are made. That is to say, if no scope is defined, then a user can navigate freely from origin to origin within from within a site (same as happens today in a web browser) - this irrespective of view mode with sensible behavior being that the url bar is displayed if the user leaves an origin:
Consider, Web Apps can:
Expand multiple sub origins
And across multiple independent domains!! The following are all the same "app"!!!
Can span multiple independent origins (the above is taobao.com, tb.com, and tmall.com!).
Can work only across one set of paths, but not across others.
For example, Evenote's applications scope would be either
I think your definition of the word "app" here is a bit too broad. I would consider this a set of mobile web sites, instead of a single discernible app/program, which you add to your device/homescreen and use as such.
@annevk, this still needs to work without service workers, tho. Looking at Alexa's top 100, it seems like a non-starter to try to impose any form of structure or the manifest will add no value to a significant percentage of existing content. I'm all for locking it down for service workers, but feel we should provide a flexible path of migration.
@skddc For instance, in toaboa.com (tmall), the navigation, breadcrums/menus, and aesthetics are made to feel like a consistent whole. The fact that a user is bouncing from one domain to another doesn't seem to matter with regards to what constitutes the boundaries of the "app". This is further evidenced by the fact that you can "install" (add to home screen) the app in iOS, and you have a consistent and cohesive application experience without ever returning to the browser (even though the app is split across multiple domains). See below:
Clearly, this is the same "app" running across multiple domains.
@annevk that's kinda exactly the point, we are not initially providing anything and not making any security guarantees. We are deferring to the user agent to provide the UI to inform the user when a boundary is crossed (as Chrome does today, by showing the URL bar if the user leaves the origin from which they started navigating from). Again, like this:
It is, of course, totally valid to argue that showing the URL bar might be completely inadequate and insufficient. However, it gives a starting point on which to build a security model from, and can serve as a "stick" that breaks the "app" experience (e.g., throwing the application out of fullscreen mode on crossing and origin boundary). "Good" developers will want to avoid that, for instance, by not crossing domain boundaries.
By default, the spec assumes the page is opened in the "browser" (using system convention for URL handling). However, the spec allows web apps to be opened in one of four display modes:
This is controlled by the
I'm part way through a study  of URL scope for web properties from the Alexa Top 500, Chrome Web Store and Firefox Marketplace. So far I've looked at the top 20 from Alexa, 10 from Chrome Web Store and 30 from the Firefox Marketplace.
I haven't reached any final conclusions about how scope should work, but I thought it might be helpful to share my findings so far.
Some conclusions so far:
Regarding the previous proposal of using the start_url or Document URL as a default origin:
Regarding the proposal to set an unbounded scope for apps by default:
Regarding the proposal to use the root of the web app's origin as the default scope:
Great summary, @benfrancis. Just some minor things: the "open pages in new tabs" pattern for Asian sites seems to be a desktop only thing (which I guess makes sense). For their mobile/small-device equivalent, when they have one, sites work similarly to what we experience in US/UE sites.
Generally, I think I'm in agreement with your initial conclusions. At the very least, it seems that "unbounded" by default (with UX security behavior like Chrome's) poses the least number of hurdles to adoption of this technology.
Question is: should we prioritize adoption over security? As long as installation is left in control of the browser, then I think it's for us to do so as we continue to work out the heuristic and UX for web app installation. We can then also work out if we need white/black lists to define actual navigation scope.
I don't think that the point should be to turn any site into an app with as little effort as possible. Quite the opposite.
If people want to create an app like experience we should make it possible for them to do so, ie. give them the right tools.
Default should be what gives the best security and a real app experience. People should invest in creating nice app experiences and not do the bare minimum. When some show what is possible, others will follow. If everyone just does the bare minimum, people will keep saying that the web isnt compelling in comparison with native offerings.
If some links etc ends up opening the browser (and closing it again for logging in) that is not the worst thing in the world, but we should make it possible for the web app author to mitigate these issues, even if that has to be done in the manifest or on their servers.
It seems there are a couple of directions this could be taken in. One is to try to accommodate all existing web content with a complex, flexible solution. The other is to go for a simpler solution which is not backwards compatible with some of the web.
A complex solution might be an unbounded scope by default and a manifest which looks like:
The scope has both a whitelist and a blacklist and there's an additional list of URLs which are not part of the app but should stay in the context of the app (e.g. third party authentication). This is very flexible but I think the danger is that the matching rules start to get quite complex for the user agent to carry out on every top level navigation and we risk creating a complex declarative list like AppCache which is unwieldy to maintain.
At the other extreme, the default scope is the root of the domain and the manifest looks like:
Scope is a single URL prefix. This would be simple for new web apps to use, but may provide a bad experience for a lot of existing content where web sites and web apps share URL spaces in complex ways. For example https://evernote.com/products/ would be forced to open inside the Evernote app, and users of GMail might be kicked out of the app into the browser when they want to log in.
If we were starting with a clean slate then my preference would be that the default scope is the root of the origin and the scope of the app is defined by a single URL prefix (or perhaps a whitelist array).
Unfortunately if we turned this feature on overnight in Firefox OS and Chrome OS for existing apps it would create a bad user experience for many existing apps which were not built to this pattern. In fact in the case of Firefox OS which has separate data jars for each app it might actually break apps which navigate to a separate origin in the top level browsing context for authentication purposes. It would also create friction for a lot of big name web properties to adopt Web Manifest because it would either require them to add in workarounds in their apps or to make big infrastructure changes in order to use a manifest.
So, should Web Manifest be backwards compatible with all web content, should it be a clean slate for new web content built with a clean URL scope, or should it take a middle ground?
An example of a middle ground might be:
which incidentally is basically what Jonas proposed in https://groups.google.com/forum/#!topic/mozilla.dev.webapi/yqLffXgokuI
I think we could also make the root of the origin the default scope as in Jonas' proposal above, but to do this for existing apps we would need to give developers advance warning.
I've added examples of this "middle ground" manifest format to the document at https://docs.google.com/a/tola.me.uk/document/d/1fOsQWOOVuKyqO7cXZoKmxZGQ9FLgLMwmCRw3OEqIKrQ/edit#heading=h.854z5nn814fy
It's kind of more complicated. Why wouldn't a scope be needed for external domains? Maybe
which would be the same as
We should probably enforce some CSP rule if we allow other domains , or ignore them otherwise.
I still think it would be good to show some sign when moving to another domain (maybe ignoring subdomains). It would still more or less feel like one app and still it would discourage the use of multiple domains somewhat.
One potential use case for scope is that an installed app could capture navigations to URLs within the scope of the app and load them in the app instead of in a browser tab. This would allow deep linking inside a web app.
You don't want evilapp.com/manifest.json to claim that facebook.com is part of its scope and then capture all of the user's navigations to Facebook. How do you safely prove that another origin is part of the same app? As I understand it the Chrome Web Store uses a centralised process using CNAME records to do this, but that doesn't scale very well to the rest of the web. In my opinion, the maximum scope of a web app should be a single origin.
However, you still might want goodapp.com to use facebook.com/login for authentication. The idea of the "stay_in_app" property is that it can be used to enumerate third party URLs which are not part of the app itself but are used by the app for something like third party authentication. The app will not capture navigations to those URLs, but if the user is already in the context of the app and then navigate to one those URLs they will stay in the app rather than being kicked out to the browser.
Jonas explains this more thoroughly in the thread linked to above.
I agree that if an unbounded scope or a stay_in_app URL causes an app window to be navigated away from its original origin, that the user should be made aware of the new origin they just entered. I don't think subdomains can be assumed to be part of the same app though because many shared hosting services used subdomains to separate sites.
On November 17, 2014 at 10:09:22 AM, Ben Francis (firstname.lastname@example.org) wrote:
True. But we've not experimented with this enough to know if scopes are the right thing here.
Exactly. Hence the same origin restriction.
Why is window.open() not sufficient? What am I missing?
Agree. For example, GitHub pages.
A lot of existing web content navigates the top level browsing context to another origin for authentication. There are web apps which span multiple origins for other reasons too (e.g. both http:// and https:// URLs or different origins for different parts of the app). The stay-in-app property is a proposal of how to be backwards compatible with this behaviour in-app, whilst not allowing malicious apps to capture outside navigations from other origins (I think capturing/deep linking is a key use case we should be designing for).
The Manifest spec could enforce that all top level navigations for an app should come from a single origin, and therefore intentionally not be backwards compatible with some content, but that should be a conscious decision.
I agree that window.open() could offer an alternative to the stay-in-app property if app developers author their apps that way. But I actually think the default behaviour for window.open() or a target=_blank hyperlink to a URL outside of the app's defined scope should be to open outside the context of the app (e.g. in the browser). The stay-in-app property would be one way to override this default, another way would be to pass a special argument to window.open().
To throw something else into the mix...
The scope of Service Workers is "restricted by the path of the Service Worker script unless the Service-Worker-Scope: ... header is set." 
We could follow the example of Service Workers and make the maximum scope of a manifest be the path of the manifest file (i.e. the same directory). This would prevent an app on a shared hosting site (e.g. http://example.com/~jack/manifest.json) from registering to manage URLs of another app (e.g. http://example.com/~jill/index.html). We could even go as far as to make this the default scope, though that would again not be backwards compatible with some content. I believe this is basically how Amazon web app manifests handle scope  and it's a pretty neat solution.
@benfrancis, I'm not a fan of binding scope to the location of a resource (because resources can, and should be able to, move and redirect independently of the navigation scope).
I think keeping Chrome's current behavior + and scoping to a single URL will work ok for now. I'm preparing a PR based on that.