Processing the manifest is no longer a function of document URL

[mgiuca]

This change (choose one):

• Breaks existing normative behavior (please add label "breaking")
• Adds new normative requirements
• Adds new normative recommendations or optional items
• Makes only editorial changes (only changes informative sections, or
  changes normative sections without changing behavior)
• Is a "chore" (metadata, formatting, fixing warnings, etc).

Implementation commitment (delete if not making normative changes):

• Safari (confirmed non-breaking, see below comment by @hober)
• Chrome (non-breaking)
• Firefox (confirmed non-breaking, see below comment by @NotWoods)
• Edge (should match Chrome)

Commit message:

If the start URL is missing or invalid, the processed manifest simply does not have a start_url, rather than defaulting to the document URL. However, in the steps to determine the installability of the document, the old behaviour is maintained (a manifest without a start_url is allowed to use the document URL).

This doesn't actually break existing behaviour, but it enables new user agent behaviours:

1. User agents are now allowed to consider a document with an invalid start_url to be non-installable (they are not required to use the document URL).
2. It is now possible to consider a manifest without an associated document. For example, a service can directly install a manifest without considering a particular document as initiating that install flow. (In practice, this was already being done.)

Closes #668 and #669.

---

Preview | Diff

[mgiuca]

Needs review and also analysis of how implementations behave to see if this will actually break sites.

From my vague understanding, at least Chrome already considers sites to be non-installable if they have no start URL, so this change doesn't affect how sites are treated.

[NotWoods]

Firefox Preview also considers sites to be non-installable if the start URL is missing.

[dominickng]

LGTM :D

[alancutter]

Is "in scope of a URL" defined? What does that mean for document URL https://example.com/a/b/c/page.html? That the start_url must have that as a prefix?

[fallaciousreasoning]

There's a section in the navigation-scope section of the spec defining in scope. https://www.w3.org/TR/appmanifest/#navigation-scope

[alancutter]

Yeah, that behaviour doesn't sound appropriate here. Would be good to have test cases for this (input -> output examples).

[mgiuca]

Oops. It's been awhile since I wrote this so I'm not sure what my intention was, but I suspect this was simply a mistake. It should have said "not same origin as", not "not within scope of".

That matches the existing check (which I'm removing), that would check if start_url was same origin as the document URL, and if not, ignore the supplied start_url and default to the document URL. Only now, we just consider it non-installable.

Does that sound right?

[alancutter]

That sounds right.

[alancutter]

The money shot.

[alancutter]

This comment is for line 1506. start_url can now be undefined, the processing for scope segfaults.

[mgiuca]

Specs don't segfault :)

This is WAI. "undefined" here doesn't mean set it to nullptr in C++. It means (according to how a JS value is converted into a dictionary member) to leave it empty as if the key didn't exist at all. So this logic means to treat an invalid start_url exactly the same as a missing start_url.

(Note that the WebIDL data structure permits only two options for start_url: missing or a valid-but-possibly-empty string. So it should not be possible to store some other null-type value in here.)

C++ implementations should make sure this is the case and of course not segfault.

[alancutter]

Can we default to processing '.' instead of returning undefined?

[mgiuca]

Essentially, that would mean if start_url is invalid, it defaults to the parent directory of the manifest file.

Currently, Chrome and Firefox (at least) do not consider a site installable if start_url is missing or invalid, behaviour which I am attempting to enshrine by returning undefined (effectively treating it as missing) and by explicitly saying the document is not installable if start_url is not present. I think it's desirable / conservative to fail in these bad cases, rather than try to come up with some "reasonable" default.

Also this default can go horribly wrong: if the manifest is hosted on a CDN, this would cause both the start_url and scope to be set to the parent directory on that CDN, which would likely be a 404 or a directory listing page.

[mgiuca]

@hober would you be able to confirm (or assign someone on Safari to confirm) whether this will break compatibility with how Safari works when installing a site.

TL;DR: Currently the manifest says if start_url is missing or invalid, use the URL of the current document. We are proposing that it simply be non-installable (which is how Chrome and Firefox already behave).

[mgiuca]

@othermaciej or @hober : Hi Safari folks, would you be able to give guidance on whether the above change would result in breakage to Safari?

[hober]

@mgiuca I asked the relevant folks internally and they think this change is reasonable.

[mgiuca]

Thanks @hober . That means we can land this (after resolving merge conflicts).

[marcoscaceres]

Given that user agents let you install whatever you want (via "add to home screen") - maybe we should say:

Suggested change

	{{Document/URL}}, consider the document not <a>installable</a>.
	{{Document/URL}}, a user agent MAY consider the document not <a>installable</a>.

[aarongustafson]

I like this suggestion. Edge also has the "Install Site as an App" (or similar) menu option for what were "Pinned Sites".

[mgiuca]

Ah, I didn't see this before. It's not so simple that we can simply say "MAY" here; otherwise that would allow user agents to install a document linking a manifest whose start_url and scope are on a different origin (where previous to this PR, we reset the start_url and scope to the document URL).

So I think if we change this, we need to have some wording similar to the "non-fatal" case of step 2 above, where we say "the user agent MAY either: 1. set start_url to the document URL and scope to match, or 2. consider the document not installable."

You're right, Chromium browsers allow this, but I'm sure we don't just let you install a manifest that has its start and scope on another origin. I'll figure out what they do.

[dominickng]

In Chromium we currently enforce that start_url and scope are same-origin with the document URL, which the spec requires. If that isn't met, we discard the values.

When this happens, Chromium browsers use the document URL in place of the manifest values when a site has a shortcut created for it. That's behaviour outside the scope of the manifest spec and a browser choice.

Since this PR removes that same-origin requirement, we would remove that enforcement in our parsing when updating to match this.

[marcoscaceres]

In Chromium we currently enforce that start_url and scope are same-origin with the document URL, which the spec requires. If that isn't met, we discard the values.

I think treating start_url as it if it was not present when it's not same origin or outside the scope makes sense, as does assigning it a sensible default (document URL) when in error. That can then remove the "installable" decision at that point (that shouldn't be part of the processing algorithm, IMO).

[mgiuca]

Right, so that does unfortunately mean that processing the manifest is still a function of the document URL (if we say "discard start_url and scope if they are not same-origin as document URL and replace them with document URL and the dir of the document URL, respectively"). That's explicitly what I was trying to avoid here.

I think I can live with it if we do that as a separate step outside the "processing the manifest".

That basically means we do "processing the manifest" which is NOT a function of the document URL, and then in a step outside of that, we do the above check if you're trying to install a document.

Marcos, I think you're right, having "installability of a document" doesn't make a lot of sense since user agents can always choose to install any document, and in any case, our current steps to determine installability are MAY MAY MAY. So we can scrap that. But I think we do need to replace this with some other steps for essentially "sanitizing a manifest for installation from a document".

What I want to allow is basically two separate high-level operations:

1. Installing a manifest. (Not associated with a document at all.) That's essentially what's enabled by this PR, by allowing the user agent to process a manifest without a document URL, then install it.
2. Installing a document. Get the associated manifest, if there is one. If there isn't, you can make one up. But this step should ensure that the manifest is same-origin as the document, to prevent sites from being able to install PWAs on another origin.

I propose that 1 is taken care of by just the definition "install" (what that means is UA-specific). And for 2, we should replace "steps to determine the installability of the document" with the following:

Steps to install a document, given a processed manifest manifest:

1. Let manifest and manifest URL be the result of obtaining the manifest.
2. If obtaining the manifest results in an error, set manifest to a processed manifest, populated from the top-level browsing context Document's metadata in a user-agent-specific way (e.g., setting manifest["name"] to the document title).
3. If manifest["start_url"] is not present, or not same origin as the Document URL, set manifest["start_url"] to the Document URL, and set manifest["scope"] to the result of parsing "." using the Document URL as the base URL.
4. Install the manifest.

This is too much work to do in this PR, so I think I'll just continue propping up the "installability of the document" concept right now, and do the above later after some more discussion.

[mgiuca]

OK, I've updated the PR. I filed a bug #912 about the above issue (wherein Marcos and I agree, we shouldn't have "installability of the document") but I haven't tackled it here.

I've updated the text so the UA can now either consider the document not installable, OR can use the document URL to replace the off-origin start_url.

And I've updated the commit description. Actually, this is no longer a breaking change (!). You can carry on behaving exactly as before, if you like, which is replacing an invalid or off-origin start_url with the document URL, or you can error out (newly allowed behaviour). In one sense, this is now just a (useful) refactor, moving the replace-start-with-document logic up out of the manifest processor and into the "steps to determine installability of the document".

The next step in #912 will be rewriting those steps to be about postprocessing the manifest, not determining installability.

@marcoscaceres how is this?

[marcoscaceres]

This looks fine - see my one suggestion.... @mgiuca, do you still want to proceed with this?

[mgiuca]

Yep.

[marcoscaceres]

Ok cool, please merge whenever you are ready.

[mgiuca]

Doing a last-minute re-read. There's a bit of a complication (as usual, because of the unfixable awkwardness that scope is based on start_url and not the other way around):

1. This PR introduces the possibility that start_url of a processed manifest is undefined. (Previously, this was impossible; a missing or invalid start_url would default to the document URL. Note that fixing this is the whole point: since there are cases where you don't have a document URL, this was unimplementable.)
2. The "processing the scope member" is unchanged by this PR. It assumes start URL is a valid URL.
3. If start URL is missing from these steps, Step 1: default will be failure (it is illegal to URL-parse "." without a base).
4. Step 5: "within scope" isn't defined when the URL is undefined. We'll assume it returns false in this case.
5. Any attempt to return default will mean that scope of a processed manifest is failure.

So if start_url is omitted, the manifest processing kinda explodes. The simplest fix for this would be to simply say "if start URL is undefined, scope is undefined". That raises a follow-on possibility of what things will do when scope is undefined. (For example, "within scope of a manifest" is meaningless when scope is undefined due to start_url being undefined, though that one's easy to fix by simply making it always false.)

Need to think a bit more about this before landing.

[marcoscaceres]

oh, lol, "insatiability"... curse you autocorrect.

[mgiuca]

Quick note that the bulk of my reply is on this discussion thread, in case you missed it.

(I often can't find GitHub discussion threads so I wonder if I should have just pasted my reply at the bottom where people can see it :/)

[mgiuca]

We have identified one potential breakage of existing behaviour with this change:

If you have a manifest with no (or invalid) start_url, then:

• Under the current spec, any shortcut URLs, share targets, file handlers or protocol handlers (the latter three in incubation) will be allowed if they are within scope of the parent directory of the document URL.
• Under the proposed text, any shortcut URLs, share targets, file handlers or protocol handlers will be invalidated unconditionally.

(This is because under the current spec, the scope is set to its fallback value before other URLs are compared using "within scope". Under the proposed text, the scope is set to undefined, then the other URLs are compared using "within scope", and then the user agent has the option of setting the scope to its fallback value.)

For example, say that you are in a document at https://example.com/foo/index.html, with the following valid manifest:

{
  "scope": "https://anothersite.com",
  "icons": [{
    "src": "https://somecdn.com/icon.png"
    }],
  "shortcuts": [{
    "name": "Click me",
    "url": "https://example.com/foo/click.html"
  }]
}

If you quickly eyeball this manifest, you'll see that both the icon and shortcut are on a different origin to the scope, so you might initially think both the icon and shortcut would be rejected. However, icons do not need to be on any particular origin, so those are OK. You'd think, though, that the shortcut would not be valid.

What actually happens, under the current spec, though, is:

1. Since there is no start_url, the start_url defaults to https://example.com/foo/index.html.
2. Since start_url is not within scope of scope, scope is changed to https://example.com/foo/.
3. Since the shortcut's url is now within scope of scope, it is valid.

In fact, the shortcut is valid in this case, which I find quite gross, as the validity of the shortcut depends on which URL you were on when you installed the manifest. If you installed the exact same manifest from https://example.com/foo/bar/, the shortcut would not be valid.

Under my proposed text for this PR, the above case would invalidate the shortcut unconditionally. Now what would happen is:

1. Since there is no start_url, the start_url defaults to undefined.
2. Since start_url is not within scope of scope, scope becomes undefined.
3. Since the shortcut's url is not within scope of scope, it is invalidated and dropped. (In fact, all shortcuts, share targets and file handlers will be invalidated.)
4. At the top level algorithm, the user agent MAY set start_url to https://example.com/foo/index.html, and scope to https://example.com/foo/. But the shortcut remains invalid.

This is a minor breaking change, but I think it's a good change, because:

1. It's bad for the validity of those other URLs to depend on the document URL.
2. This change will only affect websites that have no or invalid start_url.
3. This change only affects very new or experimental APIs (shortcuts, which has not become available in any major browser, share target, file handlers and protocol handlers, which are still in incubation). Share target is the oldest of these APIs, but as far as I know, its only implementation (Chrome for Android) requires a WebAPK which requires a valid start_url. So this won't break any stable APIs.

Edit: Adding the "breaking" label back due to the above scenario.

Edit: Added protocol handlers.

[mgiuca]

Another edge case that changes with this PR: start_url is invalid but scope is valid and encloses the document URL.

For example, say that you are in a document at https://example.com/foo/index.html, with the following valid manifest:

{
  "scope": "https://example.com/"
}

Current spec:

1. start_url defaults to https://example.com/foo/index.html.
2. Since start_url is within scope, scope remains as https://example.com/.

Proposed spec:

1. start_url remains undefined.
2. Since start_url is invalid, it isn't within scope, so scope is undefined.
3. At the top level algorithm, the user agent MAY set start_url to https://example.com/foo/index.html, and scope to https://example.com/foo/.

So the new scheme would be changing scope where the old scheme allows it to remain as defined. Again, this is probably a good thing, since the old scheme allows scope to remain valid or be invalidated, conditional on the document URL, which is what we're trying to fix. But it potentially makes a minor difference to sites that have a scope but no start_url. In my experience, sites tend not to have scope if they have no start_url either, so this shouldn't be much of a problem.

[marcoscaceres]

I think it's too risky to manipulate the scope, because as you point out, there are too many dependents on it.

So, if the scope is:

• invalid or not same origin: invalidate the entire manifest. 🔥
• missing + empty: new URL("./", document.location);

Continue checking the rest (invalidate from there).

[mgiuca]

Yeah I just had a similar thought.

What if you have an app whose start_url and scope are valid, and has shortcuts, share targets, etc, within that scope. The manifest is fully legit. Then you link to that manifest from a document on a different origin. This would replace the start_url and scope with the document, but still allow you to install it. All of the shortcuts and share targets, etc, would remain valid, but no longer be on the same origin as start_url and scope.

I think we have to 🔥 the whole manifest. Actually I think this is the subject of #784.

[mgiuca]

Chatting on the call with @marcoscaceres about nuking the whole manifest vs selectively cleaning it out. @mgiuca would prefer to clean only the members that would become suspect if we changed the scope.

Thinking about creating a definition called "scope-dependent member", then shortcuts and other specs (share target, file handlers, protocol handlers) define them as scope-dependent.

If the manifest has no scope (i.e. because it has no start_url), the manifest processor deletes all scope-dependent members.

[mgiuca]

If the manifest has no scope (i.e. because it has no start_url), the manifest processor deletes all scope-dependent members.

Wait, I just realised, this already happens automatically, due to the change to "within scope" to always return false if scope is undefined.

So we don't need to worry about the case where the manifest has an invalid scope. The only case we need to explicitly invalidate these members is the #784 case: if you link to a manifest from a document whose origin does not match the manifest's valid scope.

[mgiuca]

I did an HTTP Archive analysis, looking for manifests that would break based on the above change.

Details of the analysis

Conclusion: This change will not break any existing sites, since there appear to be no sites relying on the document URL to provide an implicit scope for a share target, shortcut, file handler or protocol handler. (i.e., every manifest in the archive with a share_target, shortcuts or file_handlers also has a start_url).

Further, there are 304 manifests that would experience a change of scope to the parent directory of the document URL, due to there being no start_url present. This seems to have little to no effect on the operation of these sites.

[marcoscaceres]

Thanks @mgiuca for validating our assumptions! As part of this change, it would be great to have some WPTests to make sure we all implement this properly.

[mgiuca]

Yeah... it looks like all the WPTs are manual right now. Is that sufficient?

It kind of sucks because the manifest processor is in theory a very cleanly testable mathematical function (input string -> output data structure, no side effects). Chromium's test suite, for example, includes literally hundreds of tiny test cases that test this processor's expected output, and that's what I've been playing with during this experiment.

But I don't see how to translate that to WPT, since we have no way of verifying the browser's internal representation of the manifest. We can do a handful of manual test cases, but I wouldn't want to translate Chromium's O(hundreds) of tests into manual tests.

TBH, I've given this particular issue way (waaaay) more time than I really can afford, and I need to move on to other things. I don't know if I have the bandwidth to add new tests here (I could update existing tests, but it's a big investment to create them from scratch).

FYI here is the corresponding Chromium CL: https://chromium-review.googlesource.com/c/chromium/src/+/2280807
You can see the updates to the test suite in the third file.

[alancutter]

we have no way of verifying the browser's internal representation of the manifest.

We could expose the parsed manifest back to JS via the Document IDL. It might be useful for web app framework libraries to read.

[marcoscaceres]

We could expose the parsed manifest back to JS via the Document IDL. It might be useful for web app framework libraries to read.

I agree that getting the processed manifest might be nice, but it has implications as it triggers fetching, etc. Probably not something we want to deal with in "v1".

[dmurph]

If I understand this right, we will have 3 notions/options of scope based on invalidations:

1. The scope in the json
2. The first implied scope, from the start_url
3. The second implied scopes, from a start_url that is changed by the document_url

and I believe the proposal is to base the share targets/etc validation on the 2nd of these?

I have a crazy idea here. Looking towards the goal of having the manifest being the source of truth for a web app (where we should be able to install a web app with just a manifest) what if we introduce a new noun into the spec - an 'isolated' or 'isolatable' manifest:

An 'isolated' manifest is a manifest that has a start_url that includes an origin. 'isolated' manifests are applicable to installing/applying to a document_url IFF the document_url is the same origin as the start_url.

This gets rid of the possibility of 3. above, and also doesn't seem to break anything currently (as per mgiuca@'s analysis above). This also starts reenforcing the idea of a manifest being 'all that you need' to install a web app, and starts that definition as something that has a start_url including an origin. It also doesn't invalidate manifest that, say, only have a theme_color, as that is fine and valid. This manifest is not 'isolated', and going towards the goal here, that manifest wouldn't be used to fully describe a webapp to be installed.

[alancutter]

Does this mean non-isolated manifests can't be installed? I think we still want to allow this and just block powerful capabilites like file handlers.

[dmurph]

No - all manifests can be installed except isolated manifests that have been invalidated due to the document url being a different origin from the start_url origin.

Having an origin in your start_url basically just adds a new constraint on the manifest being valid (and installable)- that the document url origin is the same origin as the start_url

[marcoscaceres]

I'm worried about doing this for v1, tbh... we should initially try to specify for the behavior of Chrome, Safari, and Firefox and how they work in relation to the document. We have a whole bunch of interop issues to deal with just with that.

[dmurph]

I'm hoping I didn't bikeshed this too much! You can ignore my crazy idea :)