New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Why does this spec replicate HTML features? #97

Closed
marcoscaceres opened this Issue Dec 6, 2013 · 35 comments

Comments

Projects
None yet
9 participants
@marcoscaceres
Member

marcoscaceres commented Dec 6, 2013

Elsewhere, @pornel wrote:

We already have <meta>/<link> for most of the manifest:

    {
      "name": "My App",
      "url": "/start.html",
      "mode": "standalone",
      "icons": [{
          "src": "icon/lowres",
          "width": "64",
          "type": "image/webp"
        }, {
          "src": "icon/hd",
          "width": "128"
      }]
    }

Can be written today, without reinventing the wheel, in a backwards-compatible way:

    <meta name="application-name" name="My App">
    <link rel=start href="/start.html">
    <meta name="mobile-web-app-capable" content="yes">
    <link rel=icon sizes="64x64" type="image/webp" href="icon/lowres">
    <link rel=icon sizes="128x128" href="icon/hd">

And unlike JSON, it supports comments, internationalization and doesn't invalidate entire file because of a single trailing comma :)

    <meta name="application-name" name="My App" lang="en">
    <meta name="application-name" name="Moja Apka" lang="pl">
    <!-- I can comment things without breaking syntax, wow such amaze -->
    <link rel=icon href="es.png" hreflang=es>
    <!-- text in the icon? No problemo! -->
@marcoscaceres

This comment has been minimized.

Member

marcoscaceres commented Dec 6, 2013

Chrome for Android is moving to support more of the above HTML elements as fallback and don't rely on a manifest.

For example, this bug adds support for application-name:
https://code.google.com/p/chromium/issues/detail?id=323307

This one adds support for using HTML's rel="shortcut icon":
https://code.google.com/p/chromium/issues/detail?id=296962

Ideally, we should only accept favicons that follow the specifications suggestion. <link rel=”shortcut icon” sizes=”128x128” href=”niceicon.png”> [http://www.whatwg.org/specs/web-apps/current-work/multipage/links.html#rel-icon]

@kenchris

This comment has been minimized.

Collaborator

kenchris commented Dec 6, 2013

One concern is the parsing of the content attribute of the meta tag, which is not really standardized and very quirky in current implementations.

@kenchris

This comment has been minimized.

Collaborator

kenchris commented Dec 6, 2013

Meta tags are easy to (ab)use as they are simple to add, so adding one for Chrome is not very invasive. I understand why they choose that, when it is not standardized.

@kenchris

This comment has been minimized.

Collaborator

kenchris commented Dec 6, 2013

For instance with viewport, some browser vendors implemented ; as separator because they though it was working as such on Safari, but safari just ignores trailing "trash"

@kornelski

This comment has been minimized.

kornelski commented Dec 6, 2013

One concern is the parsing of the content attribute of the meta tag

The HTML spec or the manifest spec can define parsing rules for relevant <meta> values.

Meta tags are easy to (ab)use as they are simple to add

The HTML spec defines some basic values and defers to a a registry with links to specs for other <meta> values.

Of course there's nothing stopping anybody from using any new unregistered name, but OTOH that doesn't stop manifest spec from using existing well-defined names.

<meta viewport> is a failure of standardization indeed, but that's more of specific issue with Apple's lack of effort to standardize it early and W3C's too late reaction. If the manifest spec defines <meta> values from the start, then vendors won't have to reverse-engineer and second-guess the syntax, so this problem is entirely avoidable.

@kornelski

This comment has been minimized.

kornelski commented Dec 6, 2013

I suggest, at very least, that the manifest spec should define as many properties as possible in terms of existing <meta> or <link> values, e.g. instead of defining name property from scratch it should define name as being alias for <meta name=application-name>.

Regardless of whether the manifest spec will reinvent its own properties or refer to existing meta tags, somebody will have to define what happens when a page uses both manifest name and meta application-name, etc., so addressing this in the manifest spec from the start may be best for interoperability.

@marcoscaceres

This comment has been minimized.

Member

marcoscaceres commented Dec 9, 2013

@pornel I agree. The spec should provide fallback for all things it overwrites explicitly (and makes sure that it falls back to HTML when stuff is missing). At the same time, it might also be important to make sure meta equivalents are provided in HTML for new things the manifest format introduces (e.g., some solution for orientation hinting).

@marcoscaceres

This comment has been minimized.

Member

marcoscaceres commented Dec 9, 2013

I'll also note that if we reuse the HTML meta names, as @pornel suggests, we save a lot of bikeshedding.

@kenchris

This comment has been minimized.

Collaborator

kenchris commented Dec 9, 2013

With JSON we have the benefit that is can be easily created (more easy and slightly cleaner than HTML) which is good for online stores (like what Moz is attempting).

Having worked on translation before, having all the worlds languages in the manifest is going to show down loading and parsing of the data, if not introduce corruptions once in a while due to wrong encoding etc. An online store etc could just generate another json given the user language and it ties nicely in with ServiceWorker.

@kornelski

This comment has been minimized.

kornelski commented Dec 9, 2013

@kenchris Every dynamic web page by definition has ability to create HTML.

As for "clean", I wonder whether draconian (and otherwise unspecified) JSON error handling will hold long-term. It's a prisoner's dilemma. It happened to HTML and RSS already: as the format gets more popular, inevitably there will be slightly syntactically broken files, and when any major consumer starts accepting these, everybody will have to. It starts happening to JSON already.

Lack of good i18n in JSON is indeed a problem. HTTP language negotiation or JS-initiated ServiceWorkers are not easily discoverable by crawlers.

@matthewp

This comment has been minimized.

matthewp commented Dec 12, 2013

I have to agree with @marcoscaceres and @pornel on this one, manifest files have felt very wrong since moz introduced theirs. Another benefit of using standard html tags is that a user agent can intelligently upgrade an "almost web-app" to a web-app. Makes bookmarking to homescreen much easier when there is 1 identification process.

@brucelawson

This comment has been minimized.

brucelawson commented Dec 12, 2013

What about the desire people have for an external manifest file? (Why would they want to?)

@beverloo

This comment has been minimized.

Member

beverloo commented Dec 12, 2013

While it indeed is possible to implement pretty much all functionality offered by the manifest in HTML tags, I'd like to note a few benefits I see in the approach of using an external manifest.

1) External manifest files can be cached.

Marcos' example lists five lines of and elements which replicate the behavior specified in the manifest. The manifest is 164 bytes long, the HTML approach 241 bytes. While the difference is negligible, all of the information is static to the website and does not change for page-to-page.

We're talking about adding four or five more meta tags for being able to customize some of the aspects of Chrome for Android's add-to-homescreen feature, namely for being able to modify the title, setting a launch URL, and so on.

The problem here is the naming. Do we introduce a name consistent to the current naming (mobile-web-app-capable), support Apple's scheme (apple-mobile-web-app-title) or go for Facebook's Open Graph (og:title)? Twitter has their own tag (twitter:title) for Twitter cards as well. In every case, we're talking about introducing yet another tag developers will have to send with each of their pages.

More advanced web applications will have to send a ton of properties for Chrome, a ton of properties for Safari, perhaps yet another format for Mozilla or IE, meta information to be used on Twitter cards and more meta information to be used on Facebook and Google+ references. This strongly advocates the need for UA-sniffing, which unfortunately is something most large players are already forced to do.

Externalizing this data solves the problem of having to send it with every page. Standardizing the properties within the manifest solves the problem of everyone doing their own thing by providing a canonical source of the information.

2) Values can be objects or arrays rather than just strings.

This solves the problem of the awkward and highly inconsistent formats the values of tags are currently using, especially when the tag's value can contain several sub-values (icons, viewports, notification-bar settings).

For example, say that the New York Times wants to reference their native applications in the various stores. Using tags, you would end up with something like:

<meta name="apple-itunes-app" content="app-id=357066198, affiliate-data=MyWebsite" />
<meta name="google-play-id" content="com.nytimes.android" />
<meta name="windows-store-uuid" content="8e15939d-8469-483c-9fb8-94c2e72496f6" />
<meta name="blackberry-app-id" content="21197892" />

Instead, you could refer to them in the manifest by specifing them in a simple map containing whatever meta data a vendor would like to be able to support:

"application-stores": [
  "itunes": [
    "app-id": 357066198,
    "affiliate-data": "MyWebsite"
  ],
  "google-play": "com.nytimes.android",
  "windows-store": "8e15939d-8469-483c-9fb8-94c2e72496f6",
  "blackberry": "21197892"
],

It feels a bit more verbose due to the larger number of lines (but in fact has less bytes), but is easier to parse and extend.

In regards to JSON's draconian error handling, and also on the subject of not having the ability to comment in manifests, I really think this is something we should just try to fix in JSON. I know all current parsers will reject this (aside of eval()), but the longer we wait with introducing it harder it will be to change it. It's out of scope for this discussion, however.

@marcoscaceres Whilst these bugs exist on the Chromium bug tracker, they are suggestions and no decision has been made yet about how Chrome for Android will implement support for additional icons, titles and so on.

@kornelski

This comment has been minimized.

kornelski commented Dec 12, 2013

Thanks @beverloo, these are good arguments.

External manifest files can be cached.

gzip eliminates cost of markup overhead in both cases, but you're right that it could lead to duplication on subpages. How about reusing manifest-specific <meta> from another page (which will likely be cached)?

<link rel=manifest href="/" type="text/html">

This way you could reuse app metadata of your homepage for convenience and likely bandwidth saving thanks to caching, or if that is not appropriate for some reason, generate a dedicated file.

More advanced web applications will have to send a ton of properties for Chrome, a ton of properties for Safari, perhaps yet another format for Mozilla or IE

We're developing a standard here, so if we succeed authors will only need to set properties once for all browsers (we can set standard meta names without switching to another format).

Values can be objects or arrays rather than just strings.

Indeed, deeply nested arrays/properties are awkward in <meta>/<link>. However, equivalent to top-level array is simple and usually there's a way to avoid deep nesting. In this particular case I'd suggest:

<link rel="alternate app" href="http://itunes.apple.com?id=…">
<link rel="alternate app" href="http://play.google.com?id=…">
<link rel="alternate app" href="http://store.blackberry.com?id=…">

For more complicated cases I suggested extending <meta>.

@brucelawson

This comment has been minimized.

brucelawson commented Dec 12, 2013

Is it not possible to have an external HTML fragment with lots of meta tags that's imported in, so it looks
like HTML but has the single-fetch advantage? (Sounds a bit HTML imports-like, but I don't know web components deeply enough to know if this is similar). Would perhaps be slightly bigger than JSON but it's familar syntax, familiar names,allows comments and is fetched once then cached.

@matthewp

This comment has been minimized.

matthewp commented Dec 12, 2013

@beverloo Good point about the awkwardness of having nested values in a meta tag. But this issue exists with or without a w3c manifest file. Why not just fix meta tags? Something like:

<meta name="application-stores">
  <meta name="google-play" content="com.nytimes.android">
  ...
</meta>

or if that breaks the world we invent a new tag that allows for this.

@brucelawson

This comment has been minimized.

brucelawson commented Dec 12, 2013

@matthewp, we can't invent a new tag in the head. Any new tag in the head gets re-parented into the body, with any subsequent valid head-dwelling elements.

@kornelski

This comment has been minimized.

kornelski commented Dec 12, 2013

@brucelawson breaking of <head> in this case isn't a big problem. IIRC microdata made <meta> and <link> in <body> allowed.

Since there are no existing consumers of the hypothetical future manifest/meta-grouping element there is no backwards compatibility issue. Everybody who needs to parse it will be up to date with <head>-allowed elements.

Pages that don't want to break <head> in older browsers can just move the new element to the end of <head>.

@triblondon

This comment has been minimized.

triblondon commented Dec 14, 2013

I think this discussion needs to track back to a higher level point - what is an application manifest trying to achieve, and is that goal any different from <meta> tags? I think the goal has to be one of these two:

  1. We've got an entirely new use case for which we need a new solution and it's nowt to do with meta tags, which are an existing solution to a different problem.
  2. We're effectively saying meta tags are a bit crap, and we want something better for basically the same purpose we're already using meta tags for

I think it's 2. So with that in mind, why don't we acknowledge that what we're talking about is a generic way of expressing metadata about a document (call it an application if you must), and we want to be able to express that metadata in a way that avoid existing pitfalls with meta tags. Specifically the features we want that we can't currently achieve with meta tags are:

  • Externally linkable (and therefore cachable where the same meta data applies to a large number of pages)
  • Ability to embed non-scalar data
  • Terse, JSON-like syntax

So, possibly a new standard that allows for metadata to be loaded via a <link> might be the best way of doing this, but we should be clear about how this works with existing meta tags.

Most of all, I desperately don't want to have to define both meta tags and countless proprietary external manifests just because everyone sees this as a different problem. I say it's all the same problem.

@andreasbovens

This comment has been minimized.

andreasbovens commented Dec 17, 2013

Yesterday, I tweeted that iOS 7.1 will support a minimal-ui property as part of the viewport meta tag, allowing authors to reduce the browser's visible UI to a bare minimum. As indicated, I believe that the viewport meta tag is not quite the right place to specify this (another <meta> tag would be better; a standardized solution even more so), even though being able to reduce the UI is a cool developer feature.

However, I'm pretty sure devs will jump on this and start using it even before iOS 7.1 is released (case in point: https://twitter.com/mathiasvalentin/status/412897141237899265 and similar comments). That's partially because anything <meta>, despite its obvious flaws, is somehow easy-to-implement magic sauce with immediate effect. Placing a manifest with strict syntax criteria, duplicating a bunch of things that are present in <head> in the root of your site (or even in the head of the page) is somehow harder, and less likely to be adopted. I'm happy to be proven wrong, but that's my impression.

Another example: looking at @beverloo 's example of native app references for instance, I obviously see that JSON offers more flexibility, but at the same time, the 4 lines of <meta> tags are somehow easier on the eyes and, I believe, ultimately, more likely to be used.

So, I'm just throwing this out here. I have no strong feelings about what the "right" format should look like, and will be happy to evangelize whatever comes out of this discussion, but from a developer adoption point of view, <meta> tags have a bigger chance of succeeding, I believe.

@brucelawson

This comment has been minimized.

brucelawson commented Dec 17, 2013

I think Andreas is right here "I have no strong feelings about what the "right" format should look like, and will be happy to evangelize whatever comes out of this discussion, but from a developer adoption point of view, tags have a bigger chance of succeeding, I believe." (and not just because he's my boss).

It feels like re-inventing a wheel. http://www.w3.org/TR/html-design-principles/#do-not-reinvent-the-wheel. Or, with a terse, perfect but brittle and unfamiliar syntax, like widgets' config.xml

@marcoscaceres

This comment has been minimized.

Member

marcoscaceres commented Dec 18, 2013

@beverloo wrote:

Externalizing this data solves the problem of having to send it with every page. Standardizing the properties within the manifest solves the problem of everyone doing their own thing by providing a canonical source of the information.

Yes, externalizing does DRY-up the HTML + brings the benefits of caching. But the problem of proprietary extensions don’t go away. It’s impossible to keep up with proprietary extensions (citing the explosion of members that FxOS keeps adding + seeing what happened with W3C widgets - see Cordova's extensions to config.xml) - and people don’t bring them for standardization, so we just expect them to happen.

This is again another significant limitation with JSON - it has no means to support "distributed extensibility" and thus a good way to support proprietary extensions (neither does HTML, really - but at least it meta and data- provides a way to get around that … or worst case, one falls back to XHTML and uses namespaces... hehe, only joking 😆).

So, currently, the manifest spec just throws in the towel and recommends using simple vendor prefixes - I think that’s already indicative of the limitations with JSON.

To be clear, I'm not dismissing JSON here - but we need to be realistic about what is going to happen and make sure we understand the implications for the platform in the long term.

Also, I don't think adding error handling to JSON will happen in standards land (not for a while at least, as the JSON guys are still fighting with the TC39 guys over who actually owns JSON and there is no talk of JSON 2 yet) - it will likely happen in the way @pornel describes, where some vendor will just accidentally on purpose relax the rules to accommodate some large content provider or it will just be a bug that devs exploit on mass.

@marcoscaceres

This comment has been minimized.

Member

marcoscaceres commented Dec 18, 2013

@triblondon wrote:

We're effectively saying meta tags are a bit crap, and we want something better for basically the same purpose we're already using meta tags for

The problem is that <meta> and <link>, which the proposal duplicates, are not crap at all. They are actually very powerful and currently much more flexible than what we have in the manifest spec. @pornel already demonstrated some of the niceties (commenting, internationalization, and a well-defined parsing model for both elements and attributes (the manifest spec even defers to HTML for the parsing rule of certain JSON members, like sizes). In addition, HTML already defines the (non-blocking) fetching model for links of type icon). By virtue of being HTML elements, they also benefit from inheriting the global attributes, plus participate dynamically in the DOM. The JSON document trades all that in for terseness, non-scalar data.

Let me show some of the magic - consider that in the case for icons, link rel=icon allows a user agent to discriminate on “media”, “type”, and “sizes” through the attribute. From a responsive images perspective, that’s the "holy trinity":

<!-- bookmark/address bar icon -->
<link rel="icon" href="favicon.ico" sizes="16x16 32x32">  

<!-- these all look good on 1 to 1.5, covers 24-128px sizes gamut -->
<link rel="icon" href="low-rez-icons.ico" media="(max-device-pixel-ratio: 1.5)">

<!-- only for hi-dpi displays, more details - covers 24-256px sizes gamut for retina -->
<link rel="icon" href="hi-rez.ico" media="(min-device-pixel-ratio: 1.51)">
<link rel="icon" href="foo.webp" type="image/webp" sizes="512x512" media="(min-device-pixel-ratio: 3)">

<!-- only for super hi-dpi displays that support webp! and need big-ass icons -->
<link rel="icon" href="foo.webp" type="image/webp" sizes="512x512" media="(min-device-pixel-ratio: 3)">
<link rel="icon" href="foo.webp" type="image/webp" sizes="1024x1024" media="(min-device-pixel-ratio: 3)">

Most of all, I desperately don't want to have to define both meta tags and countless proprietary external manifests just because everyone sees this as a different problem. I say it's all the same problem.

I think we even need to go up one more level than the one you suggest @triblondon. The problems we seem to be trying to solve are independent of JSON or HTML:

  1. Provide enough metadata to an OS that, when added to the home screen, a web application becomes indistinguishable from a native application - that is, home screen, task switcher, settings menu integration (at least!).
  2. Able to run outside the browser (or give the user the perception that the application is running outside the browser).
  3. Able to start in a particular orientation.
  4. Able to start fullscreen.
  5. (Not a great thing, but...) a means to add proprietary stuff to these apps outside the HTML.

1 (OS integration) is mostly solved by HTML - this is mostly application-name and link rel=icon's purpose (nice to have would be a short name for an app, but we can live without it).

2 (run outside browser) is basically "apple-mobile-web-app-capable" and its new buddy "minimum-ui" - we could just bite-the-bullet and standardize "mobile-web-app-capable" in HTML (and then work out what the hell to do when a link is clicked... navigation with no back button becomes fun ... see #95 :)).

3 is not supported by HTML out of the box. But could be done through a meta or via CSS @viewport in the <head>. Doing it with CSS provides a ton of advantages over JSON - basically you get all the media query machinery for free: media query-based orientation, ftw. See #74 (comment)

4 (start in fullscreen) is not supported by HTML - the only solution is to use the Fullscreen API (which gets messy). Could also be solved with a meta tag or CSS by extending @Viewport.

5 (proprietary extensions) is currently meta's forte.

So, what we are weighing up here is whether we add 3 (orientation hinting) and 4 (fullscreen hinting) on startup as either JSON or HTML ... plus a somewhat crappy way for people to add proprietary functionality to the platform. The rest is just mostly redundant duplication for the sake of DRYing up HTML a bit + maybe saving a few bytes on caching (at the cost of an extra HTTP request, though it's deferred until after the document has loaded or till when needed - so it's not too bad).

I hope this helps frame the problem a bit better - at least how I understand it.

@marcoscaceres

This comment has been minimized.

Member

marcoscaceres commented Dec 18, 2013

Forgot to mention the starturl link relationship. That's also missing from HTML ... the closest we have is Microsoft's msapplication-starturl. Micro format's link relationship registry has "home", but the semantics are not quite right.

@marcoscaceres

This comment has been minimized.

Member

marcoscaceres commented Jan 16, 2014

Hixie made some changes in HTML to better support internationalization of application-name:
http://html5.org/tools/web-apps-tracker?from=8347&to=8349

He also clarified the icon's size attribute at bit so to make it clear that sizes means "raw size" in pixels - so to better support high-dpi displays.

@marcoscaceres

This comment has been minimized.

Member

marcoscaceres commented Jan 22, 2014

Ok, I'm working on explicitly forcing UAs to fallback to HTML elements when things are missing from the manifest. And to always treat manifest members as their HTML counterparts. I will enforce this in the test suite.

@marcoscaceres

This comment has been minimized.

Member

marcoscaceres commented Jan 27, 2014

Emailed the TAG about the fall back strategy to get their opinion: http://lists.w3.org/Archives/Public/www-tag/2014Jan/0139.html

@matthewp

This comment has been minimized.

matthewp commented Feb 11, 2014

Hey @marcoscaceres I read the TAG thread and didn't get a clear answer on how this will proceed.. Mind summarizing?

@marcoscaceres

This comment has been minimized.

Member

marcoscaceres commented Feb 11, 2014

We've removed name and icons from the manifest. There is now nothing in the manifest that overlaps with what HTML provides, so this problem goes away (for now, at least).

@marcoscaceres marcoscaceres added this to the Last Call Working Draft milestone Feb 11, 2014

@matthewp

This comment has been minimized.

matthewp commented Feb 11, 2014

Sweet! Thanks for taking charge on this.

@marcoscaceres

This comment has been minimized.

Member

marcoscaceres commented Feb 14, 2014

After discussions with @sicking, I've put name and icons back into the spec. The fallback model is:

  1. If name and/or icons in manifest, use those. Ignore the HTML ones.
  2. If either name or icon are missing, fallback to HTML.

In 2, it's RECOMMENDED devs and implementors choose "application-name" and <link rel=icon> over proprietary stuff, of course.

It's personally not what I think is ideal (I preferred no name or icons in the manifest), but it's a fair compromise. And there may be some benefits to having name and icon in the manifest: centralization and manifest-based updates to the metadata, so devs can update their icons and the apps name (though we still need a security model for that, as if it happens automatically an app can change itself to look like another app... Which would be bad).

At least now we can move forward onto more interesting problems.

@matthewp

This comment has been minimized.

matthewp commented Feb 14, 2014

As long as we continue to have the HTML fallback, realistically few UAs are going to adopt the manifest in the short or mid term.

@kornelski

This comment has been minimized.

kornelski commented Feb 14, 2014

@matthewp Lack of backwards compatibility doesn't seem like a good motivation to adopt a new technology. Manifest should be adopted because it's a better solution.

@marcoscaceres

This comment has been minimized.

Member

marcoscaceres commented Feb 14, 2014

what @pornel said. It should be ok, but it might mean that we don't end up with good support for application-name and link rel=icon on the platform. Anyway, let's see how this plays out over the next 2 years. Still a long road ahead before any of this becomes a reality.

@sicking

This comment has been minimized.

sicking commented Feb 14, 2014

I enumerated a few reasons why having name and icons in the manifest makes more sense than relying on the HTML file here: http://lists.w3.org/Archives/Public/www-tag/2014Feb/0039.html

Once we see the manifest spec being successful, I think we should make the HTML spec specifically mark the application-name as deprecated and encourage authors to use the manifest instead. Not sure if the same makes sense for icons since the icons is also displayed in the URL-bar in most UAs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment