Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generic programs can't reliably use/manipulate documents via the DOM #640

Closed
mrmr1993 opened this issue May 8, 2017 · 67 comments
Closed

Comments

@mrmr1993
Copy link

mrmr1993 commented May 8, 2017

Alternative title: DOM does not reliably model document objects with closed shadow DOMs.

A generic program (ie. a program that uses the DOM to interact with a document of which it has no prior knowledge) has no reliable interface with which it can interact with and/or monitor documents. In particular:

  • the true active element in the document cannot be discovered when inside a closed shadow DOM. Thus,
    • the program cannot reason about the default action associated with eg. keydown, keyup and click events, and so cannot predict behaviour in response to user input or its own simulated input.
    • the program cannot reliably direct or redirect the focus of the page in order to perform further actions
    • a selection cannot be read (except as a string by Selection.toString) or modified (except by the non-standard Selection.modify)
  • the contents of the shadow DOM are completely inaccessible to the program.
    • Since the program has no specific knowledge of the document, it cannot reason about it in any reliable way.
    • Further -- even when the document/component provides an API for (some of) its contents -- the program has no way of knowing that it exists or how to use it.
    • A user may expect (or even direct) the program to interact with a closed shadow DOM child, since there is no perceptible difference from a normal element to them.
  • a program with access to the DOM but not the document's Javascript context cannot use any of the usually suggested techniques to circumvent this. Its only option is to proceed with broken or incomplete functionality.

Closed shadow DOMs may (or may not) be fine for document authors, who make the decision to use code including them. However, their existence and use makes the DOM an unreliable API for programs, libraries, etc. that are made to act upon a document by the user or document author.

If the DOM is supposed to be an effective API for developers other than document/component authors, then closed shadow DOMs are directly harmful (edit:) to all these developers. I propose depreciating mode and dropping closed shadow DOMs completely.

This issue generalises #354, all examples there also apply to here. (/cc @dylanb in case you're still interested).

@dylanb
Copy link

dylanb commented May 8, 2017

If the DOM is supposed to be an effective API for developers other than document/component authors, then closed shadow DOMs are directly harmful. I propose depreciating mode and dropping closed shadow DOMs completely.

I think this statement is too generic. Closed shadow DOM is VERY useful in masking the author from the unnecessary details of the internals of standard HTML elements but harmful otherwise.

@mrmr1993
Copy link
Author

mrmr1993 commented May 8, 2017

Closed shadow DOM is VERY useful in masking the author from the unnecessary details of the internals of standard HTML elements

I think the "standard" part here is important: how is any code not written by the document author supposed to deal with the myriad possible non-standard elements otherwise? The specs dictate that the standard elements should behave in the manner anyway, with or without a closed shadow DOM feature.

I'll concede an edit, regardless.

@rniwa
Copy link
Collaborator

rniwa commented May 9, 2017

A generic program (ie. a program that uses the DOM to interact with a document of which it has no prior knowledge) has no reliable interface with which it can interact with and/or monitor documents

What are examples of such a library / framework / program? The only concrete example people have presented so far is Google Feedback (it creates a screenshot of a Google property in JS). But any first-party feature like that could assume that everyone uses the same library or even a convention used on websites, e.g. shadowRoot is always exposed via _shadowRoot(), to implement such a feature. So the case in which components are written by the first party isn't the interesting case.

Now, let us consider the case where components are written by third party developers, or different teams within an organization with tens of thousands of developers. In this case, it's not trivial to write a code that walks across shadow trees of components written by different teams or third parties. However, this can also be a benefit. If one of those teams decide to change the internal DOM structures of a component, any JS code that relies on certain DOM structure in that component's shadow DOM would break.

In fact, this is a feature of Web Components API since the intent of Web Components API is to let developers write a custom HTML element that hides its internal structure in favor of providing a public API like any builtin HTML element. If it was so easy to travers and interact with each component's shadow tree, it would defeat the whole point of encapsulation. Instead, any generic code that walks across DOM to do some work should be treating each custom element instance like a regular builtin HTML element.

@rniwa
Copy link
Collaborator

rniwa commented May 9, 2017

I think the "standard" part here is important: how is any code not written by the document author supposed to deal with the myriad possible non-standard elements otherwise? The specs dictate that the standard elements should behave in the manner anyway, with or without a closed shadow DOM feature.

The same we'd deal with video elements, input elements, etc... which can be modeled as builtin elements with UA shadow roots (WebKit and Blink DO use shadow roots to implement these elements).

@mrmr1993
Copy link
Author

mrmr1993 commented May 9, 2017

What are examples of such a library / framework / program?

My classic examples are:

  • browser extensions (specifically WebExtensions)
    • Vimium for Chrome is a solid example that closed DOMs hurt
  • bookmarklets (ie. bookmarks with javascript: URIs)
  • in-browser testing frameworks
  • generic libraries (e.g. for jQuery)
  • programs using a spec.-compliant DOM API outside of a browser.
    • PhantomJS programs will suffer from this if/when it is updated to use a newer WebKit
    • Programs using a node.js library (eg. jsdom) which meets the spec. will struggle similarly
  • any other accessibility, testing or automation framework that uses the DOM as its interface to the document.

In this case, it's not trivial to write a code that walks across shadow trees of components written by different teams or third parties.

I thought this was the point of shadow DOM: to make it harder. If your (generic) program needs to support the increasing number of shadow DOM enabled documents, then it seems pretty manageable to me (unless I've missed anything):

  • use event.path[0] instead of event.target
  • recursively walk down shadow DOMs of activeElement from document, instead of document.activeElement
  • any DOM traversal just needs to check for and walk into a shadowRoot when element an element has one, falling back to normal walking through children
  • add a listener to blur/focus/etc. events on each shadowRoot in event.path (to capture changes of focus within a shadow DOM)

any JS code that relies on certain DOM structure in that component's shadow DOM would break.

I agree that this is undesirable. But any JS code doesn't rely on (or even know about) components and their closed shadow DOMs, but need access to them, are broken by their use in the first place. I don't think the extra (circumventable) safety warrants the (often uncircumventable) breakage for this code.

any generic code that walks across DOM to do some work should be treating each custom element instance like a regular builtin HTML element.

The same we'd deal with video elements, input elements, etc... which can be modeled as builtin elements with UA shadow roots (WebKit and Blink DO use shadow roots to implement these elements).

We know what to do with regular builtin HTML elements. The only thing we can do with custom elements we don't know about is ignore them. This is a failure of the DOM.

@dylanb
Copy link

dylanb commented May 9, 2017

@mrmr1993 I can definitely see how Vimium would need access to all shadow DOM - including standard elements. It would be easier for programs that don't require access to these internals to explicitly exclude them like axe-core is planning to do for the <marquee> element - which appears with an open shadow DOM in Chrome.

@rniwa
Copy link
Collaborator

rniwa commented May 10, 2017

  • browser extensions (specifically WebExtensions)
    • Vimium for Chrome is a solid example that closed DOMs hurt

What's exposed to browser extensions is up to each UA vendor. So each browser vendor can totally expose a new API that forces all shadow roots to be open, for example.

  • bookmarklets (ie. bookmarks with javascript: URIs)
  • in-browser testing frameworks

I could imagine that scripts injected by WebDriver should have access to closed shadow roots. That work could be spec'ed in WebDriver.

  • generic libraries (e.g. for jQuery)

Why would generic library need to access things inside a closed shadow tree?

  • programs using a spec.-compliant DOM API outside of a browser.
    • PhantomJS programs will suffer from this if/when it is updated to use a newer WebKit
    • Programs using a node.js library (eg. jsdom) which meets the spec. will struggle similarly

How could closed shadow trees cause problems for PhantomJS? Why are nodes.js and PhantomJS special with regards to closed shadow trees?

  • any other accessibility, testing or automation framework that uses the DOM as its interface to the document.

Saying that some libraries and frameworks may have an issue because they need to access nodes in a closed shadow tree is a bit tautological. We need a refutable statement. Otherwise, we can keep going around in circles based on each person's opinion.

@dylanb
Copy link

dylanb commented May 10, 2017

@rniwa how many examples do you want before you admit that your a priori opinion is possibly not correct?

What is a valid argument in your opinion?

Is the Vimium example not obviously a valid example of an application? What about that application makes you think it does not require access to the shadow DOM?

Here is another example, albeit forward-looking: the Web standardization process has accepted polyfilling future functionality as a way to prove out the validity/value and appropriateness of a new extension. It is impossible to polyfill anything that would need access to all shadow DOMs if they are closed. This cannot be circumvented with user agent permissions.

@matthewp
Copy link

matthewp commented May 10, 2017

I think he did address the Vimium example; browser extensions are non-standard, the browser can allow them to access closed Shadow DOM if they want to. It's a bug to report to Chrome's extension API.

Personally I do not use closed shadow roots because I just don't care if outside code accesses it, but I can see the argument for it just fine. Let me ask you this @dylanb, does it cause problems for you that you can't access built-in element's shadow DOM? If not, then why is it a problem for custom elements?

@dylanb
Copy link

dylanb commented May 10, 2017

Example of a polyfill which will not be possible:

Implementing a CSS4 :has() polyfill will not work in all the situations, in particular in combination with the >>> combinator.

@rniwa
Copy link
Collaborator

rniwa commented May 10, 2017

@rniwa how many examples do you want before you admit that your a priori opinion is possibly not correct?

The number of example doesn't matter as much as the content of examples.

What is a valid argument in your opinion?

There must be a use case that can't be addressed in the presence of any closed shadow trees, which outweighs all the benefits closed shadow trees provide. But the existence of such a use case seems very unlikely at this point because:

  1. We always have an option not to use shadow trees
  2. We've had 3-4 years of continuous discussion on this matter and nobody has come up with such an example.

But I welcome anyone to prove me wrong.

Is the Vimium example not obviously a valid example of an application? What about that application makes you think it does not require access to the shadow DOM?

No, because it's a browser extension. Each UA can expose whatever API to make closed shadow trees accessible to browser extensions. In fact, I've added a special API internal to Safari that enables such a feature should Safari team decide to add such a capability to its extension API. In general, browser extensions exist outside the realm of DOM and HTML standards; we don't encourage or discourage the existence of any feature or lack thereof in that space.

Here is another example, albeit forward-looking: the Web standardization process has accepted polyfilling future functionality as a way to prove out the validity/value and appropriateness of a new extension. It is impossible to polyfill anything that would need access to all shadow DOMs if they are closed. This cannot be circumvented with user agent permissions.

Example of a polyfill which will not be possible:

Implementing a CSS4 :has() polyfill will not work in all the situations, in particular in combination with the >>> combinator.

Why would a web component needs to be able to use the polyfill included outside the shadow tree? Again, if this is the first party or collaborating component, then such a component can use either a convention to expose its shadow tree (e.g. getShadowRoot() method) or call some function in the polyfill to enable it in the shadow tree.

If the component was written by a third party or is otherwise non-collaborating, then such a component is unlikely to be expecting the existence of such a polyfill. In fact, the presence of such a polyfill may break or have unintended consequences on the component depending on what the polyfill is doing. So I'd argue that the fact a polyfill doesn't spill into each component's shadow tree is a feature, not a bug, of shadow DOM.

@dylanb
Copy link

dylanb commented May 11, 2017

Are you saying that a polyfill for the CSS4 selectors is not a valid/useful thing to implement?

Are you saying that an author of a document who wants to use a CSS4 selector in combination with the >>> combinator is not a valid use case of a CSS4 selector?

@dylanb
Copy link

dylanb commented May 11, 2017

Here is a product that cannot work properly on a site that uses closed shadow DOM components that are not also standard HTML elements https://sitecues.com/

@rniwa
Copy link
Collaborator

rniwa commented May 11, 2017

Are you saying that a polyfill for the CSS4 selectors is not a valid/useful thing to implement?

That's not at all I'm saying. I'm pointing out that a polyfill included outside the shadow tree not spilling into a component's shadow tree is a feature, not a bug if the component was written by someone else. It avoids breaking that component even if it wasn't written to work well with the newly polyfill.

Here is a product that cannot work properly on a site that uses closed shadow DOM components that are not also standard HTML elements https://sitecues.com/

So it seems that Sitecues have two products. One for users which adds extra accessibility features to the browser, and another which adds those features on a website if the author wishes to have it.

The first product exists outside the realm of the standardization process since it's basically a browser plugin / extension. For the second product, either each component can expose extra information to the library using some API, website can annotate each component with whatever information this tool needs (this is probably the most preferable approach since we can add new builtin HTML elements, and there should be a mechanism to make it compatible with this tool), or each website can include a script synchronously at the top which overrides Element.prototype.attachShadow to provide some hooks.

But really, a well written component shouldn't need any extra work to make it possible to adjust the font size since each component should be specifying its font size using rem, em, or %.

Dictation is a bit hard to implement in a website without a component or a website cooperating for sure but then if a website contains an cross-origin iframe, it won't work across iframe boundaries so I'm not sure if adding dictation feature to a component written by a third party which contains semantically important content in its shadow tree is really a common use case.

In cases where a component is written by a third party, a tab view for example, the content semantically important to the user would be included as children of its shadow host so dictation would just work. In cases where a component is an integral part of the website, e.g. app container is a component itself, then that component can just expose whatever needed for dictation to work.

@dylanb
Copy link

dylanb commented May 12, 2017

ok, so we have two good examples then of valid applications that cannot be implemented with closed shadow DOM:

  1. Sitecues dictation
  2. polyfills

@rniwa
Copy link
Collaborator

rniwa commented May 12, 2017

I wouldn't say either example is "good".

The dictation can be implemented by either making websites or components collaborate. Furthermore, adding the dictation to a website that's not otherwise designed to do so is usually done at the browser level as a plugin / extension. If a website truly needed a dictation, e.g. e-book site, then that website can natively support & implement dictation as needed everywhere.

For polyfils, I don't think polyfills spilling into a component's shadow tree by default is a good idea at all because polyfills can break components that don't expect them to exist. These kind of isolations and opt-ins are precisely what shadow DOM API should provide.

@rniwa
Copy link
Collaborator

rniwa commented May 12, 2017

I'd note that we didn't spend a week or two thinking about these problems and come to the conclusion that the closed shadow tree is the way to go. We spent 2+ years continuously discussing amongst experienced software engineers who had been working on the Web technology over decades to reach our consensus. And this is precisely why we reached the consensus to keep mode as the mandatory option during W3C F2F meeting: because this topic is extremely contentious and either party arguing for open or closed shadow tree couldn't convince the other party.

It would take some serious thinking and groundbreaking argument or discovery on your part to convince us otherwise.

@rniwa
Copy link
Collaborator

rniwa commented May 12, 2017

Another thing. The canvas element might be hostile to Sitecues's dictation feature because any text drawn in the canvas can't be read by Sitecues unless it implements OCR of some sort. But we didn't remove the canvas element. We instead added enhancements to the canvas to make it more accessible so that tools like Sitecues and browser's native assertive technology could get necessary information out of a canvas element.

Similarly, if there is a specific scenario in which a closed shadow trees would break use cases, then we can make improvements to custom elements or shadow DOM API to address those use cases instead of letting them explore contents inside shadow trees manually. This approach has an added benefit of letting each component decide what's visible to tools like Sitecues.

@dylanb
Copy link

dylanb commented May 12, 2017

@rniwa I have a different proposal: how about user agents treat the built-in standard HTML elements in a special way (by closing their shadow DOM) and make all other shadow DOM open.

This would meet the original requirements of having closed shadow DOM for UA elements but solve all the problems that you are explaining away by (and I am paraphrasing your posts) essentially telling people either not to use shadow DOM, not to use closed shadow DOM (collaborate) or simply not to implement their application the way they would like to.

@mrmr1993
Copy link
Author

mrmr1993 commented May 12, 2017

What's exposed to browser extensions is up to each UA vendor. So each browser vendor can totally expose a new API that forces all shadow roots to be open, for example.

@rniwa If only there was a place to standardise such an API, or some action we could take to make the API unnecessary, to help reduce the burden for developers... It's also worth noting that none of the browsers (Chrome, Firefox, Edge) supporting Web Extensions have previously needed to expose any supplemental APIs for DOM access; the DOM has been generic, well-specified and powerful enough to read and modify the document in all the necessary ways.

We need a refutable statement.

I'll give you four(edit:) five:

  • With closed shadow roots, the DOM is no longer generic, well-specified and powerful enough to read and modify the document in the ways the page's scripts can as a whole.
  • It is a waste of implementers' time and effort to each discover the limited access and develop a non-standard workaround.
  • It is a waste of developers' time and effort to each discover the limited access, request a non-standard workaround and/or work out how it should be used.
  • The protection that the component developer is given from the document's author are illusory.
  • (edit:) Closed shadow DOMs let any developer create arbitrary self-contained, undocumented extensions to the DOM. The average developer cannot be relied upon to do this usefully, compatibly or well.

How could closed shadow trees cause problems for PhantomJS? Why are nodes.js and PhantomJS special with regards to closed shadow trees?

Scripts that use PhantomJS to interact with documents use the DOM to do so. Some developers will need to be able to read/manipulate content in shadow DOMs programmatically to make their applications work correctly with pages using them. They cannot do this without PhantomJS and node.js adding their own non-standard workarounds.

Why would generic library need to access things inside a closed shadow tree?

Suppose a library wanted to provide an easy way to attach keyboard shortcuts to a page, except when the user is entering text into an input (<input>, <textarea> <* contenteditable="true">, etc.). This library will function incorrectly (ie. interpret input-generating keypresses as keyboard shortcuts) for any pages containing an input inside a closed shadow DOM.

(Full disclosure: this is what Vimium does, but browser-wide instead of for a single document.)

@matthewp
Copy link

I disagree completely @mrmr1993, closed shadow DOMs just give developers the same abilities that built-in elements have. If a new built-in form element were created then Vimium wouldn't work with it either. The solution here is to define an interface to interact with form elements that is generic enough that Vimium and others don't need to special case for every element type. These are the events to listen to changes and the properties to get/set the element's underlying value. There's no reason why if my custom element (with a closed shadow) emitted those events and had the correct properties why Vimium wouldn't just work with those.

@mrmr1993
Copy link
Author

does it cause problems for you that you can't access built-in element's shadow DOM? If not, then why is it a problem for custom elements

@matthewp built-in elements' behaviours and properties are comprehensive, well-defined, and easy-to-find. There are occasional limitations (e.g. setting the cursor position in an <input type="date" />), but nearly everything you might want to do with them is possible.

By contrast, custom elements need not have any useful/relevant behaviours and properties. Even when they do, code that isn't written for them has no reliable way of using them, since it can't "read the documentation".

@mrmr1993
Copy link
Author

There's no reason why if my custom element (with a closed shadow) emitted those events and had the correct properties why Vimium wouldn't just work with those.

@matthewp Of course there isn't. But Vimium (or a library doing the same) is unreliable if it depends on every developer (using a shadow DOM) on every page correctly and consistently implementing the same features. This makes for awful UX.

@mrmr1993
Copy link
Author

To put it another way: open shadow DOMs give a (in fact the only) reliable standard API for non-standard components; closed shadow DOMs give you nothing.

@rniwa
Copy link
Collaborator

rniwa commented May 12, 2017

It's also worth noting that none of the browsers (Chrome, Firefox, Edge) supporting Web Extensions have previously needed to expose any supplemental APIs for DOM access; the DOM has been generic, well-specified and powerful enough to read and modify the document in all the necessary ways.

That's absolutely untrue. Browsers already have to provide a mechanism to inject code into every cross-origin iframe's document even though the top-level document ordinarily don't have access to.

@mrmr1993
Copy link
Author

the DOM has been generic, well-specified and powerful enough to read and modify the document in all the necessary ways.

That's absolutely untrue. Browsers already have to provide a mechanism to inject code into every cross-origin iframe's document even though the top-level document ordinarily don't have access to.

I think you're being a bit disingenuous. That's not an API to read or modify the document in any way. It does create an execution context with access to read or modify the document, but this is still solely via the DOM.

@rniwa
Copy link
Collaborator

rniwa commented May 12, 2017

We need a refutable statement.

I'll give you four(edit:) five:

  • With closed shadow roots, the DOM is no longer generic, well-specified and powerful enough to read and modify the document in the ways the page's scripts can as a whole.

This is demonstratively false. You can simply override Element.prototype.attachShadow to do the same. But more importantly, the whole point of shadow DOM is to provide encapsulation. Encapsulation by definition hides information of each component from the rest. This is a feature, not a bug.

  • It is a waste of implementers' time and effort to each discover the limited access and develop a non-standard workaround.

Why is not being able to access a shadow tree's content considered a limitation and something that requires a workaround? This isn't really a refutable but rather an opinion.

  • It is a waste of developers' time and effort to each discover the limited access, request a non-standard workaround and/or work out how it should be used.

Ditto.

  • The protection that the component developer is given from the document's author are illusory.

Illusory as in they can work around by overriding Element.prototype.attachShadow, or any other method in the same global? That is the limitation of the ECMAScript, not DOM. If ECMAScript provided a way to get a separate realm (and its own global object), or had a reliable way to get the original builtin functions, the protection would not be illusory.

  • (edit:) Closed shadow DOMs let any developer create arbitrary self-contained, undocumented extensions to the DOM. The average developer cannot be relied upon to do this usefully, compatibly or well.

This seems like an argument against any average developer writing any component at all. It doesn't matter whether a component uses a shadow tree or not since any component can be ill-formed. FWIW, a component can be written using a cross-origin iframe or a canvas element in which case nobody else wouldn't have access to its content even today.

@mrmr1993
Copy link
Author

It is a waste of implementers' time and effort to each discover the limited access and develop a non-standard workaround.

Why is not being able to access a shadow tree's content considered a limitation and something that requires a workaround? This isn't really a refutable but rather an opinion.

If useful programs can't be written to function as intended, that seems like a limitation. Especially if they could be before. Is it not a limitation that a developer can't write a keyboard shortcut library of the kind I described as above?

  • The protection that the component developer is given from the document's author are illusory.

Illusory as in they can work around by overriding Element.prototype.attachShadow, or any other method in the same global? That is the limitation of the ECMAScript, not DOM. If ECMAScript provided a way to get a separate realm (and its own global object), or had a reliable way to get the original builtin functions, the protection would not be illusory.

This is the whole point of the closed shadow DOM though, no? Either the DOM is important for multiple contexts, in which case they may not have document awareness and need shadow DOM access to behave correctly with custom elements, or they're not, in which case they're embedded in the document, use ECMAScript and the protection is illusory.

@mrmr1993
Copy link
Author

But it was discussed, for several years. And a lot of implementation details have been considered, there have always been opposition against /closed/ mode and their arguments were heard, but the decision have to be made and have been made. The argument is the other way around: we have the experience, it works, it works exactly as it should and it's pretty much always stressed to use encapsulation

This still doesn't mean that the right decision has been made, either in this spec. or by other language implementers. In general, experience with a bad decision is still experience with a bad decision. Unfortunately I'm not at all persuaded that this isn't one.

Of course the decision on /open///closed /should not be on the whim, even usage of shadow dom of any kind should not be, it should not be used just because we have it.

I only meant it shouldn't be used on a whim, only when it is useful and unavoidable.

This is not about changing, this is about getting the component and knowing immediately, it is about knowing, that it's pretty much always save to upgrade, knowing, that public is safe to mess around with, knowing that you can rely on that.

I know it's not about changing: I was hoping for an example of a tangible benefit, and thought that might be a good way to demonstrate one. Still, if you aren't using code that touches the shadow DOM of a component, I can't see why you wouldn't be safe to upgrade it regardless.


If the closed shadow DOM is supposed to give the document author these assurances, why is it that the component developer is the one getting the say over open vs. closed? The document author is at the receiving end of the safety vs. power trade-off, so surely they should have control over the decision.

@rniwa
Copy link
Collaborator

rniwa commented May 15, 2017

I think the same point as for C++ and Java applies. It's not a 'thing' as far is the user is concerned. Go wild.

Well, then that same argument holds for closed shadow trees.

Hardly. The DOM is always meaningful, since it represents the user interface/user-facing content in a standardised way. If you give me a DOM (and the relevant styles, admittedly), I can tell you what it represents and do useful things with it. Not so with arbitrary internal private states -- their representation is only meaningful in the context of that specific program.

That is patently false.

If the closed shadow DOM is supposed to give the document author these assurances, why is it that the component developer is the one getting the say over open vs. closed? The document author is at the receiving end of the safety vs. power trade-off, so surely they should have control over the decision.

Web components provide powers to both library/component authors as well as page authors. The primary benefits of closed shadow trees is the "guarantee" or a strong implication that the components are free to change its implementation without having to worry about its details. This is extremely important for a popular component intended to be upgraded and maintained for a long period of time.

We have quite a bit of experience dealing with such a problem in UIKit and AppKit here at Apple. We frequently encounter binary compatibility issues whereby which a third party app was relying on the implementation details of UIKit, AppKit, and other system frameworks. They are not supposed to do that (clearly stated in the policy and what not) but in the practice, many apps do because they can. In those cases, we must either can't make a change (e.g. maintaining same view / CA layer hierarchy), or break the app and notify the authors to have them fix them before those changes take effect. With AppStore and once a year release cycle, this is all possible. However, in the world of the Web, where developers would like to move faster and innovate in a rapid pace, and there is no repository of all Web apps which use a given framework or a library, this will become an impossible issue to resolve.

Just as another example, Microsoft once upon a time (not sure if they still do) decided to make it possible for any given Windows system to have multiple versions of the same DLL so that each app that links against a particular version of DLL will continue to work to combat the infamous DLL hell.

@rniwa
Copy link
Collaborator

rniwa commented May 15, 2017

I think the same point as for C++ and Java applies. It's not a 'thing' as far is the user is concerned. Go wild.

Well, then that same argument holds for closed shadow trees.

Hardly. The DOM is always meaningful, since it represents the user interface/user-facing content in a standardised way. If you give me a DOM (and the relevant styles, admittedly), I can tell you what it represents and do useful things with it. Not so with arbitrary internal private states -- their representation is only meaningful in the context of that specific program.

That isn't quite true. A li can be used to represent either an item in a shooping list, a file in the list of uploaded file, etc...
The semantics of each element and content depends heavily on the context in which it exists.

It's not really a thing to go into some component's shadow tree and mess with it.

I think you may have misinterpreted me: I meant that ECMAScript internal state isn't a virtual object in the mind of the user. DOM elements certainly are. Nonetheless, this is patently false; programs already do this. That you don't like it doesn't make it not a 'thing'.

Well, not really. In the presence of closed shadow trees, those scripts don't / can't do this.
Also, just because it's currently done, it's a good idea.

For example, we used to write code with goto and label everywhere before higher-level languages were invented.
Do you think a language which hides goto is harmful because it disallows certain use cases?

The thing that's made the web so powerful up until now is the DOM. Search engines rely on the DOM. Libraries rely on the DOM. Extensions rely on the DOM. Accessibility programs rely on the DOM. People have been able to re-use huge amounts of code because the underlying objects are all the same, and all accessible.

This all dies when your documents are full of opaque <main-content>s and <autocomplete-input>s and mine are full of <article-body>s and <filter-dropdown>s. Or worse, 'empty' <div>s. The objects in the DOM can't reliably convey any meaning to cross-document programs, unless they learn to read the developers' minds. This ruins the web.

We already live in that world. People are writing apps with a bunch of divs and spans,
and that's completely orthogonal to the concept of closed shadow trees.

I'll quote my example from above here, in the hopes that you have a response to it:

Suppose a library wanted to provide an easy way to attach keyboard shortcuts to a page, except when the user is entering text into an input (<input>, <textarea> <* contenteditable="true">, etc.). This library will function incorrectly (ie. interpret input-generating keypresses as keyboard shortcuts) for any pages containing an input inside a closed shadow DOM.

What happens when one of the components on a page already implemtns its own keyboard shortcuts?
With an open shadow tree or without a shadow tree at all, this library will go in and add new keyboard shortcuts.

This is why it's very important for each component to opt-in to whatever polyfill or library each page is importing. The idea of automatically enhancing an arbitrary component written by a third party quickly fails in practice.

@mrmr1993
Copy link
Author

mrmr1993 commented May 16, 2017

This is why it's very important for each component to opt-in to whatever polyfill or library each page is importing. The idea of automatically enhancing an arbitrary component written by a third party quickly fails in practice.

This seems to be the only point where we're not looping around the same arguments.

Why is it very important for each component to do this? Sometimes it will be what the document author intends and wants; other times not. Can we not find a compromise where the document author gets to decide which scripts can and can't access which shadow DOMs? Then, those authors have a clear idea of what might break if/when a component is updated, and what will not.

For example, would there be any issues with extending the script tag so that:

  • <script ... shadow-dom="true"> allows the script it includes (and any execution it spawns, such as evals, event listeners, etc.) to access shadow DOMs;
  • <script ... shadow-dom="false"> does not give the script any shadow DOM access (under the same conditions);
    • it should also likely restrict scripts it imports to shadow-dom="false"
  • possibly <script ... shadow-dom="request">, which eg. dispatches an event on the shadow host on .shadowRoot accesses that scripts with shadow-dom=true can catch (event.relatedTarget perhaps being the script element), denying access using event.preventDefault()?
    • it should also likely convert scripts it imports from shadow-dom="true" to shadow-dom="request"

Adding a with {shadowDom: "true" | "false" | "request"} to the spec. for ECMAScript 6 imports could easily provide the same for those. I also think that a default to false for all scripts would be reasonable in this case, for the reassurances it provides.

The major issue I have with closed shadow DOMs is that the decision is made piecemeal, by the developer of each component, and (necessarily) without the complete information of their interactions with each author's document. Giving this control to the author at the script level would seem to give the assurances and flexibility that we want between us.

@rniwa
Copy link
Collaborator

rniwa commented May 16, 2017

Why is it very important for each component to do this? Sometimes it will be what the document author intends and wants; other times not. Can we not find a compromise where the document author gets to decide which scripts can and can't access which shadow DOMs? Then, those authors have a clear idea of what might break if/when a component is updated, and what will not.

Because otherwise pages would start depending on the implementation details of components. It's possible, for example, for a graphing component to be implemented using SVG in one iteration and canvas in another. If page's code ever depended on the component using SVG, then the page would break when the component's version gets updated and it starts using canvas instead of SVG.

Ultimately, page authors have the power not to use a component which doesn't expose its shadow tree, or demand that they provide some API to mess with component's shadow tree. However, it's very important for the authors of each component to understand that doing so results in its shadow tree's structure being dependent by its users.

If the authors of a given component doesn't care about this, or is committed to maintain the backwards compatibility, then they can either not use shadow tree at all, or use an open mode shadow tree.

Either case, we don't believe we should give the unilateral power to users of components to decide whether it's okay to intrude into components' shadow tree or not.

@mrmr1993
Copy link
Author

Because otherwise pages would start depending on the implementation details of components.

I'm still not horrified by this. Sometimes this is useful and/or necessary.

It's possible, for example, for a graphing component to be implemented using SVG in one iteration and canvas in another. If page's code ever depended on the component using SVG, then the page would break when the component's version gets updated and it starts using canvas instead of SVG.

If a page breaks, is it not an issue for the page's author, rather than the component developer?

However, it's very important for the authors of each component to understand that doing so results in its shadow tree's structure being dependent by its users.

If a page depends on its internals, then they've taken that risk that updates will break things. It's their decision and their issue. I'm astounded that this position is controversial.

Ultimately, page authors have the power not to use a component which doesn't expose its shadow tree, or demand that they provide some API to mess with component's shadow tree.

So you would prefer page authors to beg component developers instead of making the decision themselves?

Either case, we don't believe we should give the unilateral power to users of components to decide whether it's okay to intrude into components' shadow tree or not.

Why can the component decide to opt them in or out of protections, but the page author themself cannot? The power dynamics of the current situation are really unusual, and don't seem to have any real advantages

@treshugart
Copy link

The power dynamics of the current situation are really unusual, and don't seem to have any real advantages

Especially when you can just:

const oldAttachShadow = HTMLElement.prototype.attachShadow;
HTMLElement.prototype.attachShadow = function () {
  return oldAttachShadow.call(this, { mode: 'open' });
}

@rniwa
Copy link
Collaborator

rniwa commented May 17, 2017

Because otherwise pages would start depending on the implementation details of components.

I'm still not horrified by this. Sometimes this is useful and/or necessary.

The problem is that we've already seen a similar problem with the shadow piercing combinator.

It's possible, for example, for a graphing component to be implemented using SVG in one iteration and canvas in another. If page's code ever depended on the component using SVG, then the page would break when the component's version gets updated and it starts using canvas instead of SVG.

If a page breaks, is it not an issue for the page's author, rather than the component developer?

The unfortunate reality is that library and framework authors would have to worry about those dependency. We deal with those issues all over the place in AppKit and UIKit or even WebKit / WebKit2 themselves embedded in other applications, and it imposes serious constrains on what we can and cannot change in our implementations.

And there is evidence after evidence that this is an issue throughout the industry as I've already enumerated a few.

However, it's very important for the authors of each component to understand that doing so results in its shadow tree's structure being dependent by its users.

If a page depends on its internals, then they've taken that risk that updates will break things. It's their decision and their issue. I'm astounded that this position is controversial.

The problem is that it quickly becomes library & framework author's problem once enough websites decide to take that risk.

Ultimately, page authors have the power not to use a component which doesn't expose its shadow tree, or demand that they provide some API to mess with component's shadow tree.

So you would prefer page authors to beg component developers instead of making the decision themselves?

Yes. They don't even have to beg. Don't use a badly written component, or a component you don't like. Nobody is forcing you to use a component that you don't like so much that you want to go in & modify its shadow tree.

Either case, we don't believe we should give the unilateral power to users of components to decide whether it's okay to intrude into components' shadow tree or not.

Why can the component decide to opt them in or out of protections, but the page author themself cannot? The power dynamics of the current situation are really unusual, and don't seem to have any real advantages

Because they're the one writing a component? If the license allows, page author can just copy / fork the component and modify its script not to use a shadow tree or make it use an open shadow tree; or better yet, directly make necessary changes desired on a particular website.

@mrmr1993
Copy link
Author

The problem is that we've already seen a similar problem with the shadow piercing combinator.

I can't necessarily see that these are problems. Allowing a document author to theme a component, or to use a library that does so, seem like completely reasonable use cases. That shadow DOMs prevent the unintentional downward leaking of styles is excellent, but it shouldn't mean that they should be completely unstylable from the outside.

(Also, the :part() selector seemed like a nice go-between for component theming: well-specified components can be styled in a non-implementation-dependent way, and under-specified components can still be styled by /deep/.)

If a page breaks, is it not an issue for the page's author, rather than the component developer?

The unfortunate reality is that library and framework authors would have to worry about those dependency. We deal with those issues all over the place in AppKit and UIKit or even WebKit / WebKit2 themselves embedded in other applications, and it imposes serious constrains on what we can and cannot change in our implementations.

I think the comparison with AppKit/UIKit and the WebKits is misleading. Since the AppKit/UIKit components are the lowest-level UI components (and the lowest-level, well-specified, reliable API to these components), the comparison should be with UA's native elements.

I'll happily agree with @dylanb's statement that UA elements should not expose their shadow DOMs, and by analogy that AppKit/UIKit should not expose their internals, since they are not described in a well-specified, reliably manipulated format. (WebKit is another beast, and an entire browser engine is far removed from a DOM, so any comparisons there are sketchy at best.)

We don't and can't expect to hold non-UA component developers to the same standards as UA component developers, so it's hard to justify giving them the same carte blanche.

The problem is that it quickly becomes library & framework author's problem once enough websites decide to take that risk.

Beyond styling, it doesn't seem likely that many websites will depend on specific implementation details of specific components -- it's quite a headache to retrofit extras into an existing custom element so they're more likely to just modify it directly. For e.g. analytics libraries, or any of my other examples, the code is component-agnostic, and so the implementation can change without breaking anything.

Don't use a badly written component, or a component you don't like. Nobody is forcing you to use a component that you don't like so much that you want to go in & modify its shadow tree.

This argument looks reasonable at first, but once you consider vendor lock-in, things start to change. If you want to add e.g. twitter integration to your document and their component is under-specified, your only option is put up and shut up.

page author can just copy / fork the component and modify its script not to use a shadow tree

This seems like an inconvenient and roundabout way of achieving the same behaviour as my proposal, except without the fine-grained control. If the developer wants to make changes to the component, then yes, they should fork and modify. If they plan to use its DOM for any other purpose, having to modify the component for mere access seems unduly heavy-handed.

@dylanb
Copy link

dylanb commented May 18, 2017

@matthewp

I disagree completely @mrmr1993, closed shadow DOMs just give developers the same abilities that built-in elements have. If a new built-in form element were created then Vimium wouldn't work with it either. The solution here is to define an interface to interact with form elements that is generic enough that Vimium and others don't need to special case for every element type. These are the events to listen to changes and the properties to get/set the element's underlying value. There's no reason why if my custom element (with a closed shadow) emitted those events and had the correct properties why Vimium wouldn't just work with those.

There are a bunch of conditions that hold for the standard HTML elements that do not hold for custom elements

  1. Their APIs are very rich and standardized. The ways that other parts of the standard can interact with them is well thought through and relatively complete. They provide open, consistent and standardized APIs that allow interaction with them

  2. They can (for the most part) be relied upon to support accessibility APIs and keyboard interaction

  3. You can implement polyfills and other interactions purely in JavaScript without any limitations

All of these things mean that there are very few (I do not exclude the possibility, but cannot think of any) situations that have not already been identified (and are being worked on) where a document author would legitimately like to look inside and reach inside the shadow DOM.

We cannot predict the ways the shadow DOM spec will be used by developers (just like we could not predict how HTML would be used by developers) and therefore, we cannot claim that the same will even remotely be possible with custom components that use shadow DOM.

@rniwa
Copy link
Collaborator

rniwa commented May 18, 2017

I think the comparison with AppKit/UIKit and the WebKits is misleading. Since the AppKit/UIKit components are the lowest-level UI components (and the lowest-level, well-specified, reliable API to these components), the comparison should be with UA's native elements.

We can agree to disagree on this point, and say this is a difference in our opinions.

I'll happily agree with @dylanb's statement that UA elements should not expose their shadow DOMs, and by analogy that AppKit/UIKit should not expose their internals, since they are not described in a well-specified, reliably manipulated format. (WebKit is another beast, and an entire browser engine is far removed from a DOM, so any comparisons there are sketchy at best.)

We don't and can't expect to hold non-UA component developers to the same standards as UA component developers, so it's hard to justify giving them the same carte blanche.

Why not?

Don't use a badly written component, or a component you don't like. Nobody is forcing you to use a component that you don't like so much that you want to go in & modify its shadow tree.

This argument looks reasonable at first, but once you consider vendor lock-in, things start to change. If you want to add e.g. twitter integration to your document and their component is under-specified, your only option is put up and shut up.

We already live in that world. If some social network only provides their social engagement widget in HTTP, for example, there is nothing you can do to make them use HTTPS instead. Also, social networks have incentives to make their widgets not customizable for branding purposes. Allowing random websites to allow changing the color of the buttons, etc... would be detrimental to their branding & marketing effort for example. Again, this should really be left to each social network site to decide. If one doesn't like the default UI provided by a social network, once could either not put their widget or use a hyperlink with a custom UI, etc...

page author can just copy / fork the component and modify its script not to use a shadow tree

This seems like an inconvenient and roundabout way of achieving the same behaviour as my proposal, except without the fine-grained control. If the developer wants to make changes to the component, then yes, they should fork and modify. If they plan to use its DOM for any other purpose, having to modify the component for mere access seems unduly heavy-handed.

This whole argument hinges on the existence of such a convincing use case in which developers try to access and mutate shadow tree, yet nobody has given one so the premise of the statement has not been well established.

@dylanb
Copy link

dylanb commented May 19, 2017

Here is a very useful polyfill that will not work for custom components with closed shadow DOM https://github.com/WICG/inert/blob/master/src/inert.js

@rniwa
Copy link
Collaborator

rniwa commented May 19, 2017

Here is a very useful polyfill that will not work for custom components with closed shadow DOM https://github.com/WICG/inert/blob/master/src/inert.js

For the hundredth time, I don't think polyfill should automatically work inside a shadow tree. If a component wasn't written with that polyfill in mind, what's the point of making it work in its shadow tree? It won't be using the polyfilled feature anyway. If it was written with the assumption that some feature is available in all browsers but it isn't, then that component should itself be including that polyfill.

I don't think it's productive for you to keep posting a polyfill as an example. We would never be convinced that it's a compelling use case for not having closed shadow trees.

@mrmr1993
Copy link
Author

mrmr1993 commented May 19, 2017

I think the comparison with AppKit/UIKit and the WebKits is misleading. Since the AppKit/UIKit components are the lowest-level UI components (and the lowest-level, well-specified, reliable API to these components), the comparison should be with UA's native elements.

We can agree to disagree on this point, and say this is a difference in our opinions.

Since this is your basis for disagreeing with the proposal, I struggle to let it go. Could you clarify how the AppKit or UIKit components are comparable to custom elements with (edit:) closed shadow roots in ways they are not comparable to UA native elements?

We don't and can't expect to hold non-UA component developers to the same standards as UA component developers, so it's hard to justify giving them the same carte blanche.

Why not?

In theory? There are a long tail of developers where we might expect consistency and competence to degrade. In practise? Experience.

We already live in that world. If some social network only provides their social engagement widget in HTTP, for example, there is nothing you can do to make them use HTTPS instead.

Sure. That's bad. I'm even less inclined to surrender them complete control over parts of the DOM on this basis.

Also, social networks have incentives to make their widgets not customizable for branding purposes. Allowing random websites to allow changing the color of the buttons, etc... would be detrimental to their branding & marketing effort for example.

Web standards are not the place to enforce trademarks, nor licensing arrangements or other branding concerns. These also don't preclude under-specification in other aspects.

Again, this should really be left to each social network site to decide. If one doesn't like the default UI provided by a social network, once could either not put their widget or use a hyperlink with a custom UI, etc...

Only if a sufficient hyperlink exists. Otherwise, I think 'put up and shut up' is a fair appraisal.

This whole argument hinges on the existence of such a convincing use case in which developers try to access and mutate shadow tree, yet nobody has given one so the premise of the statement has not been well established.

In my opinion, several were given in the original issue.

Nonetheless: a great many websites monitor the actions a user takes, in order to calculate what made their interactions successful/failures. If these analytics record in insufficient detail -- or even fail to record -- actions in shadow DOMs, then document authors are measurably damaged by shadow DOMs, and have a convincing use case for inspecting both them and the user's interactions with them.

@rniwa
Copy link
Collaborator

rniwa commented May 19, 2017

I think the comparison with AppKit/UIKit and the WebKits is misleading. Since the AppKit/UIKit components are the lowest-level UI components (and the lowest-level, well-specified, reliable API to these components), the comparison should be with UA's native elements.

We can agree to disagree on this point, and say this is a difference in our opinions.

Since this is your basis for disagreeing with the proposal, I struggle to let it go. Could you clarify how the AppKit or UIKit components are comparable to custom elements with closed shadow roots in ways they are not comparable to UA native elements?

Because we see web components something akin to views in UIKit and windows in Win32 API. But this is just one perspective and heavily subjective matter so I don't think keep discussing this matter is useful. At the end of the day, we don't really need to convince one or other; we just need to move the standards forward based on rough consensus.

Also, social networks have incentives to make their widgets not customizable for branding purposes. Allowing random websites to allow changing the color of the buttons, etc... would be detrimental to their branding & marketing effort for example.

Web standards are not the place to enforce trademarks, nor licensing arrangements or other branding concerns. These also don't preclude under-specification in other aspects.

Right, but we also have to take different parties' incentives into mind. Any feature is not useful if parties that are supposed to enforce the policy doesn't have an incentive to do so.

This whole argument hinges on the existence of such a convincing use case in which developers try to access and mutate shadow tree, yet nobody has given one so the premise of the statement has not been well established.

In my opinion, several were given in the original issue.

Again, we can agree to disagree on this. Since if we had accepted your use cases or you had accepted that there were no good use cases, then we wouldn't be having this conversation in the first place.

Nonetheless: a great many websites monitor the actions a user takes, in order to calculate what made their interactions successful/failures. If these analytics record in insufficient detail -- or even fail to record -- actions in shadow DOMs, then document authors are measurably damaged by shadow DOMs, and have a convincing use case for inspecting both them and the user's interactions with them.

Why? Regardless of the mode, mouse and keyboard events that got dispatched inside a shadow tree will propagate outside the shadow tree unless they're stopped to propagate.

By the way, that's a much better use case than polyfills or anything else presented thus far if it were true that you couldn't observe mouse and keyboard events that happen within a shadow tree. But that isn't. We specifically made mouse, keyboard, and other kinds of events that stem from user interaction composed so that web pages can observe them.

@mrmr1993
Copy link
Author

Because we see web components something akin to views in UIKit and windows in Win32 API. But this is just one perspective and heavily subjective matter so I don't think keep discussing this matter is useful.

More than anything, I'm trying to understand. I can't see how the DOM is deficient (if not better) as compared to its UIKit and Win32 analogues under my proposal.

At the end of the day, we don't really need to convince one or other; we just need to move the standards forward based on rough consensus.

Great! Hopefully we can move the standards forward towards my proposal. I'll start making efforts to build rough consensus on this.

we also have to take different parties' incentives into mind. Any feature is not useful if parties that are supposed to enforce the policy doesn't have an incentive to do so.

I'm accustomed to "useful" meaning that things may be used to achieve something productive that wasn't possible/as easy without it.

Perhaps you are proposing that we should remove the <video> element, since "parties that are supposed to enforce the policy [of video copyright] doesn't have an incentive to do so";
the <img> element, since "parties that are supposed to enforce the policy [of image copyright] doesn't have an incentive to do so";
and DOM text APIs, since "parties that are supposed to enforce the policy [of literary copyright] doesn't have an incentive to do so".
I am not an advocate of this position.

Why? Regardless of the mode, mouse and keyboard events that got dispatched inside a shadow tree will propagate outside the shadow tree unless they're stopped to propagate.

If the component has any kind of complex state and/or user-specific variation, it can become extremely hard to reverse engineer what these meant originally. This seems like a waste, when the information could have all been there to start with.

I'll also mentioned nested scrolling elements in passing, to highlight how non-trivial this might be for stateless, user-agnostic components.

By the way, that's a much better use case than polyfills or anything else presented thus far ...

I still maintain that a page should be able to provide keyboard shortcuts without munging user input into components' inputs.

@rniwa
Copy link
Collaborator

rniwa commented May 19, 2017

Perhaps you are proposing that we should remove the <video> element, since "parties that are supposed to enforce the policy [of video copyright] doesn't have an incentive to do so";
the <img> element, since "parties that are supposed to enforce the policy [of image copyright] doesn't have an incentive to do so";
and DOM text APIs, since "parties that are supposed to enforce the policy [of literary copyright] doesn't have an incentive to do so".
I am not an advocate of this position.

That's not all all what I'm saying. The point you were making is that with closed shadow trees, it's hard to modify components made by social networks. I'm saying that given they have incentives not to let this happen, they can already make this hard e.g. by using a cross origin iframe. They can also add a licensing terms which prohibits you from messing with their UI, etc... So your argument that closed shadow tree makes it harder for third party page authors to customize those social widgets isn't convincing given that social networks can already circumvent this, and even in the presence of shadow trees, page authors can still use a hyperlink or simply override Element.prototype.attachShadow before those social widgets are loaded as the last resort.

Why? Regardless of the mode, mouse and keyboard events that got dispatched inside a shadow tree will propagate outside the shadow tree unless they're stopped to propagate.

If the component has any kind of complex state and/or user-specific variation, it can become extremely hard to reverse engineer what these meant originally. This seems like a waste, when the information could have all been there to start with.

That's precisely what encapsulation should hide. Those semantics should be communicated via ARIA attributes, slotted contents, public methods, etc... of custom elements.

By the way, that's a much better use case than polyfills or anything else presented thus far ...

I still maintain that a page should be able to provide keyboard shortcuts without munging user input into components' inputs.

And I would disagree to that position.

@dylanb
Copy link

dylanb commented May 19, 2017

@rniwa

For the hundredth time, I don't think polyfill should automatically work inside a shadow tree. If a component wasn't written with that polyfill in mind, what's the point of making it work in its shadow tree? It won't be using the polyfilled feature anyway. If it was written with the assumption that some feature is available in all browsers but it isn't, then that component should itself be including that polyfill.

I don't think it's productive for you to keep posting a polyfill as an example. We would never be convinced that it's a compelling use case for not having closed shadow trees.

Well I disagree. Have you asked the writers and users of that polyfill what they think? Their opinion matters much more than yours does.

@robdodson @alice What do you think about closed shadow DOM making your polyfill impossible to implement for custom controls?

@rniwa
Copy link
Collaborator

rniwa commented May 19, 2017

Well I disagree. Have you asked the writers and users of that polyfill what they think? Their opinion matters much more than yours does.

Well, we can keep agree to disagree on this. What matters most, to us, is the opinions of component authors and page authors. The opinions of polyfill authors has a lot less weight compared to those two to us.

@andyearnshaw
Copy link

@dylanb @mrmr1993 I'm a webcomponent author (and sometimes a polyfill author) and the biggest advantage of using a closed shadow tree, for me, is knowing that I can make internal changes to my component without compromising its integrity in the wild.

Let's say, for instance, I release my amazing component, which is self hosted on my domain at path /components/Amazing/1/Amazing.js. Any web page author can use it by referencing this script. Now, let's say that there's an edge case bug which can be easily fixed by changing the internal DOM in my component in some way. If my tree is closed and I want to make this change, I can do so transparently by updating the script at its current URL. If it's open and I want to make this change, I have to concern myself with how this may affect sites using the component and possibly make a separate, opt-in release, which will leave instances of the buggy version out in the wild on 3rd-party websites.

This is not me "telling kids to get off my lawn". This is me having peace of mind and not painting myself into a corner. This is where you can draw a parallel with UA-defined components. By keeping their internal tree hidden, they don't need to worry about breaking things when they change that internal tree. You both indicated that a UA should be allowed this privilege, but a component author should not, which I find extremely puzzling!

@BronislavKlucka
Copy link

BronislavKlucka commented May 22, 2017

Well I disagree. Have you asked the writers and users of that polyfill what they think? Their opinion matters much more than yours does.

the point here is, he's hardly alone...

This question have already been solved and it's again and again showing up by people, who think, they should have full control over 3rd party component. This is what it boils down to, the control... usecases (polyfills, keyboard shortcuts) are just examples of that. The point is it's always the author, who should have full control. By law and by tradition. If he/she chooses to release under public licence and /open/ mode, it's his/her call, if he chooses to release it under "do not touch the internals" and /closed/ mode, it's again his/her call. Page author must respect this choice, the choice he/she has is whether to use it, or not. The simple fact, that someone wants to poke inside component actually means nothing, regardless of how "good" reason they thing they have.
I am not saying, that license and open/closed are the same (licence can allow source code changes), I'm pointing out, that the choice, of both, is component's author in the first place, not anyone else.

there is a lot of "I do not agree here", the fact is, that while the need for /closed/ was agreed by majority, the need for total DOM control was not. This was never search for 100% agreement. There were always arguments about control/visibility. The component developers choice may mean very little to you, but it means very much for majority. The need of private states outweighs the fact, that some code will not work as originally intended.

@trusktr
Copy link
Contributor

trusktr commented Jul 22, 2017

Would a way to access the composed tree be helpful?

@dylanb
Copy link

dylanb commented Jul 22, 2017

@trusktr I asked for this years ago...it would solve some of the issues raised

@dead-claudia
Copy link

dead-claudia commented Oct 12, 2017

Edit: Just pretend this doesn't exist.

Poof!
May I add a couple legitimate use cases for hard-private, inaccessible DOM: \*
  1. Guarding an <input type="username">/<input type="password"> from successful script injection and/or malicious extensions.
  2. Hiding sensitive information like exposed financial data or one-time tokens from successful script injection and/or untrusted extensions.

Currently, all it takes is a single successful XSS attack to literally read a user's password as they type it, and a payload doesn't have to be large.** The only way to stop it at this stage is via CSP policy, and some form of MITM mitigation like TLS (assuming the attacker doesn't catch the initial connection).

Of course, browsers could (and still) should offer APIs for password managers, screen readers, etc. to read such inaccessible text, but it should be guarded by a permission instead of just being wide open to all.


* I'm by no means a security expert. I'm just super security conscious, and I like to be more proactive in my opsec. (No, I don't like writing exploits, and that's part of why I immediately deleted and did not post the payload referenced below.)

** I created within 30 minutes just now an easily modifiable, rudimentary payload for pages with single <form>s that's <300 bytes raw, <350 bytes encoded with encodeURIComponent, which can send the submitted password to a C&C server on submit without any way to stop it. The fact I'm just a security-conscious web developer who almost never writes exploits for anything outside of tests should tell you how easy it is to do once you're in.

@mrmr1993
Copy link
Author

May I add a couple legitimate use cases for hard-private, inaccessible DOM

@isiahmeadows please do

Guarding an <input type="username">/<input type="password"> from successful script injection and/or malicious extensions.

This is not something the shadow DOM was created for, and it doesn't give any guarantees of security. To quote from an earlier email on the topic:

Stay after class and write 100 times on the board: "Type 2 is not a security boundary".

Currently, all it takes is a single successful XSS attack to literally read a user's password as they type it[...] The only way to stop it at this stage is via CSP policy, and some form of MITM mitigation like TLS

This is also a problem with (session) cookies, etc. A properly set CSP is the way to make these guarantees. If you are MITM-ed, game over anyway.

Shadow DOM may help in this way, but it's not the intention of the design, and I don't think it's a good argument for closed shadow DOMs.

@dead-claudia
Copy link

@mrmr1993

This is not something the shadow DOM was created for, and it doesn't give any guarantees of security.
[quote...]

Would using a local, unexposed weakmap mitigate that?

This is also a problem with (session) cookies, etc. A properly set CSP is the way to make these guarantees. If you are MITM-ed, game over anyway.

Very true, especially if they catch you establishing the initial connection (as ISPs have been known to do). The only way you can really hope to side-step it at all is through dedicated protocols like Signal.

Shadow DOM may help in this way, but it's not the intention of the design, and I don't think it's a good argument for closed shadow DOMs.

Agreed, and it's like trying to apply duck tape to a broken structural support. Maybe 1% of the time, it actually works, but the other 99% of the time, you've got bigger problems than just that one break.

@mrmr1993
Copy link
Author

Would using a local, unexposed weakmap mitigate that?

@isiahmeadows Using it for what? My instinct is no: closed shadow DOM creation can be circumvented (depending on order of script execution), and it is easy to leak the shadow root, intentionally or not. Obviously, if your code is careful and you can ensure that no XSS script can run before you get your reference to Element.prototype.attachShadow, then the (IMO harmful) behaviour of closed shadow roots should be as specified.

I'll leave the rest of your comment; I don't think this is the place to discuss network vulnerabilities/mitigations.

@annevk
Copy link
Collaborator

annevk commented Feb 18, 2018

OP is basically a non-starter; closed shadow trees were a necessary part of getting shadow trees at all. It's not realistic to expect them to go away at this point; see the long mailing list threads on the topic. I'm therefore going to close this issue.

@annevk annevk closed this as completed Feb 18, 2018
@mrmr1993
Copy link
Author

Pity, they still have the potential to ruin browser extensions for Chrome/Firefox/Edge. Since Firefox is planning to land in the next development version and Edge has it in the works, we should start seeing usage in sites and the corresponding breakage.

I've opened a FF issue here, which will hopefully influence the WebExtensions spec too, but I've possibly left it too late for a fix before it lands. I haven't done anything for WebDriver, or for Chrome/Edge extensions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests