Query nodes within shadow roots

[ebidel]

Currently, we don't have a way to find/query nodes with shadow roots. Users need to traverse .shadowRoots themselves.

It would be ideal if the DTP supported an option to make this easier, but maybe we can provide options to page.$eval to pierce through shadow roots?

[pemrouz]

Related: WICG/webcomponents#78

[elf-pavlik]

workaround for WebdriverIO mentioned in that webcomponents issue: https://gist.github.com/ChadKillingsworth/d4cb3d30b9d7fbc3fd0af93c2a133a53

this comment mentions benchmarking >>> in Blink compared to JS based recursive traversals

[ebidel]

>>> is deprecated: https://www.chromestatus.com/feature/6750456638341120

[aslushnikov]

I like a pierce option for the page.$ / page.$$:

const handle = await page.$('div', {pierce: true});
await handle.click();
await handle.dispose();

This would satisfy all the use cases I can think of while narrowing the scope of the shadowdom-related api to a single method. Implementation-wise, it's fine to implement this in-page.

I'd be happy to review a pull request.

[pemrouz]

>>> is deprecated: https://www.chromestatus.com/feature/6750456638341120

It's deprecated in CSS, not JS. Details in the thread.

[ebidel]

I'm aware of that thread and its history :) Because no vendor has agreed on a solution, chrome is removing the feature from JS too: https://www.chromestatus.com/feature/4964279606312960

…

On Tue, Oct 3, 2017, 3:11 AM Pedram Emrouznejad ***@***.***> wrote: >>> is deprecated: https://www.chromestatus.com/feature/6750456638341120 It's deprecated in CSS, not JS. Details in the thread. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#858 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAOigKyeNKJR0TBE2kWBnH2mGYntAY6Wks5sojKVgaJpZM4PhYg4> .

[Jamesernator]

It's probably a separate issue but it'd be nice if to use { pierce: true } we could also use $ off an element handle directly e.g.:

const componentHandle = await page.$('#nextPageButton')
const textHandle = await componentHandle.$('#buttonText', { pierce: true })

const textContent = await page.evaluate(
    textElement => textElement.textContent, 
    textHandle,
}

assert(textContent.includes('continue'), "Expected text button to include text")

Obviously this isn't tied to { pierce: true } but it's certainly a case I'd expect to be using such a feature commonly.

EDIT: Looks like this was added at some point so thanks to whoever added that.

[AliMD]

@ebidel Thank you for making this issue

We really need this feature, { pierce: true } not work for me, I can do something like this
myApp.root.querySelector('page-signin').root.querySelector('#myElement');

But I need to many methods like waitForSelector
I can't handle load and visible waiting, any temporary solution?

[ebidel]

Not other than drilling into the trees as you're doing. I've labeled this a "good first issue" if someone wants to submit a PR to add {pierce: true} options to page.$ and page.$$.

[AliMD]

@ebidel
We make something like this but need some suggestion/advice for add {pierce: true} option.
Thank you in advance.

await page.evaluate(() => {
  window.queryDeepSelector = (selectorStr, container = document) => {
    const selectorArr = selectorStr.replace(new RegExp('//', 'g'), '%split%//%split%').split('%split%');

    for (const index in selectorArr) {
      const selector = selectorArr[index].trim();

      if (!selector) continue;

      if (selector === '//') {
        container = container.root;
      }
      else {
        container = container.querySelector(selector);
      }
      if (!container) break;
    }

    _log('queryDeepSelector: %s', selectorStr, container);
    return container;
  };
});

and waitForDeepSelector

page.waitForDeepSelector = (selector, container) => {
  return page.waitForFunction(
    (selector, container) => queryDeepSelector(selector, container),
    { polling: 'raf' },
    selector, container
  );
};

How to use

await page.waitForDeepSelector('lava-app // lava-menu // iron-selector a[name="product"]');

await page.evaluate(() => {
  queryDeepSelector('lava-app // lava-menu // a[name="product"]').click();
});

[ebidel]

On closer look, I don't think it's trivial to replicate what shadow dom piercing in querySelector gave us. page.$/$$ both take a css selector. CSS selectors read right to left. You'd need to start with the node, get its ancestor flattened tree, then do filtering/checks as you walk up the tree to figure out if the element matches the selector.

The DTP protocol doesn't support and easy way to achieve this.

[notwaldorf]

@ebidel What if we don't do it as a flattened tree, but just one level deep? So something like

page.$evalDeep(parentSelector, shadowSelector, ...)

did

page.querySelector(parentSelector).shadowRoot.querySelector(shadowSelector)

for the selector part and applied that?

Is this chaos? It might be chaos. I can't think of how to generalize that to multiple shadow roots deep tho :(

[ebidel]

Could do that but it probably wouldn't be worth adding new API just for 1 level deep.

It's easy to create a getShadowRootChild or similar that drills 1-N levels deep into shadowRoots. Example one level deep:

const elHandle = await page.$(parentSelector);
const getShadowRootChildText = (el, childSelector) => {
  return el.shadowRoot.querySelector(childSelector).textContent;
});
const result = await page.evaluate(getShadowRootChildText, elHandle, shadowSelector);

[a-xin]

@notwaldorf I'm using something like this to query multiple specified levels:

const shadowSelectorFn = (el, selector) => el.shadowRoot.querySelector(selector);

const queryDeep = async (page, ...selectors) => {
  if (!selectors || selectors.length === 0) {
    return;
  }

  const [ firstSelector, ...restSelectors ] = selectors;
  let parentElement = await page.$(firstSelector);
  for (const selector of restSelectors) {
    parentElement = await page.evaluateHandle(shadowSelectorFn, parentElement, selector);
  }

  return parentElement;
};

You'd use it as follows, where all the given selectors reference the element whose shadowRoot should be queried next, except for the last one, which just references the actual element to return:

const email = await queryDeep(
  page,
  'my-app', '.main my-auth', 'my-auth-login', 'input[type="email"]'
);

@ebidel Would it make sense to have something like this in the API?

[ebidel]

@a-xin That would work, but it would require the user already be familiar with theshadowRoot structure of app/components. So I'm not sure how practical it would actually be to use in the real world.

[elf-pavlik]

@a-xin approach seams similar to @ChadKillingsworth approach in this custom webdriver.io command https://gist.github.com/ChadKillingsworth/d4cb3d30b9d7fbc3fd0af93c2a133a53 and based on it npm module by @MORLACK https://www.npmjs.com/package/wdio-webcomponents

I don't know if we could really avoid user having to know the shadowRoot structure of app/components since avoiding it might go against the encapsulation and possibility for the same selector getting matched in different shadow roots if all get traversed. Maybe using some kind of wildcard */** as selector could still provide opt in for traversing everything?

While particular API may differ between puppeteer, webdriver.io etc. I think all those tools face similar challenges with respect to querying nodes within shadow roots. Sharing notes and ideas across the teams could maybe help with coming up with good solutions.

[ChadKillingsworth]

Apple has a proposal to solve this. I'm not sure where the discussion around it currently exists: WICG/webcomponents#78

[elf-pavlik]

Thanks @ChadKillingsworth, the last comment in that issue seems to mention dequelabs/axe-core#317 by @dylanb (bit confusing since posted from @WebAppsWG handle).

[dylanb]

We solved this problem ourselves but I did not get any sympathy from the WG for my "invalid requirements". You probably know this but Apple is <redacted> and seems to look on testing with disdain. Quite understandable seeing as their browser is all but meaningless when it comes to Web development but it does end up negatively impacting us.

Its not a simple problem to solve if you want a solution that works in all possible scenarios (like shadow DOM within shadow DOM within shadow DOM...) and you also want to support "unique selectors" (which a testing product needs to be able to do). The shadow piercing combinator ('>>>') could not do this.

[elf-pavlik]

@dylanb do you think that your solution could work as npm module which puppeteer, webdriver.io etc. could potentially re-use?

[dylanb]

@elf-pavlik Our solution has been crafted to work well in our specific scenario. I don't think it would work well for puppeteer because it selects items based on the flattened tree and requires a cache of the flattened tree to work. This would probably perform very badly in puppeteer because unlike the accessibility checker axe-core, you cannot assume that the DOM will not change between selectors (in fact it most likely will change), so you would constantly have to update the cache - which you don't need anyway.

I could certainly use my knowledge of the problem domain to help.

One thing we will have to do is define a "selector syntax" for piercing shadow DOM.

One option for a selector syntax would be to override the DOM piercing combinator '>>>'. Alternatives include '>>' or '<<<'

I would define the behavior such that the right hand selector component applies to the shadow root of the results of the left hand selector component.

A selector of 'my-component:nth-of-type(2) >>> #thing-i-want-to-target' would, for example, find (all the) element(s) with the id of thing-i-want-to-target within the specific instance of a custom element <my-component>.

This is different from @aslushnikov example in that it would not pierce with a simple selector 'div' but would pierce with the selector '* >>> div' and would only pierce one level. To pierce multiple levels, you would have to specify the number of levels you want. E.g. '* >>> * >>> .thing'.

The advantage of this is you can very specifically target elements. The downside is you have to know how many levels down your target is.

We could also possibly implement two different selectors, one that behaves as described above and one that allows you to select a specific shadow host and then pierce down all its levels. Perhaps '>>' and '>>>' respectively.

thoughts?

[Kikobeats]

I saw a lot of comments related to a query for specific selectors.

A more generic solution could be being possible make shadow components available.

Just for inspiration, rendertron loads the polyfill for achieve this goal:

https://github.com/GoogleChrome/rendertron/blob/720710b98b764445d321f314934e312343095a04/src/renderer.js#L74

or this inside on combination with puppeteer logic for wait an event?

https://github.com/webcomponents/webcomponentsjs#webcomponents-loaderjs

[Georgegriff]

Put this together which lets you query shadow dom elements without knowing the full path to a node. Might not be very performant but seems to work in dev tools and in my tests so far:
https://www.npmjs.com/package/query-selector-shadow-dom

[aslushnikov]

TL;DR: try "Copy JS Path" in Chrome Canary to get a "js selector" for nodes inside Shadow DOM.

Hi everybody!

In Chrome 72 (current Canary) we introduced a new option - "Copy JS Path", located right next to the "Copy Selector" option:

This produces a javascript one-liner that fetches the selected element. For some element "#foo" inside shadow DOM the one-liner might look like this:

document.querySelector('#container').shadowRoot.querySelector('#foo')

In puppeteer, one-liner can be used to fetch ElementHandles like this:

// Fetch element using the "js selector"
const buttonHandle = await page.evaluateHandle(`document.querySelector('#container').shadowRoot.querySelector('#foo')`);
// Click element
await buttonHandle.click();

We think this should help in many cases when dealing with Shadow DOM. Please give it a shot
and let us know what you think!

-Puppeteer Team

[AndreasGalster]

I was trying your suggested solution to .focus() an input element (in paper-input) within shadow DOM and then do page.type('xyz') without success. Any suggestion how to achieve this?

[aslushnikov]

@AndreasGalster the following works quite fine for me:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({
    headless: false
  });
  const page = await browser.newPage();
  await page.goto('https://npm-demos.appspot.com/@polymer/paper-input@3.0.1/demo/index.html', {waitUntil: 'networkidle2'});
  const input = await page.evaluateHandle(`document.querySelector('body > div > demo-snippet:nth-child(2) > paper-input:nth-child(2)').shadowRoot.querySelector('#input-2 > input')`);
  await input.focus();
  await input.type('.foo', 'woof woof!');
//  await browser.close();
})();

[aslushnikov]

Closing this since #858 (comment) addresses the issue.

[felixfbecker]

It's unfortunate that all the utilities like page.click() or page.select() become useless when using shadow roots. As using shadow roots is becoming more popular, maybe Puppeteer could consider first-class support in these utilities? For example, wherever selector: string is accepted, accept selector: string | string[] where an array is a series of selectors where the next selector is queried on the shadow root of the result of the previous selector.

await page.select(['#container', 'select#foo'], 'value');

[aslushnikov]

@felixfbecker this doesn't simplify the life though - getting this "array" of selectors is quite cumbersome.

What I like about "Cope JSpath" is that it makes it easy to generate selectors for shadow DOM.
I'd rather support jsPaths first-hand in Puppeteer API somehow.

await page.click(`document.querySelector('body > toolbar-component > toolbar-section.left')`);

Filed #4171 to discuss this further.

[aroduribe]

@AndreasGalster the following works quite fine for me:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({
    headless: false
  });
  const page = await browser.newPage();
  await page.goto('https://npm-demos.appspot.com/@polymer/paper-input@3.0.1/demo/index.html', {waitUntil: 'networkidle2'});
  const input = await page.evaluateHandle(`document.querySelector('body > div > demo-snippet:nth-child(2) > paper-input:nth-child(2)').shadowRoot.querySelector('#input-2 > input')`);
  await input.focus();
  await input.type('.foo', 'woof woof!');
//  await browser.close();
})();

@aslushnikov How are you calling the type method on a JSHandle object?

[josenobile]

@aslushnikov what about page.waitForSelector() ? Any workaround? ::part is not accepted in querySelector

[josenobile]

Also .waitForXPath('//*[@id="theid"]') doesn't work

[aslushnikov]

@aslushnikov what about page.waitForSelector() ? Any workaround? ::part is not accepted in querySelector

@josenobile you can use page.waitForFunction with jsPath:

const handle = (await page.waitForFunction(() => document.querySelector('body > toolbar-component`))).asElement();

[VersLaFlamme]

@aslushnikov what about page.waitForSelector() ? Any workaround? ::part is not accepted in querySelector

@josenobile you can use page.waitForFunction with jsPath:

const handle = (await page.waitForFunction(() => document.querySelector('body > toolbar-component`))).asElement();

thanks a lot, it works like a clock!

[serhiiarkhypiuk]

Did anybody find a working solution on how to interact with shadow-root elements?

I am able to 'click' on the element using the following code

async function selectShadowElement(elementSelector) {
    try {
        const container = document.querySelector('parentSelector')
        return container.shadowRoot.querySelector(elementSelector)
    } catch (err) {
        return null
    }
}

const element = 'elementSelector'
const result = await page.waitForFunction(selectShadowElement, {timeout: 15 * 1000}, element)

if (!result) {
  console.error('Shadow element was not found!')
  return
}

await(await page.evaluateHandle(selectShadowElement, element)).click()

but did not able to get the value from an attribute