feat(runtime): support declarative shadow DOM

[christian-bromann]

What is the current behavior?

This patch introduces support for declarative shadow DOM in Stencil 🎉

GitHub Issue Number: #4010

What is the new behavior?

This patch includes:

• New Feature: new new flags to the renderToString method:
  • fullDocument: allows to define whether to return a full document or just the component
  • serializeShadowRoot: defines whether a component marked with shadow: true is being rendered in a declarative shadow DOM:

    <my-cmp>
      <template shadowrootmode="open">
        <style>...</style>
        <div>Hello World</div>
        <slot></slot>
      </template>
    
      Slotted content
    </my-cmp>

    or as scoped component:

    <my-cmp>
      <div>Hello World</div>
      Slotted content
    </my-cmp>

• New Feature: the hydrate module now exports a streamToString function that returns a Readable object that can be passed into a server response.
• Potentially Breaking: users doing DOM snapshot tests of shadow components will need to update their snapshots as all shadow components are now serialized in a Declarative Shadow DOM
• Runtime changes as suggested in feat(runtime): support already-existing shadow roots #5787
• ESM version of the hydrate script
• Removal of isPromise helper in the code
• Support for objects in component properties, e.g. renderToString(<car-detail car=${JSON.stringify({ year: 1234 })}>)
• Removal of duplicated logic within renderToString and hyrdrateDocument
• Rename of serializeToHtml to streamToHtml which uses generators
• some changes to how we call setTimeout or setInterval in mock doc in order to enable support for using the hydrate script in the browser

Documentation

ToDo missing

Does this introduce a breaking change?

• Yes
• No

Testing

Added e2e tests for using the hydration script in the browser as well as part of the e2e test suite.

Other information

n/a

[github-actions]

@stencil/core@4.18.3 ts
tsc --noEmit --project scripts/tsconfig.json && tsx scripts/tech-debt-burndown-report.ts

--strictNullChecks error report

Typechecking with --strictNullChecks resulted in 1067 errors on this branch.

That's 13 fewer than on main! 🎉🎉🎉

reports and statistics

Our most error-prone files

Path	Error Count
src/dev-server/index.ts	37
src/dev-server/server-process.ts	32
src/compiler/prerender/prerender-main.ts	22
src/runtime/vdom/vdom-render.ts	22
src/runtime/client-hydrate.ts	20
src/runtime/vdom/test/patch.spec.ts	19
src/runtime/vdom/test/util.spec.ts	19
src/screenshot/connector-base.ts	19
src/testing/puppeteer/puppeteer-element.ts	19
src/dev-server/request-handler.ts	15
src/compiler/prerender/prerender-optimize.ts	14
src/compiler/sys/stencil-sys.ts	14
src/runtime/connected-callback.ts	14
src/sys/node/node-sys.ts	14
src/compiler/prerender/prerender-queue.ts	13
src/compiler/sys/in-memory-fs.ts	13
src/runtime/set-value.ts	13
src/compiler/output-targets/output-www.ts	12
src/compiler/transformers/test/parse-vdom.spec.ts	12
src/compiler/transformers/transform-utils.ts	12

Our most common errors

Typescript Error Code	Count
TS2322	336
TS2345	322
TS18048	185
TS18047	99
TS2722	27
TS2532	23
TS2531	19
TS2790	11
TS2454	10
TS2352	9
TS2769	8
TS2416	7
TS2538	4
TS2493	3
TS18046	2
TS2684	1
TS2430	1

Unused exports report

There are 15 unused exports on this PR. That's the same number of errors on main, so at least we're not creating new ones!

Unused exports

File	Line	Identifier
src/runtime/bootstrap-lazy.ts	21	setNonce
src/screenshot/screenshot-fs.ts	18	readScreenshotData
src/testing/testing-utils.ts	198	withSilentWarn
src/utils/index.ts	145	CUSTOM
src/utils/index.ts	245	NODE_TYPES
src/utils/index.ts	269	normalize
src/utils/index.ts	7	escapeRegExpSpecialCharacters
src/compiler/app-core/app-data.ts	25	BUILD
src/compiler/app-core/app-data.ts	116	Env
src/compiler/app-core/app-data.ts	118	NAMESPACE
src/compiler/fs-watch/fs-watch-rebuild.ts	123	updateCacheFromRebuild
src/compiler/types/validate-primary-package-output-target.ts	82	satisfies
src/compiler/types/validate-primary-package-output-target.ts	82	Record
src/testing/puppeteer/puppeteer-declarations.ts	485	WaitForEventOptions
src/compiler/sys/fetch/write-fetch-success.ts	7	writeFetchSuccessSync

[github-actions]

PR built and packed!

Download the tarball here: https://github.com/ionic-team/stencil/actions/runs/9569595832/artifacts/1613920781

If your browser saves files to ~/Downloads you can install it like so:

unzip -d ~/Downloads ~/Downloads/stencil-core-4.18.3-dev.1718732065.7afd94d.tgz.zip && npm install ~/Downloads/stencil-core-4.18.3-dev.1718732065.7afd94d.tgz

[tanner-reits]

Ran into one issue when testing this out. I setup a test app that used the hydrate output from a Stencil starter project. Only change I made to the starter was adding a slot element in the render function return value for my-component. Then, if I added the following snippet to my test server, it would result in 3 instances of the component getting rendered:

const res = await renderToString(
    '<my-component>Jimmy</my-component><my-component first="Bob"></my-component>'
  );
  response.writeHead(200);
  response.write(res.html);
  response.end();

The first two elements that get rendered are correct, but the third is an instance without any slot content or input property values. Same thing happens when using streamToString. I checked with a published version of Stencil and didn't see this behavior, so something we might wanna look into. I'm sure it's an issue client-side when the runtime takes over since the server wasn't spitting out any additional elements in the strings it returned.

Also, code looks good, but would be good to cut down the number of new SNC violations this introduces :)

[christian-bromann]

@tanner-reits thanks for reviewing!

I am struggling reproducing the issue you are describing. I've set-up a new Stencil project with the following component:

import { Component, Prop, h } from '@stencil/core';
import { format } from '../../utils/utils';

@Component({
  tag: 'my-component',
  styleUrl: 'my-component.css',
  shadow: true,
})
export class MyComponent {
  /**
   * The first name
   */
  @Prop() first: string;

  /**
   * The middle name
   */
  @Prop() middle: string;

  /**
   * The last name
   */
  @Prop() last: string;

  private getText(): string {
    return format(this.first, this.middle, this.last);
  }

  render() {
    return <div>
      Hello, World! I'm {this.getText()}
      <slot></slot>
    </div>;
  }
}

When creating an hydration script and calling this script:

const { renderToString } = require('./hydrate');

(async () => {
  console.log(await renderToString('<my-component>Jimmy</my-component><my-component first="Bob"></my-component>', {
    prettyHtml: true,,
    serializeShadowRoot: true
  }));
})()

I correctly get the following output:

<!doctype html>
<html class="hydrated" data-stencil-build="u56d8u2z">
  <head>
    <meta charset="utf-8">
  </head>
  <body>
    <my-component class="hydrated sc-my-component-h" s-id="1">
      <template shadowrootmode="open">
        <style sty-id="sc-my-component">
          /*!@:host*/.sc-my-component-h{display:block}
        </style>
        <div c-id="1.0.0.0" class="sc-my-component">
          <!--t.1.1.1.0-->
          Hello, World! I'm
          <slot c-id="1.2.1.1" class="sc-my-component"></slot>
        </div>
      </template>
      <!--r.1-->
      Jimmy
    </my-component>
    <my-component class="hydrated sc-my-component-h" first="Bob" s-id="2">
      <template shadowrootmode="open">
        <style sty-id="sc-my-component">
          /*!@:host*/.sc-my-component-h{display:block}
        </style>
        <div c-id="2.0.0.0" class="sc-my-component">
          <!--t.2.1.1.0-->
          Hello, World! I'm Bob
          <slot c-id="2.2.1.1" class="sc-my-component"></slot>
        </div>
      </template>
      <!--r.2-->
    </my-component>
  </body>
</html>

Which just shows two elements. Mind sharing your example?

[tanner-reits]

@christian-bromann The renderToString method is generating the correct output, but it doesn't render correctly when handed over to the browser

[tanner-reits]

LGTM! renderToString and streamToString are both emitting the expected HTML and the browser is rendering correctly!

[mayerraphael]

@christian-bromann I tried your PR build in a more advanced use case (nested components, multiple named slots) and with serializeShadowRoot in renderToString we get duplicated renderings (only for some components), which do not occur without.

With serializeShadowRoot: true

With serializeShadowRoot: false

The components that have duplicated rendering, some elements inside are missing c-id or html comments:

Tested with 4.18.3-dev.1718220868.532557e

[christian-bromann]

@mayerraphael thanks so much for providing feedback 🙏 is there any chance you can create a minimal reproducible example? I will try to reproduce this myself but haven't come across this behavior.

[mayerraphael]

@christian-bromann I invited you to my repository.

Its the same repo as before. Disable javascript in the browser to see the correct rendering (but with missing c-ids in the my-other-component), as soon as Stencil hydrates in the browser, those elements get rendered double.

With disabled JS:

After hydration:

I hope the example helps :)

[christian-bromann]

@mayerraphael thanks for reporting, I proposed a fix and made a dev release (4.18.3-dev.1719344120.e32bcd8) which resolves the issue.

[mayerraphael]

@christian-bromann Thanks, that was fast.

Fixes some duplicate rendering, but not all. Also got some missing s-ids on top level with multiple components. Will try to replicate in my repository again.

Edit:

I updated my repository again. I couldn't replicate the missing s-ids. But i got another bug with s-ids beeing set on child-components.

From our demo page we still get some duplicated rendering and missing c-id/s-ids:

[thure]

@christian-bromann I was working from babe807, which correctly hydrated my components, however none of the releases on NPM will hydrate the components, they instead seem not to recognize them.

In my use-case:

  const { html } = await renderToString(
  '<ch-oklch-picker lightness="0.43" chroma="0.4" hue="256"></ch-oklch-picker>'
  {
    serializeShadowRoot: true,
    fullDocument: false,
  });

should return what babe807 returned:

<ch-oklch-picker lightness="0.43" chroma="0.4" hue="256" role="group" class="hydrated" s-id="1"><!--r.1--><label id="hue-0329" c-id="1.0.0.0"><!--t.1.1.1.0-->Hue (0–360)</label><input property="hue" aria-labelledby="hue-0329" type="number" min="0" max="360" step="1" value="256" c-id="1.2.0.1"><input property="hue" aria-labelledby="hue-0329" type="range" min="0" max="360" step="1" value="256" c-id="1.3.0.2"><label id="chroma-66e5" c-id="1.4.0.3"><!--t.1.5.1.0-->Chroma (0–0.4)</label><input property="chroma" aria-labelledby="chroma-66e5" type="number" min="0.000" max="0.400" step="0.004" value="0.4" c-id="1.6.0.4"><input property="chroma" aria-labelledby="chroma-66e5" type="range" min="0.000" max="0.400" step="0.004" value="0.4" c-id="1.7.0.5"><label id="lightness-4fd3" c-id="1.8.0.6"><!--t.1.9.1.0-->Lightness (0–1)</label><input property="lightness" aria-labelledby="lightness-4fd3" type="number" min="0.00" max="1.00" step="0.01" value="0.43" c-id="1.10.0.7"><input property="lightness" aria-labelledby="lightness-4fd3" type="range" min="0.00" max="1.00" step="0.01" value="0.43" c-id="1.11.0.8"></ch-oklch-picker>

However 4.19.0 and any of the prereleases just return the input string.

Did the API change, or is there something I should be doing to configure the hydrate app properly?

My prototype Astro integration which uses the hydrate app is here: https://github.com/ch-ui-dev/ch-ui/blob/thure/feat-astro/packages/astro-stencil/src/server.ts

[thure]

I’ve posted a comparison PR for the hydrate outputs of babe807 and 4.19.0 here:
https://github.com/thure/hydrate-diff/pull/1/files?diff=split&w=1

Could one of these differences account for the component not hydrating?

[christian-bromann]

Did the API change, or is there something I should be doing to configure the hydrate app properly?

No, we've build it to be backward compatible. Let me take a look.

[christian-bromann]

@thure it seems like setting up your project and the branch and running this script:

import { renderToString } from './packages/elements-hydrate-temp/index.mjs'

const { html } = await renderToString(
  '<ch-oklch-picker lightness="0.43" chroma="0.4" hue="256"></ch-oklch-picker>',
  {
    serializeShadowRoot: true,
    fullDocument: false,
  }
);

console.log(html);

gives me above mentioned hydrated string. Can you provide some concrete steps I can walk through to properly reproduce what you see?

[christian-bromann]

I updated my repository again. I couldn't replicate the missing s-ids. But i got another bug with s-ids beeing set on child-components.

@mayerraphael thanks again for your feedback, I check out your repository, updated Stencil to v4.19.0 which we released yesterday and ran server.js which returned the following HTML code:

<my-component last-page="5" class="sc-my-component-h hydrated" s-id="1">
    <template shadowrootmode="open">
      <style sty-id="sc-my-component">
        /*!@:host*/
        .sc-my-component-h {
          display: block
        }
      </style>
      <div class="sc-my-component" c-id="1.0.0.0">
        <div class="pagination sc-my-component" c-id="1.1.1.0">
          <div class="pagination-pages pagination-notation sc-my-component" c-id="1.2.2.0">
            <my-other-component class="sc-my-component sc-my-other-component-h hydrated" c-id="1.3.3.0" s-id="2">
              <template shadowrootmode="open">
                <style sty-id="sc-my-other-component">
                  /*!@:host*/
                  .sc-my-other-component-h {
                    display: block
                  }
                </style>
                <div class="pagination-item sc-my-other-component" c-id="2.0.0.0">
                  <!--t.2.1.1.0-->0
                </div>
              </template>
              <!--r.2-->
            </my-other-component>
            <my-other-component class="sc-my-component sc-my-other-component-h hydrated" c-id="1.4.3.1" s-id="3">
              <template shadowrootmode="open">
                <style sty-id="sc-my-other-component">
                  /*!@:host*/
                  .sc-my-other-component-h {
                    display: block
                  }
                </style>
                <div class="pagination-item sc-my-other-component" c-id="3.0.0.0">
                  <!--t.3.1.1.0-->1
                </div>
              </template>
              <!--r.3-->
            </my-other-component>
            <my-other-component class="sc-my-component sc-my-other-component-h hydrated" c-id="1.5.3.2" s-id="4">
              <template shadowrootmode="open">
                <style sty-id="sc-my-other-component">
                  /*!@:host*/
                  .sc-my-other-component-h {
                    display: block
                  }
                </style>
                <div class="pagination-item sc-my-other-component" c-id="4.0.0.0">
                  <!--t.4.1.1.0-->2
                </div>
              </template>
              <!--r.4-->
            </my-other-component>
            <my-other-component class="sc-my-component sc-my-other-component-h hydrated" c-id="1.6.3.3" s-id="5">
              <template shadowrootmode="open">
                <style sty-id="sc-my-other-component">
                  /*!@:host*/
                  .sc-my-other-component-h {
                    display: block
                  }
                </style>
                <div class="pagination-item sc-my-other-component" c-id="5.0.0.0">
                  <!--t.5.1.1.0-->3
                </div>
              </template>
              <!--r.5-->
            </my-other-component>
            <my-other-component class="sc-my-component sc-my-other-component-h hydrated" c-id="1.7.3.4" s-id="6">
              <template shadowrootmode="open">
                <style sty-id="sc-my-other-component">
                  /*!@:host*/
                  .sc-my-other-component-h {
                    display: block
                  }
                </style>
                <div class="pagination-item sc-my-other-component" c-id="6.0.0.0">
                  <!--t.6.1.1.0-->4
                </div>
              </template>
              <!--r.6-->
            </my-other-component>
          </div>
        </div>
      </div>
    </template>
    <!--r.1-->
  </my-component>
  <div>
    <my-other-component label="2" class="sc-my-other-component-h hydrated" s-id="7">
      <template shadowrootmode="open">
        <style sty-id="sc-my-other-component">
          /*!@:host*/
          .sc-my-other-component-h {
            display: block
          }
        </style>
        <div class="pagination-item sc-my-other-component" c-id="7.0.0.0">
          <!--t.7.1.1.0-->2
        </div>
      </template>
      <!--r.7-->
    </my-other-component>
  </div>
  <my-component last-page="2" class="sc-my-component-h hydrated" s-id="8">
    <template shadowrootmode="open">
      <style sty-id="sc-my-component">
        /*!@:host*/
        .sc-my-component-h {
          display: block
        }
      </style>
      <div class="sc-my-component" c-id="8.0.0.0">
        <div class="pagination sc-my-component" c-id="8.1.1.0">
          <div class="pagination-pages pagination-notation sc-my-component" c-id="8.2.2.0">
            <my-other-component class="sc-my-component sc-my-other-component-h hydrated" c-id="8.3.3.0" s-id="9">
              <template shadowrootmode="open">
                <style sty-id="sc-my-other-component">
                  /*!@:host*/
                  .sc-my-other-component-h {
                    display: block
                  }
                </style>
                <div class="pagination-item sc-my-other-component" c-id="9.0.0.0">
                  <!--t.9.1.1.0-->0
                </div>
              </template>
              <!--r.9-->
            </my-other-component>
            <my-other-component class="sc-my-component sc-my-other-component-h hydrated" c-id="8.4.3.1" s-id="10">
              <template shadowrootmode="open">
                <style sty-id="sc-my-other-component">
                  /*!@:host*/
                  .sc-my-other-component-h {
                    display: block
                  }
                </style>
                <div class="pagination-item sc-my-other-component" c-id="10.0.0.0">
                  <!--t.10.1.1.0-->1
                </div>
              </template>
              <!--r.10-->
            </my-other-component>
          </div>
        </div>
      </div>
    </template>
    <!--r.8-->
  </my-component>
  <script type="module">
    import {
      defineCustomElements
    } from "./static/loader/index.js";
    defineCustomElements().catch(console.error);
  </script>

I can see all c-ids properly assigned. I can't find any of the patternlib components though so you might look at a different example. Can you get something reproducible for me? Thank you!

[thure]

@christian-bromann Yes, for sure:

1. git clone git@github.com:ch-ui-dev/ch-ui.git
2. cd ch-ui
3. git checkout thure/debug-stencil – this branch is one commit behind thure/feat-astro where I’d committed the hydrate app produced by babe807 into its own package, which might be why you couldn’t repro the issue
4. pnpm install
5. pnpm nx build docs
6. Observe that any HTML files in the build e.g. apps/docs/dist/index.html have a non-hydrated <ch-oklch-picker

If you then git checkout thure/feat-astro, which has the pinned hydrate app, then repeat steps 5 and 6, the build will have the hydrated component.

[thure]

I should note that ch-oklch-picker is a shadow: false component — did 4.19.0 drop support for hydrating light DOM components?

[christian-bromann]

did 4.19.0 drop support for hydrating light DOM components?

No!

then repeat steps 5 and 6, the build will have the hydrated component.

Unfortunately I am getting this error:

> nx run icons:build-esm

✘ [ERROR] No loader is configured for ".node" files: node_modules/.pnpm/@resvg+resvg-js-darwin-arm64@2.6.2/node_modules/@resvg/resvg-js-darwin-arm64/resvgjs.darwin-arm64.node

    node_modules/.pnpm/@resvg+resvg-js@2.6.2/node_modules/@resvg/resvg-js/js-binding.js:1:2588:
      1 │ ...eBinding=require("@resvg/resvg-js-darwin-arm64")}catch(e){loadEr...
        ╵                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[christian-bromann]

@thure I think I was able to reproduce this using internal test infrastructure.

[christian-bromann]

@thure false alarm 🙈 it does seem with our current tests that scoped components hydrate just fine. Mind providing more minimalistic example of what you experience?

[thure]

@christian-bromann I’ve set up a Codesandbox with the hydrate apps produced by both babe807 and 4.19.0 — these are the same files as the diff I provided earlier.

If you have the ch-ui repo locally and want to build the hydrate app yourself, you could cd packages/elements and run pnpm link $pathToYourLocalStencil, then switch to different versions in your local stencil project (rebuild stencil when doing so) and run pnpm nx build elements in the root of ch-ui to see the different outputs. The elements package doesn’t depend on the icons package so you shouldn’t encounter the issue you encountered trying to build the docs.

I can isolate @ch-ui/elements if that would help, will just need some time to do so, let me know.

[mayerraphael]

I updated my repository again. I couldn't replicate the missing s-ids. But i got another bug with s-ids beeing set on child-components.

@mayerraphael thanks again for your feedback, I check out your repository, updated Stencil to v4.19.0 which we released yesterday and ran server.js which returned the following HTML code:

...

</script> ```

I can see all c-ids properly assigned. I can't find any of the patternlib components though so you might look at a different example. Can you get something reproducible for me? Thank you!

Sorry those patternlib components are some internal ones.

In the debugger it looks like addHostEventListeners crashes, which is inside the hydrateComponent function call.

Exception has occurred: TypeError: Cannot read properties of undefined (reading 'addEventListener')
  at Object.ael
    at hydrate\index.js:1712:11
    at Array.map (<anonymous>)
    at addHostEventListeners (\hydrate\index.js:1708:15)
    at hydrateComponent (\hydrate\index.js:2071:7)
    at connectElement2 (\hydrate\index.js:2030:18)

I am not sure why this is used inside hydrateComponent, but it is always undefined. Components that have a @Listen() crash.

@christian-bromann I updated my example with an @Listen(), this replicated the error:

The my-whatever-component crashes and does not render/hydrate:

That was a tough one :)

Repo: https://github.com/mayerraphael/stencil-dsd-ssr-playground

Edit:

This also affects the "old" serializeShadowRoot: false.

[mayerraphael]

@christian-bromann

I issued individual tickets as i found more problems with the new Version:

#5869
#5870

[christian-bromann]

@mayerraphael thanks a lot! I will take a look.