Storybook: Use experimental docgen server

[aduth]

What?

Updates Storybook to use the new React meta docgen service through its new experimentalDocgenServer option in Storybook 10.5.0.

Note: The pull request in its current shape is heavily AI-coded and I haven't fully reviewed the code changes. It's unlikely that this would land in its current shape, but is useful to understand for context to the chain of pull requests in the stack (#80692, #80655) that it influences.

Why?

The new docgen service is meant to be more performant and accurate, and it's something we already use for the components manifest, so enabling it everywhere also brings consistency between the manifest and live Storybook site.

How?

• Replaces experimentalReactComponentMeta with experimentalDocgenServer.
• Removes react-docgen-typescript, including its settings and related dependencies. The new docgen service is meant to replace react-docgen-typescript.

Unfortunately, this update brings some issues which either block this or will need to be worked through.

• @ignore does not currently seem to be respected, causing ignored props to be shown on the live Storybook site.
• The shape of the manifest has changed yet again, and the new "v": 1 is not compatible in affected clients like the design system MCP server.

The latter is a pretty substantial issue, and what we'll likely want to do is extract the pieces here for the "dual client" to make it so that the published @wordpress/design-system-mcp is capable of handling the new manifest shape before we make that the default in the live Storybook site. We can remove the dual handling later once clients are updated.

This is also a reminder of why it's hazardous to try to leverage experimental features, and this is something we "signed up for" by choosing to do so. It may also be prudent to wait for the manifest shape to fully stabilize before committing to this upgrade.

Testing Instructions

TBD

Use of AI Tools

Used Cursor IDE + Auto (likely Composer) model for the sequence of pull requests in this stack of pull requests, reviewed and iterated manually prompted by myself.

[aduth]

Likely next steps here:

1. Extract the changes around dependency on the path ➡️ package handling in design-system-mcp that will become incompatible with the new docgen server. This can happen today on trunk and will make this future work simpler.
2. Then, the rest of the "dual client" approach can also be migrated to land on trunk, independent of pull requests in the chain. The idea is that the client would be able to support both Storybook 10.4 (legacy-enhanced) manifests produced under experimentalReactComponentMeta, as well as Storybook 10.5 $ref-referenced manifests produced under experimentalDocgenServer.