Migrate to Yarn workspaces #146

adamziel · 2023-03-08T16:41:20Z

Description

This PR splits the repository into five node packages that are also yarn workspaces:

php-cli
php-wasm
typescript-reference-doc-generator
wordpress-playground
wordpress-plugin-ide

This effectively undoes Use folders instead of separate npm packages, so you may ask "why bother?"

Rationale

With this PR:

The build process is straightforward. There's no gulp and there's no mountain of custom logic in esbuild.js. Files are moved around the repo using a simple copy plugin or an import statement. There's no custom cache busting logic – it's handled by the build tools instead
php-wasm is now reusable and can be published in npm. This means it can be easily reused outside of WordPress Playground.
Each package has its own build process that caters to its specific needs. For example, php-cli outputs a node.js script. Web workers can now be bundled as iife (which is the only way to use them in firefox) while other assets are es modules.
There are less tasks in general. For example, TypeScript types are handled by rollup in a few packages.

At the same time, none of the original downsides of using packages are present:

There's no global build pipeline. Every package outputs a complete build artifact that can be directly imported in case of libs, or used in the browser in case of the Playground website.
Yarn workspaces ensure the tasks are executed in the correct order
tsconfig.json and package.json can mostly be copied, there's no api-documenter.json at all.

adamziel · 2023-03-08T16:41:52Z

This PR is mostly there. I just need to reapply the last few trunk commits.

adamziel · 2023-03-10T14:21:44Z

I discovered one downside of using this approach with Vite: Setting up a dev mode is challenging. Every package has a separate build config with its own resolve.alias, external, and viteStaticCopy setup.

Vite doesn't seem to support merging multiple configurations. The monorepo dependencies must be either:

Built before the dev script can be started, ideally in a watch mode
Aliased inside the Playground's vite.config.ts and configured again like in their original vite.config.js

The former is slow and inconvenient, and the latter wouldn't work with wordpress-plugin-ide that requires two separate vite configs.

Either way for now I'll ship yarn run dev as mostly working best-effort solution, and yarn run build; yarn run dev as the preferred one.

Perhaps Turborepo or Turbopack would help here?

adamziel · 2023-03-10T15:36:47Z

Follow-up issue: #148

This PR splits the repository into five node packages that are also yarn workspaces: - php-cli - php-wasm - typescript-reference-doc-generator - wordpress-playground - wordpress-plugin-ide This effectively undoes [Use folders instead of separate npm packages](#70), so you may ask "why bother?" With this PR: * The build process is straightforward. There's no `gulp` and there's no mountain of custom logic in `esbuild.js`. Files are moved around the repo using a simple `copy` plugin or an `import` statement. There's no custom cache busting logic – it's handled by the build tools instead * `php-wasm` is now reusable and can be published in npm. This means it can be easily reused outside of WordPress Playground. * Each package has its own build process that caters to its specific needs. For example, `php-cli` outputs a node.js script. Web workers can now be bundled as `iife` (which is the only way to use them in firefox) while other assets are es modules. * There are less tasks in general. For example, TypeScript types are handled by rollup in a few packages. At the same time, none of the [original downsides of using packages ](#70) are present: * There's no global build pipeline. Every package outputs a complete build artifact that can be directly imported in case of libs, or used in the browser in case of the Playground website. * Yarn workspaces ensure the tasks are executed in the correct order * tsconfig.json and package.json can mostly be copied, there's no `api-documenter.json` at all.

adamziel · 2023-03-10T16:06:55Z

I merged this one manually due to a high amount of conflicts: a7601f0

## Description Sets up the correct build pipeline for all parts of Playground and PHP.wasm. This enables a public release of the [Playground API](#149) npm package! I've been [struggling](#146) with [this](#70) for [a while](#150) and couldn't understand what's so hard. NX made it apparent – look at this dependency graph! <img width="1291" alt="CleanShot 2023-03-16 at 23 16 26@2x" src="https://user-images.githubusercontent.com/205419/225764795-7fa8e972-09f8-41ef-aac2-1c96bd100ea0.png"> No wonder it's been almost impossible to set everything up by hand! ## Usage Start with `yarn install` ### Shortcuts To start a copy of `wasm.wordpress.net` locally, run `yarn run dev`. To build it, run `yarn run build`. ### Fully qualified commands In reality, these `yarn run` commands are just triggering the underlying project's nx `dev` and `build` commands: ```bash nx dev playground-website nx build playground-website ``` Here's a few more interesting commands: ```bash # Build and run PHP.wasm CLI nx start php-wasm-cli # Build latest WordPress releases nx recompile-wordpress:all playground-remote # Recompile PHP 5.6 - 8.2 releases to .wasm for web nx recompile-php:all php-wasm-web # Recompile PHP 5.6 - 8.2 releases to .wasm for node nx recompile-php:all php-wasm-node # Builds markdown files for readthedocs site nx build docs-site # Builds the Playground Client npm package nx build playground-client ``` ## NX is the tool Playground needed from the outset It's ridiculous how many problems this solves: * The build pipeline is explicitly defined and easy to modify * Tasks only run once their dependencies are ready * The dev mode works and is fast * The build works and is fast * We get CI checks to confirm the entire build process still works (which solves #150) * Cross-package TypeScript just works * There are linters and formatters (which solves #14) * Documentation is correctly generated from the latest built artifacts * There are nice generators for bootstraping new packages and moving the existing ones around * There are checks to ensure the private `php-wasm-common` package is not imported by anything else than `php-wasm-web` and `php-wasm-node` ## Next steps * Add Lerna to harness package publishing * Additional developer documentation for the nx-based flow Related to #148 and #147

## Description Sets up the correct build pipeline for all parts of Playground and PHP.wasm. This enables a public release of the [Playground API](WordPress/wordpress-playground#149) npm package! I've been [struggling](WordPress/wordpress-playground#146) with [this](WordPress/wordpress-playground#70) for [a while](WordPress/wordpress-playground#150) and couldn't understand what's so hard. NX made it apparent – look at this dependency graph! <img width="1291" alt="CleanShot 2023-03-16 at 23 16 26@2x" src="https://user-images.githubusercontent.com/205419/225764795-7fa8e972-09f8-41ef-aac2-1c96bd100ea0.png"> No wonder it's been almost impossible to set everything up by hand! ## Usage Start with `yarn install` ### Shortcuts To start a copy of `wasm.wordpress.net` locally, run `yarn run dev`. To build it, run `yarn run build`. ### Fully qualified commands In reality, these `yarn run` commands are just triggering the underlying project's nx `dev` and `build` commands: ```bash nx dev playground-website nx build playground-website ``` Here's a few more interesting commands: ```bash # Build and run PHP.wasm CLI nx start php-wasm-cli # Build latest WordPress releases nx recompile-wordpress:all playground-remote # Recompile PHP 5.6 - 8.2 releases to .wasm for web nx recompile-php:all php-wasm-web # Recompile PHP 5.6 - 8.2 releases to .wasm for node nx recompile-php:all php-wasm-node # Builds markdown files for readthedocs site nx build docs-site # Builds the Playground Client npm package nx build playground-client ``` ## NX is the tool Playground needed from the outset It's ridiculous how many problems this solves: * The build pipeline is explicitly defined and easy to modify * Tasks only run once their dependencies are ready * The dev mode works and is fast * The build works and is fast * We get CI checks to confirm the entire build process still works (which solves #150) * Cross-package TypeScript just works * There are linters and formatters (which solves #14) * Documentation is correctly generated from the latest built artifacts * There are nice generators for bootstraping new packages and moving the existing ones around * There are checks to ensure the private `php-wasm-common` package is not imported by anything else than `php-wasm-web` and `php-wasm-node` ## Next steps * Add Lerna to harness package publishing * Additional developer documentation for the nx-based flow Related to WordPress/wordpress-playground#148 and WordPress/wordpress-playground#147

adamziel added 6 commits March 7, 2023 21:50

First stab at the packages split

649ee9b

Working migration to workspace packages

1ae0b8d

Add yarn

b2b2ed3

Yarn refresh

ca8e769

Setup topological build

1d84cec

Setup dev tasks

f8ba1b2

adamziel changed the title ~~Use Yarn workspaces~~ Migrate to Yarn workspaces Mar 8, 2023

adamziel added 11 commits March 8, 2023 21:11

Re-apply commit b84dfee

d657e34

Reapply commit db74b80

bf578af

Apply the importing/exporting commit

fdf8bc4

Working dev and production vite config

661eab5

Always use module workers

d2bd70f

Fix the iframe worker build process

7dc3f86

Fast dev mode for playground, a bunch of bugfixes

37077b3

Remove unused function

fb68258

Update tsconfig

d8553eb

Iron out remaining kinks in the configuration

1c6fcbb

Iron out remaining kinks in the configuration

916af7b

adamziel added 2 commits March 10, 2023 15:28

Update npm to yarn in the docs

fe917c9

Update the docs

f502e3b

adamziel closed this Mar 10, 2023

adamziel mentioned this pull request Mar 16, 2023

Migrate to NX-based build #151

Merged

adamziel deleted the cleanup-separate-different-logical-parts branch May 10, 2023 08:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate to Yarn workspaces #146

Migrate to Yarn workspaces #146

adamziel commented Mar 8, 2023 •

edited

Loading

adamziel commented Mar 8, 2023 •

edited

Loading

adamziel commented Mar 10, 2023 •

edited

Loading

adamziel commented Mar 10, 2023

adamziel commented Mar 10, 2023

Migrate to Yarn workspaces #146

Migrate to Yarn workspaces #146

Conversation

adamziel commented Mar 8, 2023 • edited Loading

Description

Rationale

adamziel commented Mar 8, 2023 • edited Loading

adamziel commented Mar 10, 2023 • edited Loading

adamziel commented Mar 10, 2023

adamziel commented Mar 10, 2023

adamziel commented Mar 8, 2023 •

edited

Loading

adamziel commented Mar 8, 2023 •

edited

Loading

adamziel commented Mar 10, 2023 •

edited

Loading