Use folders instead of separate npm packages #70

adamziel · 2022-11-15T18:56:09Z

Description

This PR greatly simplifies the repository setup by getting rid of the separate package.json files as they made the setup unnecessarily complex.

Before this PR

Multiple tsconfig.json setups meant we needed to rebuild types in a particular order to even get the dev environment to work.
Esbuild needed two steps to first build each package separately and only then build the entire project. Worse, step one used ES Modules and step two used CommonJS. Using tsconfig.json made it extra hard. Substituting env variables caused conflicts.
Generating the documentation required 8 separate configuration files
Adding a new package was tedious. It required adding new package.json, tsconfig.json, api-documenter.json, adjusting all the build pipelines, ensuring the build steps are triggered in the correct order etc.

After this PR

There are two tsconfig.json files in the repo root and that's it. One is for the main-thread code and the other is for the worker code.
ESbuild runs in a single step and is configured with a single JS object.
There are no more documentation generator configs, it's all automated.
Introducing a new section of the repository is as simple as creating a new directory. There's no need to think about configs, build pipelines etc

This PR splits the repository into five node packages that are also yarn workspaces: - php-cli - php-wasm - typescript-reference-doc-generator - wordpress-playground - wordpress-plugin-ide This effectively undoes [Use folders instead of separate npm packages](#70), so you may ask "why bother?" With this PR: * The build process is straightforward. There's no `gulp` and there's no mountain of custom logic in `esbuild.js`. Files are moved around the repo using a simple `copy` plugin or an `import` statement. There's no custom cache busting logic – it's handled by the build tools instead * `php-wasm` is now reusable and can be published in npm. This means it can be easily reused outside of WordPress Playground. * Each package has its own build process that caters to its specific needs. For example, `php-cli` outputs a node.js script. Web workers can now be bundled as `iife` (which is the only way to use them in firefox) while other assets are es modules. * There are less tasks in general. For example, TypeScript types are handled by rollup in a few packages. At the same time, none of the [original downsides of using packages ](#70) are present: * There's no global build pipeline. Every package outputs a complete build artifact that can be directly imported in case of libs, or used in the browser in case of the Playground website. * Yarn workspaces ensure the tasks are executed in the correct order * tsconfig.json and package.json can mostly be copied, there's no `api-documenter.json` at all.

## Description Sets up the correct build pipeline for all parts of Playground and PHP.wasm. This enables a public release of the [Playground API](#149) npm package! I've been [struggling](#146) with [this](#70) for [a while](#150) and couldn't understand what's so hard. NX made it apparent – look at this dependency graph! <img width="1291" alt="CleanShot 2023-03-16 at 23 16 26@2x" src="https://user-images.githubusercontent.com/205419/225764795-7fa8e972-09f8-41ef-aac2-1c96bd100ea0.png"> No wonder it's been almost impossible to set everything up by hand! ## Usage Start with `yarn install` ### Shortcuts To start a copy of `wasm.wordpress.net` locally, run `yarn run dev`. To build it, run `yarn run build`. ### Fully qualified commands In reality, these `yarn run` commands are just triggering the underlying project's nx `dev` and `build` commands: ```bash nx dev playground-website nx build playground-website ``` Here's a few more interesting commands: ```bash # Build and run PHP.wasm CLI nx start php-wasm-cli # Build latest WordPress releases nx recompile-wordpress:all playground-remote # Recompile PHP 5.6 - 8.2 releases to .wasm for web nx recompile-php:all php-wasm-web # Recompile PHP 5.6 - 8.2 releases to .wasm for node nx recompile-php:all php-wasm-node # Builds markdown files for readthedocs site nx build docs-site # Builds the Playground Client npm package nx build playground-client ``` ## NX is the tool Playground needed from the outset It's ridiculous how many problems this solves: * The build pipeline is explicitly defined and easy to modify * Tasks only run once their dependencies are ready * The dev mode works and is fast * The build works and is fast * We get CI checks to confirm the entire build process still works (which solves #150) * Cross-package TypeScript just works * There are linters and formatters (which solves #14) * Documentation is correctly generated from the latest built artifacts * There are nice generators for bootstraping new packages and moving the existing ones around * There are checks to ensure the private `php-wasm-common` package is not imported by anything else than `php-wasm-web` and `php-wasm-node` ## Next steps * Add Lerna to harness package publishing * Additional developer documentation for the nx-based flow Related to #148 and #147

See WordPress/playground-tools#32 See WordPress#503 (comment) ## What? Adds a failing test case for WordPress/playground-tools#32 and marks it as a skipped test. ## Why? It will make it easier to evaluate whether WordPress#503 fixes the issue. ## Testing Instructions 1. Change `test.skip` to `test.only`. 2. Run `nx test wp-now --watch`. 3. Verify the test fails on the second assertion. 4. Comment out the second assertion and hit save. 5. Verify the test fails on the third assertion. 6. Comment out the third assertion and hit save. 7. Verify the test fails on the fourth assertion.

This PR splits the repository into five node packages that are also yarn workspaces: - php-cli - php-wasm - typescript-reference-doc-generator - wordpress-playground - wordpress-plugin-ide This effectively undoes [Use folders instead of separate npm packages](WordPress/wordpress-playground#70), so you may ask "why bother?" With this PR: * The build process is straightforward. There's no `gulp` and there's no mountain of custom logic in `esbuild.js`. Files are moved around the repo using a simple `copy` plugin or an `import` statement. There's no custom cache busting logic – it's handled by the build tools instead * `php-wasm` is now reusable and can be published in npm. This means it can be easily reused outside of WordPress Playground. * Each package has its own build process that caters to its specific needs. For example, `php-cli` outputs a node.js script. Web workers can now be bundled as `iife` (which is the only way to use them in firefox) while other assets are es modules. * There are less tasks in general. For example, TypeScript types are handled by rollup in a few packages. At the same time, none of the [original downsides of using packages ](WordPress/wordpress-playground#70) are present: * There's no global build pipeline. Every package outputs a complete build artifact that can be directly imported in case of libs, or used in the browser in case of the Playground website. * Yarn workspaces ensure the tasks are executed in the correct order * tsconfig.json and package.json can mostly be copied, there's no `api-documenter.json` at all.

## Description Sets up the correct build pipeline for all parts of Playground and PHP.wasm. This enables a public release of the [Playground API](WordPress/wordpress-playground#149) npm package! I've been [struggling](WordPress/wordpress-playground#146) with [this](WordPress/wordpress-playground#70) for [a while](WordPress/wordpress-playground#150) and couldn't understand what's so hard. NX made it apparent – look at this dependency graph! <img width="1291" alt="CleanShot 2023-03-16 at 23 16 26@2x" src="https://user-images.githubusercontent.com/205419/225764795-7fa8e972-09f8-41ef-aac2-1c96bd100ea0.png"> No wonder it's been almost impossible to set everything up by hand! ## Usage Start with `yarn install` ### Shortcuts To start a copy of `wasm.wordpress.net` locally, run `yarn run dev`. To build it, run `yarn run build`. ### Fully qualified commands In reality, these `yarn run` commands are just triggering the underlying project's nx `dev` and `build` commands: ```bash nx dev playground-website nx build playground-website ``` Here's a few more interesting commands: ```bash # Build and run PHP.wasm CLI nx start php-wasm-cli # Build latest WordPress releases nx recompile-wordpress:all playground-remote # Recompile PHP 5.6 - 8.2 releases to .wasm for web nx recompile-php:all php-wasm-web # Recompile PHP 5.6 - 8.2 releases to .wasm for node nx recompile-php:all php-wasm-node # Builds markdown files for readthedocs site nx build docs-site # Builds the Playground Client npm package nx build playground-client ``` ## NX is the tool Playground needed from the outset It's ridiculous how many problems this solves: * The build pipeline is explicitly defined and easy to modify * Tasks only run once their dependencies are ready * The dev mode works and is fast * The build works and is fast * We get CI checks to confirm the entire build process still works (which solves #150) * Cross-package TypeScript just works * There are linters and formatters (which solves #14) * Documentation is correctly generated from the latest built artifacts * There are nice generators for bootstraping new packages and moving the existing ones around * There are checks to ensure the private `php-wasm-common` package is not imported by anything else than `php-wasm-web` and `php-wasm-node` ## Next steps * Add Lerna to harness package publishing * Additional developer documentation for the nx-based flow Related to WordPress/wordpress-playground#148 and WordPress/wordpress-playground#147

adamziel added 2 commits November 9, 2022 10:58

Experiment with removing the notion of a pakcage:

4b08258

Migrate separate npm packages to separate directories

96bacf9

adamziel force-pushed the refactor/no-packages branch 2 times, most recently from d534592 to 96bacf9 Compare November 15, 2022 18:59

adamziel added 4 commits November 15, 2022 19:08

Remove api extractor configuration files

e5d6d22

Clean ts cache before rebuilding types

d66df21

Refresh the markdown docs

29c5062

Simplify the docs building pipeline

e0fdfe2

adamziel changed the title ~~Refactor/no packages~~ Use folders instead of separate npm packages Nov 15, 2022

adamziel added Infrastructure [Type] Developer Experience labels Nov 15, 2022

adamziel self-assigned this Nov 15, 2022

adamziel marked this pull request as ready for review November 15, 2022 19:24

adamziel merged commit 44289e6 into trunk Nov 15, 2022

adamziel deleted the refactor/no-packages branch November 15, 2022 19:25

adamziel mentioned this pull request Mar 8, 2023

Migrate to Yarn workspaces #146

Closed

adamziel mentioned this pull request Mar 16, 2023

Migrate to NX-based build #151

Merged

adamziel removed the Infrastructure label Jun 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use folders instead of separate npm packages #70

Use folders instead of separate npm packages #70

adamziel commented Nov 15, 2022 •

edited

Loading

Use folders instead of separate npm packages #70

Use folders instead of separate npm packages #70

Conversation

adamziel commented Nov 15, 2022 • edited Loading

Description

Before this PR

After this PR

adamziel commented Nov 15, 2022 •

edited

Loading