Merge Modular 4 Changes to Main (#2264)

* Node 18 & Jest 29 (#2210)

* Add Node 18 to tests

* Add Node 18 to supported engines

* Pinned rollup-plugin-esbuild to 4.10.1 as newer versions drop support for node 14.17 and below

* Updated yarn.lock

* Update build.test snapshot

* Update modular-scripts snapshot

* Change rollup-plugin-eslint to latest compatible version

* Added changeset

* Update lockfile

* Resolve to source-map ^0.7.0 to try and fix mozilla/source-map#432

* Try to resolve v8-to-istanbul to fix mozilla/source-map#432

* Dropped support for Node 14.17.0 (14.18.0 and greater) to allow upgrate to rollup-plugin-eslint 5

* Update tests to use Node 14.18.0

* Make tests use Node 14.18.0

* update jest to 29

* change Node 16 supported version to 16.10+ as that's what Jest supports

* Upgrade to Jest 29 and fix problems caused by the upgrade

* Update changelog

* Update snapshots (default no longer includes escape characters etc)

* Update snapshots not to include espace caracters

* More snapshots updates

* Change env for port tests to Node

* Update snapshots and switch default test env to Node (while setting it to JSDOM for specific tests)

* Update more snapshots

* Remove broken Rename test (Rename functionality to be removed with Modular 4)

* Fix tests (test.test wasn't closing after running with the --watchAll flag, and adding __fixtures__ to the lint exception list made tests fail)

* Try to fix tests on Windows

* DEBUG windows tests

* Fix windows tests

* Try more things to fix tests

* More attempts to fix windows tests

* More desperate attempts to fix windows tests

* Clean up windows fix

* Cleanup

* Update changeset

* Set Jest watchAll default to false regardless of if CI or not

* Added ESLint ignore comments to svgr.ts for new TS complaints

* Documentation

* Update docs

* Add feature/v4 to branches to run CI Tests on (Needs to be reverted before merging to main)

* Small fixes

* Refactor out generateJestConfig flag logic to reduce duplication

* Update to ESLint 8 & Supported TypeScript to >4.5.3 (#2216)

* Update ESLint to ^8.0.0, minimum Typescript to 4.5.3 & update dependencies

* Update changeset

* Fix warnings generated by updated ESLint

* Remove commands and update documentation (#2220)

* Remove commands and update documentation

* Update Readme & Changeset

* Merge main (#2225)

Merge changes to modular 3.x from main since feature/v4 was split out

* Use Config file instead of Environment Variables  (#2227)

* Use cosmiconfig to read modular configuration, while allowing Env variables to override it 

* Add configuration tests

* Change the default CDN to esm.sh

* Update documentation to reflect new configuration approach

* Use INTERNAL_ env variables to read configuration in JS files that can't call the config function

* Refactor config() logic 
Co-authored-by: Steve King <steve@mydev.co>

* Add jsx to test glob pattern (#2234)

* Add jsx to test glob pattern

* Merge main into Modular 4 (#2237)

* Merge main into Modular 34

* Sunset modular-site

* Revert "Sunset modular-site"

This reverts commit a27388b.

* Sunset modular-site  (#2238)

* Sunset modular-site & update test snapshot

* Remove requirement for Apps to be private in modualr check (#2241)

* Update lockfile

* Docs/support (#2248)

* Initial cleanup

* Add compatibility page, remove recipes/yarn

* Better markdown section depth

* update nav order

* Document package types (#2246)

* Start documenting package types

* Describe App and ESM View types

* Packages

* Views, sources and templates

* Link add command page

* Link package types in index

* Update docs/commands/add.md

Co-authored-by: Sam Brown <sam.brown@jpmorgan.com>

* Update docs/concepts/package-types.md

Co-authored-by: Sam Brown <sam.brown@jpmorgan.com>

* Update docs/concepts/package-types.md

Co-authored-by: Sam Brown <sam.brown@jpmorgan.com>

* Link esm-views section in add page

* Update docs/commands/add.md

Co-authored-by: Sam Brown <sam.brown@jpmorgan.com>

* Link app section for more info

* Update docs/concepts/package-types.md

Co-authored-by: Sam Brown <sam.brown@jpmorgan.com>

* Clarify manifest -> package.json + fix incorrect description

* Add reasons why we don't support things

* Disambiguate words

* Phrase 'default export' concept better

* Update docs/concepts/package-types.md

Co-authored-by: Sam Brown <sam.brown@jpmorgan.com>

* Update docs/concepts/package-types.md

Co-authored-by: Sam Brown <sam.brown@jpmorgan.com>

* Update docs/commands/add.md

Co-authored-by: Sam Brown <sam.brown@jpmorgan.com>

* Reword tricky sentence

* Update docs/commands/add.md

Co-authored-by: Sam Brown <sam.brown@jpmorgan.com>

* Update docs/concepts/package-types.md

Co-authored-by: Sam Brown <sam.brown@jpmorgan.com>

* Split long sentence

* Add all package types to configuration docs

* Various typos

* Split packge types and move template page

* Fix links to package types and fix table layout

* Slim down table

* build/start/entrypoint/template sections

* Fix configuration

* Remove list of types in documentation

Co-authored-by: Sam Brown <sam.brown@jpmorgan.com>

* Generate nicer READMEs inside new projects / apps / views / packages (#2253)

* Start documenting package types

* Describe App and ESM View types

* Packages

* Views, sources and templates

* Link add command page

* Link package types in index

* Update docs/commands/add.md

Co-authored-by: Sam Brown <sam.brown@jpmorgan.com>

* Update docs/concepts/package-types.md

Co-authored-by: Sam Brown <sam.brown@jpmorgan.com>

* Update docs/concepts/package-types.md

Co-authored-by: Sam Brown <sam.brown@jpmorgan.com>

* Link esm-views section in add page

* Update docs/commands/add.md

Co-authored-by: Sam Brown <sam.brown@jpmorgan.com>

* Link app section for more info

* Update docs/concepts/package-types.md

Co-authored-by: Sam Brown <sam.brown@jpmorgan.com>

* Clarify manifest -> package.json + fix incorrect description

* Add reasons why we don't support things

* Disambiguate words

* Phrase 'default export' concept better

* Update docs/concepts/package-types.md

Co-authored-by: Sam Brown <sam.brown@jpmorgan.com>

* Update docs/concepts/package-types.md

Co-authored-by: Sam Brown <sam.brown@jpmorgan.com>

* Update docs/commands/add.md

Co-authored-by: Sam Brown <sam.brown@jpmorgan.com>

* Reword tricky sentence

* Update docs/commands/add.md

Co-authored-by: Sam Brown <sam.brown@jpmorgan.com>

* Update docs/concepts/package-types.md

Co-authored-by: Sam Brown <sam.brown@jpmorgan.com>

* Split long sentence

* Add all package types to configuration docs

* Various typos

* Split packge types and move template page

* Fix links to package types and fix table layout

* Slim down table

* build/start/entrypoint/template sections

* Add root project README

* Add per-package READMEs + fix docs

* Lint view documentation

* Update snapshots

* Upddate app snapshots

* Update snapshots in index test

* Update app.esbuild.test snapshots

* update cmra tests

* Correct snapshots for cmra

* Update snapshots for cli.test.ts

* Remove packages README + fix all READMEs

* Add README to fixtures + update snapshots

* Update various snapshots

* Update app.esbuild.test snapshots

* Add default workspace README

* Update index snapshots

* Create odd-bees-speak.md

* Update fixture READMEs

Co-authored-by: Sam Brown <sam.brown@jpmorgan.com>
Co-authored-by: Alberto Brusa <alberto.brusa@live.com>

* Sunset philosophy / views, write intro, fix web workers, correct templates (#2254)

* Sunset philopsophy / views, move web workers, correct templates

* Better front page

* Fix command format

* Add web workers docs

* Enhance modular-scripts Tests (#2240)

* Update tests in modular-scripts to use tmp directories instead of changing things within the repo

* Consolidate all execa calls to yarn modular under one standardised function

* Change calls to modular during tests to not run checks 

* Refactor tests for brevity, code reuse and raedability 

* Remove unnecessary build snapshots from esmVies.test

* Modify getConfig to get workspace specific configuration (#2258)

* Modify getConfig to get workspace specific configuration

* Update config docs to reflect per workspace configuration

* Document supported CRA features (#2255)

* Rename limitations + start CRA page

* Add CRA supported features

* Fix typo

* Fixes in CRA page

* Test default template tests work (#2260)

* Unify modular test command interface (#2259)

* regex -> option, package -> argument

* Windows test use --package

* Remove space that fails test

* Add debug statement

* Implement selective testing and merging with user regexes

* Update docs

* Better wording

* Fix typo

* Test selective options combinations

* modular build builds all packages

* Test selective builds

* Document behaviour of modular build

* Create green-shrimps-build.md

* Fix test docs

* various docs fixes (#2262)

* Small fixes (#2261)

* Remove unnecessary error log

* Move jest-environment-jsdom to correct package.json

* Update typescript set by CMRA

* More small fixes (#2263)

* Remove unnecessary error log

* Move jest-environment-jsdom to correct package.json

* Remove jest-environment-jsdom from root package.json

* reorder dependencies

* v4.0.2

* v1.1.1

* v4.0.0

* Update typescript set by CMRA

* Revert accidentally changed versions and bump CMRA version by major due to update TS version

* Remove feature/v4 from GitHub Action tests & introduce forgotten CMRA changeset

* Update 4.0.x release notes

* Slight changes to release notes

* Additions to release doc

Co-authored-by: Cristiano Belloni <cristiano.belloni@jpmorgan.com>
Co-authored-by: Cristiano Belloni <cristiano-belloni@users.noreply.github.com>
Co-authored-by: Sam Brown <sam.brown@jpmorgan.com>

.changeset/blue-turtles-shake.md

@@ -0,0 +1,5 @@
+---
+'modular-scripts': major
+---
+
+Removed commands: convert, init, port, rename

.changeset/cyan-flowers-rest.md

@@ -0,0 +1,6 @@
+---
+'modular-scripts': major
+---
+
+Changed default CDN from Skypack to esm.sh as skypack is no longer actively
+maintained. Add support for configuring modular through a configuration file.

.changeset/fair-kangaroos-retire.md

@@ -0,0 +1,5 @@
+---
+'modular-scripts': patch
+---
+
+Test pattern now includes jsx

.changeset/fluffy-dragons-behave.md

@@ -0,0 +1,14 @@
+---
+'create-modular-react-app': major
+'eslint-config-modular-app': major
+'modular-scripts': major
+'modular-template-app': patch
+'modular-template-esm-view': patch
+'modular-template-view': patch
+---
+
+Added Node 18 engine support
+Upgraded Jest from 26 to 29 as 26 wasn't compatible with Node 18
+Upgraded to rollup-plugin-esbuild 5, dropping support for Node 14.17 and below
+Supported Node versions now: ^14.18.0 || >=16.10.0 || >=18.0.0
+Changed Jest flag --watchAll default to false (was previously true if running locally and not in CI)

.changeset/green-shrimps-build.md

@@ -0,0 +1,5 @@
+---
+"modular-scripts": major
+---
+
+Fully selective `modular test` command interface, compatible with `modular build`.

.changeset/itchy-dancers-brush.md

@@ -0,0 +1,6 @@
+---
+'eslint-config-modular-app': major
+'modular-scripts': major
+---
+
+Updated eslint to ^8.0.0 and minimum supported TypeScript version to 4.5.3

.changeset/light-countries-report.md

@@ -0,0 +1,5 @@
+---
+'create-modular-react-app': major
+---
+
+Bumped default TypeScript version to ^4.8.3

.changeset/odd-bees-speak.md

@@ -0,0 +1,12 @@
+---
+"create-modular-react-app": minor
+"modular-scripts": minor
+"modular-template-app": minor
+"modular-template-esm-view": minor
+"modular-template-package": minor
+"modular-template-source": minor
+"modular-template-view": minor
+---
+
+Generate README inside newly created packages
+Improve root and default workspaces container README

.changeset/purple-numbers-camp.md

@@ -0,0 +1,5 @@
+---
+'modular-scripts': minor
+---
+
+App type modular packages are no longer required to be private

.changeset/wet-gifts-count.md

@@ -0,0 +1,5 @@
+---
+'modular-scripts': major
+---
+
+Sunset modular-site package

.github/workflows/release.yml

@@ -20,7 +20,7 @@ jobs:
       - name: Use Node.js
         uses: actions/setup-node@v3.3.0
         with:
-          node-version: '14.17.0'
+          node-version: '14.18.0'
           registry-url: https://registry.npmjs.org/
           cache: 'yarn'
 

.github/workflows/static.yml

@@ -29,7 +29,7 @@ jobs:
       - name: Use Node.js
         uses: actions/setup-node@v3.3.0
         with:
-          node-version: '14.17.0'
+          node-version: '14.18.0'
           cache: 'yarn'
 
       - name: 'Install Dependencies'

.github/workflows/test-windows.yml

@@ -20,7 +20,7 @@ jobs:
       fail-fast: false
       matrix:
         os: [windows-latest]
-        node-version: ['14.17.0', '16.x']
+        node-version: ['14.18.0', '16.x', '18.x']
 
     steps:
       - uses: actions/checkout@v3
@@ -34,4 +34,6 @@ jobs:
       - name: 'Build internal prerequisites'
         run: yarn workspace @modular-scripts/workspace-resolver build
       - name: Run Windows tests
-        run: yarn test esmView.test.ts workspace-resolver addPackage.test.ts
+        run:
+          yarn test --regex esmView.test.ts workspace-resolver
+          addPackage.test.ts

.github/workflows/test.yml

@@ -22,7 +22,7 @@ jobs:
       fail-fast: false
       matrix:
         os: [ubuntu-latest]
-        node-version: ['14.17.0', '16.x']
+        node-version: ['14.18.0', '16.x', '18.x']
 
     steps:
       - uses: actions/checkout@v3

.gitignore

@@ -53,3 +53,9 @@ yarn-error.log*
 .tsbuildinfo
 *.tsbuildinfo
 
+# Local Jekyll preview of docs
+docs/_site
+docs/.bundle
+docs/vendor
+docs/Gemfile*
+

__fixtures__/templates/modular-template-app/README.md

@@ -0,0 +1 @@
+# This is a README placeholder for this template fixture

__fixtures__/templates/modular-template-filter/README.md

@@ -0,0 +1 @@
+# This is a README placeholder for this template fixture

__fixtures__/templates/modular-template-no-filter/README.md

@@ -0,0 +1 @@
+# This is a README placeholder for this template fixture

__fixtures__/test-config/.modular.js

@@ -0,0 +1,3 @@
+module.exports = {
+  useModularEsbuild: true,
+};

docs/commands/add.md

@@ -3,48 +3,79 @@ parent: Commands
 title: modular add
 ---
 
-# `modular add <packageName>`
+# `modular add [options] [packageName]`
 
-Adds a new package by creating a new workspace at `packages/<packageName>`,
-omitting the scope if the package is
+Adds a new package by creating a new package at the workspace located at
+`packages/<packageName>`, omitting the scope if the package is
 [scoped](https://docs.npmjs.com/cli/v8/using-npm/scope). If `--path <somePath>`
-is specified, create the workspace at `<somePath>/<packageName>`.
+is specified, the command creates the workspace at `<somePath>/<packageName>`.
 
 (i.e. `modular add my-app` would create a package in `packages/my-app`,
 `modular add @scoped/my-scoped-app` would create a package in
 `packages/my-scoped-app` and `modular add lib-a --path libs` would create a
 package in `libs/lib-a`)
 
-Packages can currently be one of the following types:
+The `modular add` command prompts the user to choose the Modular `type` of the
+package it's about to create. The next section briefly describes the various
+types that can be created by the `modular add` command. For an in-depth
+discussion of the available package types and their characteristics, please see
+[this page](../package-types/index.md).
 
-- A standalone `app`. This corresponds to a static Single Page Application (SPA)
-  project in a workspace. Inside this workspace, you can import packages from
-  other workspaces freely, and features like jsx and typechecking work out of
-  the box.
+### Standalone (bundled) package types
 
-- An `esm-view`, which is a package that typically exports a React component by
-  default. ESM Views are built as ES modules that can be `import`ed at runtime
-  by a host to implement a [micro frontend](../concepts/microfrontends.md)
-  architecture or started as a normal standalone application. See also
-  [the view building reference](../esm-views/index.md)
+These package types are built with [Webpack v5](https://webpack.js.org/) or, if
+specified in the [configuration](../configuration.md),
+[esbuild](https://esbuild.github.io/). Modules imported in the source of these
+package types are bundled in the final result (in case of `esm-view`s, only
+local modules get bundled, and external dependencies are rewritten to use an
+external ESM CDN. [This section](../esm-views/index.md) explains the process in
+more depth).
 
-- A `view`, which is a `package` that exports a React component by default. Read
-  more about Views in [this explainer](../concepts/views.md).
+- [`app`](../package-types/app.md). This package type corresponds to a static
+  Single Page Application (SPA) project in a workspace. It's possible to specify
+  a custom `index.html` file and public assets in the `public` directory. See
+  [this page](../package-types/#app) for more information about apps.
 
-- A generic JavaScript `package`. You can use this to create a library with an
-  entry point that gets transpiled to Common JS and ES Module format when built.
-  Packages can be [built](../commands/build.md) but not
-  [start](../commands/start.md)ed by Modular.
+- [`esm-view`](../package-types/esm-view.md). This package type is an app that
+  gets built as an ES module that can be imported at runtime. `esm-view`s are
+  typically used to implement a [micro-frontend](../concepts/microfrontends.md)
+  architecture. `esm-views`, when [built](./build.md) or [started](./start.md)
+  will also generate a `index.html` file that tries to load the ES Module and
+  render its default export as a React component onto the DOM (standalone mode).
+  See also [the esm-view reference](../esm-views/index.md) for an in-depth
+  introduction.
 
-- A `source`, which is a shared package that is imported by other packages from
-  source (i.e. directly importing its source), and it's never built standalone
-  or published. This kind of package is never [built](../commands/build.md) or
-  [start](../commands/start.md)ed by Modular.
+### Library package types
+
+These package types are either built with
+[Rollup.js](https://rollupjs.org/guide/en/) as CommonJS and ES Modules or, in
+case of `source` modules, they are not built at all. Library package types get
+typically published to NPM (`package` and `view` types) or get imported by other
+packages in the monorepo (`source` type). For this reason, files are transpiled
+separately on build and external dependencies are never "pulled in" (i.e. not
+included in a bundle).
+
+- [`package`](../package-types/package.md). This is a generic package with a
+  single entry point. It's normally used to create a publishable library that
+  gets transpiled to CommonJS and ES Module format when built. Packages can be
+  [built](../commands/build.md) but not [start](../commands/start.md)ed by
+  Modular.
+
+- [`view`](../package-types/view.md). This is a `package` that exports a default
+  React component. Views are built exactly like `package`s, but, since Modular
+  knows that the default export can be rendered, `view`s can be
+  [`modular start`](../start.md)ed to preview them locally.
+
+- [`source`](../package-types/source.md). A shared package that is imported by
+  other package types in the monorepo, directly specifying one or more of its
+  source files. This kind of package can be never [built](../commands/build.md)
+  or [start](../commands/start.md)ed by Modular.
 
 ## Options:
 
-`--path`: Optionally set the directory in which the workspace is created. If the
-provided path is outside (i.e., not a descendant) of the paths specified in
+`--path <targetPath>`: Optionally set the directory in which the workspace is
+created. If the provided path is outside (i.e., not a descendant) of the paths
+specified in
 [the `workspaces` field](https://classic.yarnpkg.com/lang/en/docs/workspaces/#toc-how-to-use-it)
 of the root `package.json`, the command will fail
 
@@ -55,4 +86,4 @@ of the root `package.json`, the command will fail
 
 `--template <templateName>`: Use the package `templateName` from the repository
 or the registry as a template for the new package. Find more information about
-Modular templates [in this page](../concepts/templates.md)
+Modular templates [in this page](../package-types/template.md)

docs/commands/build.md

@@ -3,7 +3,7 @@ parent: Commands
 title: modular build
 ---
 
-# `modular build [packages...]`
+# `modular build [options] [packages...]`
 
 Search workspaces based on their `name` field in the `package.json` and build
 them according to their respective `modular.type`, in order of dependency (e.g.
@@ -13,6 +13,13 @@ The output directory for built artifacts is `dist/`, which has a flat structure
 of modular package names. Each built app/view/package is added to the `dist/` as
 its own folder.
 
+When `packages` is empty and no selective options have been specified (for
+example when running `yarn modular build`), all packages in the monorepo will be
+built. When `packages` contains one or more non-existing package name, the
+non-existing packages will be ignored without an error. If any package or
+selective option have been defined but the final set of regular expressions is
+empty, Modular will write a message to `stdout` and exit with code `0`.
+
 For views and packages, package names are transformed to `Param case` (e.g.
 this-is-param-case) in `dist/`
 
@@ -25,7 +32,7 @@ this-is-param-case) in `dist/`
 
 `--preserve-modules`: Preserve module structure in generated modules.
 
-`--changed`: Build only packages whose workspaces contain files that have
+`--changed`: Build only the packages whose workspaces contain files that have
 changed. Files that have changed are calculated comparing the current state of
 the repository with the branch specified by `compareBranch` or, if
 `compareBranch` is not set, with the default git branch.

docs/commands/check.md

@@ -3,11 +3,13 @@ parent: Commands
 title: modular check
 ---
 
-# `modular check`
+# `modular check [options]`
 
 Checks the modular root repo has yarn workspaces and modular packages are set up
 properly and checks your package tree for issues with your dependencies.
 
 ## Options:
 
-`--fix`: Run autofix over applications.
+`--fix`: Run autofix over applications
+
+`--verbose`: Run yarn commands with the --verbose flag set

docs/commands/lint.md

@@ -3,7 +3,7 @@ parent: Commands
 title: modular lint
 ---
 
-# `modular lint`
+# `modular lint [options] [regexes...]`
 
 `modular lint` will check the diff between the current branch and your remote
 origin default branch (i.e. `master` or `main`) and only lint the `.ts`, `.tsx`,