Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor build scripts and npm package process #5876

Merged
merged 32 commits into from
Apr 25, 2024

Conversation

etrepum
Copy link
Collaborator

@etrepum etrepum commented Apr 11, 2024

Addresses refactoring of npm/build processes per #5869

  • Adds some tests to sanity check packages (their package.json, the existence of flow and docs)
  • Refactors basically all of scripts/* to use a generated model for the available packages
  • Update scripts/www/rewriteImports.js to use the new infrastructure
  • Generate tsconfig.json paths (npm run update-tsconfig)
  • Generate .flowconfig paths (npm run update-flowconfig)
  • Generate boilerplate README.md and docs/packages/{package}.md files (npm run update-docs)
  • jest config
  • viteModuleResolution
  • docusaurus config
  • Started a Maintainers' Guide section in the docs (currently fairly hidden, only referenced in the repo README)
  • Adds build integration tests to make sure all of the examples and several other framework scenarios (astro, svelte, next.js) build and run with release artifacts

RE #5869 this mostly addresses the "Reduce number of changes needed to add new package" - since I have not added a new package myself yet I think another maintainer (@StyleT?) could add those docs to the guide using this as a baseline?

I was also thinking about maybe starting on an eslint plugin in the next week or two to help with the "rules of $functions" so if someone else doesn't get to it first I could add that documentation then when I am more familiar with all of those details.

Closes #5940 (parts of this PR were extracted there to maybe expedite review)

Copy link

vercel bot commented Apr 11, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
lexical ✅ Ready (Inspect) Visit Preview 💬 Add feedback Apr 24, 2024 10:51pm
lexical-playground ✅ Ready (Inspect) Visit Preview 💬 Add feedback Apr 24, 2024 10:51pm

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Apr 11, 2024
@etrepum
Copy link
Collaborator Author

etrepum commented Apr 11, 2024

@StyleT I wanted to put this up earlier on in case you had any feedback on this approach. Probably just look at the last commit, the rest of them are from the other PRs.

@etrepum etrepum changed the title [WIP] Refactor build scripts and npm package process [WIP][NO MERGE] Refactor build scripts and npm package process Apr 11, 2024
@etrepum etrepum force-pushed the refactor-npm-package-process-gh-5869 branch from d71489d to 2fdad6c Compare April 11, 2024 22:16
@etrepum etrepum force-pushed the refactor-npm-package-process-gh-5869 branch from 2fdad6c to 8c6a75f Compare April 11, 2024 22:19
@etrepum etrepum force-pushed the refactor-npm-package-process-gh-5869 branch from 8c6a75f to e0b5027 Compare April 12, 2024 07:04
@etrepum etrepum force-pushed the refactor-npm-package-process-gh-5869 branch from e0b5027 to 1267509 Compare April 12, 2024 23:33
@etrepum etrepum force-pushed the refactor-npm-package-process-gh-5869 branch from 1267509 to a7c7b2e Compare April 12, 2024 23:40
@etrepum etrepum force-pushed the refactor-npm-package-process-gh-5869 branch from a7c7b2e to 2b992e0 Compare April 13, 2024 17:09
@etrepum etrepum force-pushed the refactor-npm-package-process-gh-5869 branch from 2b992e0 to 64a414f Compare April 13, 2024 18:48
@etrepum etrepum force-pushed the refactor-npm-package-process-gh-5869 branch from 64a414f to a814bf1 Compare April 13, 2024 18:59

async function updateAllTsconfig() {
const prettierConfig = (await prettier.resolveConfig('./')) || {};
await updateTsconfig({
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that we shall not generate commited files w/o any mean for people to figure out that they are generated.

More correct approach would be to use the same approach that WXT does within devtools:

  1. You have tsconfig.json that extends generated config
  2. You generate "parent" config on postinstall hook and you do not commit it (or you may, but call it tsconfig.generated.json)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's fine, we can add more files if that's the preferred style. I was just following the existing convention.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extracted to tsconfig*.paths.json 224e201

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

224e201

Awesome! I would still suggest to name it tsconfig*.generated.json to emphasize that they are not manually written

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Had to do a follow-up fix since the jest config expected a specific layout of tsconfig.json 262fed0

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it seems that esbuild does not support having an array for the extends argument so we can't have the build json extend both the main tsconfig and a generated file, so I will roll back those changes and someone can revisit splitting up the generated tsconfig separately.

[vite:esbuild] The "path" argument must be of type string. Received an instance of Array
file: /Users/bob/src/lexical/packages/lexical-playground/src/index.tsx
error during build:
TypeError: The "path" argument must be of type string. Received an instance of Array
    at validateString (node:internal/validators:162:11)
    at Object.isAbsolute (node:path:1160:5)
    at resolveExtends (/Users/bob/src/lexical/node_modules/vite/dist/node/chunks/dep-6e2fe41e.js:19612:22)
    at parseExtends (/Users/bob/src/lexical/node_modules/vite/dist/node/chunks/dep-6e2fe41e.js:19593:34)
    at parse$f (/Users/bob/src/lexical/node_modules/vite/dist/node/chunks/dep-6e2fe41e.js:19544:24)
    at async loadTsconfigJsonForFile (/Users/bob/src/lexical/node_modules/vite/dist/node/chunks/dep-6e2fe41e.js:19933:24)
    at async transformWithEsbuild (/Users/bob/src/lexical/node_modules/vite/dist/node/chunks/dep-6e2fe41e.js:19736:36)
    at async Object.transform (/Users/bob/src/lexical/node_modules/vite/dist/node/chunks/dep-6e2fe41e.js:19821:32)
    at async transform (/Users/bob/src/lexical/node_modules/rollup/dist/shared/rollup.js:22085:16)
    at async ModuleLoader.addModuleSource (/Users/bob/src/lexical/node_modules/rollup/dist/shared/rollup.js:22311:30)

}
});
['examples', 'scripts/__tests__/integration/fixtures']
.flatMap((packagesDir) => glob.sync(`${packagesDir}/*/package.json`))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So if we use real packages as fixtures - why can't we simply download them as zip via npm pack?

And if those are not real NPM packages - better give them more of a semantic naming so it's clear what exactly are we testing here with every one of them.

Alternatively - we can put them as examples and add to docs. So it's not a "dead" code.
Imagine if one of them breaks test - that would be quite a challenge for anyone w/o context to figure out what's going wrong or how to fix it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also if aim is to simply test that Lexical exports code correctly - we can simply use Jest with prebuilt Lexical to test for concrete exported API

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They are real application packages, but not published to npm. Adding them as zip files would be something that could be done but then it is even harder to see why and how the tests broke. We can add them to examples but I did not do that here because I didn't want to have the discussions about whether these are officially endorsed or not since there are third party packages for some of these frameworks that offer better integration.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They are real application packages, but not published to npm. Adding them as zip files would be something that could be done but then it is even harder to see why and how the tests broke. We can add them to examples but I did not do that here because I didn't want to have the discussions about whether these are officially endorsed or not since there are third party packages for some of these frameworks that offer better integration.

Makes sense. Then can we give them more of a semantic naming so it's clear what exactly we test with every one of them. I'm puzzled why exactly svelte/astro/etc is important here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm puzzled why exactly svelte/astro/etc is important here.

Tons of people use these frameworks in open source

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would be open to cleaning up the test fixtures for these frameworks and making them "full" examples, but the goal of adding them here was to prevent future compatibility regressions since I've caused and fixed so many along the way to getting esm support. It is really hard to test so many combinations of frameworks that all have their own special way of using webpack/swc/esbuild/vite/etc. unless it's already in the monorepo test suite.

Angular would be a good one to do as a follow-up. The choice of next.js, sveltekit and astro were made because we are not testing them with examples in-tree (we are only testing a basic vite config with and without react), and we have gotten issues or discord problem reports about those frameworks specifically. I had them sitting around for my own testing.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What we are testing with them right now is simple:

  • does lexical build with this framework
  • did lexical render (tested by starting the build product & checking with playwright)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, got it. Sounds good to me.

Out of curiosity:
Can you give some example of any change to Lexical that won’t break vanilla js integration but will break any of these “non-React” frameworks

Copy link
Collaborator Author

@etrepum etrepum Apr 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The regressions that I noticed were mostly around changes to the package.json files and the fork modules. Rather than vanilla js vs. non-react frameworks it's usually of a question of vite vs swc vs webpack plus whatever configuration to those bundlers come with the framework.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#5823 is a concrete next.js issue that this test would've caught

.prettierignore Outdated Show resolved Hide resolved
scripts/update-docs.js Outdated Show resolved Hide resolved
test('install & build succeeded', () => {
expect(true).toBe(true);
});
test(`installed lexical ${monorepoVersion}`, () => {
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@StyleT this is the part of the integration tests that check to make sure that the node_modules of the example project or integration test fixture includes the latest version of lexical (at least up to the version number). There isn't really anything especially unique about the build artifacts to test against since it is trying to simulate what would happen if the commit of lexical under test was released to npm and then third party projects tried to use it

@acywatson acywatson merged commit 881258b into facebook:main Apr 25, 2024
46 checks passed
@etrepum etrepum deleted the refactor-npm-package-process-gh-5869 branch April 25, 2024 23:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants