[core] Change package manager to pnpm #36287

Janpot · 2023-02-21T18:17:00Z

Migrated the Core repo from Yarn Classic to pnpm.

Motivation and getting started with pnpm: https://mui-org.notion.site/pnpm-32e8449c77ce403a9f4582a86f1a6c79

Summary of the changes:

set up the packageManager field in the root package.json
created the pnpm-workspace.yaml file with all projects in the repository
changes workspace project inter-dependencies to use the workspace: protocol
replaced all occurrences of "yarn" to "pnpm" across scripts, comments, and docs (adjusting the commands where necessary)
updated Lerna, Renovate, and CI configs to be aware of pnpm
set up pnpm to link workspace project using build outputs (usually the "build" directory under project root). Bundlers and other tools use aliases to point to the untranspiled source code.

Since the question came up in Slack. Just to see what a migration looks like. This PR mostly to see how the @mui/monorepo dependency will behave in X/Toolpad

Migrated the repo up until pnpm docs:dev works. None of the other commands have been tested or migrated. Nothing in the CI pipeline has been migrated.

Some quick notes

https://dev.to/andreychernykh/yarn-npm-to-pnpm-migration-guide-2n04

pnpm import
 ERR_PNPM_INVALID_OVERRIDE_SELECTOR  Cannot parse the "**/@babel/core" selector in the overrides

=> remove **/ everywhere?

added .npmrc with
```
auto-install-peers = true
```
replace workspace dependencies versions in package.json files with workspace:*
scripts:
- yarn workspace docs dev => pnpm --dir ./docs dev

oliviertassinari · 2023-02-21T18:53:43Z

I would be curious to see if it speed-up the CI, and by how much. It might be the biggest selling point. https://mui-org.notion.site/Migrate-to-pnpm-2a4f2e9869cf496eae183eff2aeef9a9

Janpot · 2023-02-21T19:34:48Z

Yep, it already feels quite fast locally. I think there's two roads to continue with this

first try to migrate CI in this repo, to see the impact on build time to see if we should even bother porting to the other repos
first try to make the docs work on Toolpad or X to see if we should even bother spending time migrating CI

packages/mui-utils/package.json

mnajdova

I left few clarifying questions/comments. Would be great to validate them before merging.

docs/README.md

mnajdova · 2024-01-03T09:01:39Z

packages/api-docs-builder-core/tsconfig.json

@@ -14,7 +14,7 @@
    "strict": true,
    "baseUrl": "./",
    "paths": {
-      "react-docgen": ["../api-docs-builder/react-docgen.d.ts"]


Why is this no longe required?

React-docgen types live in a separate package and are simply added as a dev dependency.

mnajdova · 2024-01-03T09:03:05Z

packages/mui-base/package.json

-    "@mui-internal/test-utils": "^1.0.0",
+    "@mui-internal/babel-macros": "workspace:^",
+    "@mui-internal/test-utils": "workspace:^",
+    "@mui/base": "workspace:*",


Hmm, do we need to define the same package as its own dependency?

Yes, otherwise, we won't be able to access the @mui/base contents in tests.

mnajdova · 2024-01-03T09:03:50Z

packages/mui-base/src/Badge/Badge.tsx

@@ -94,10 +94,10 @@ const Badge = React.forwardRef(function Badge<RootComponentType extends React.El
 }) as PolymorphicComponent<BadgeTypeMap>;

 Badge.propTypes /* remove-proptypes */ = {
-  // ----------------------------- Warning --------------------------------


These changes look unrelated, should we revert in order to have a smaller pull request?

I had to change the yarn to pnpm in the message, so I figured I might as well fix the grammar and formatting bit. Revering those additional changes won't make the number of changed files smaller.

Aaah, I missed that, I thought it is just the formatting. Got ya, let's keep them.

mnajdova · 2024-01-03T09:06:03Z

packages/mui-joy/src/Autocomplete/Autocomplete.tsx

-  // | These PropTypes are generated from the TypeScript type definitions |
-  // |     To update them edit TypeScript types and run "yarn proptypes"  |
-  // ----------------------------------------------------------------------
+  // ┌────────────────────────────── Warning ──────────────────────────────┐


I just left a comment above too. The pull request is anyways heavy, it would be great if we can revert this particular change.

mnajdova · 2024-01-03T09:07:28Z

packages/mui-material-nextjs/tsconfig.json

@@ -1,6 +1,7 @@
 {
  "extends": "../../tsconfig.json",
  "compilerOptions": {
+    "skipLibCheck": true,


What was failing here?

Mismatched NextJS version. I just pinned the nextjs version and removed this compiler option.

mnajdova

We are ready to go! 🎉 🚀

Co-authored-by: Michał Dudak <michal@mui.com>

oliviertassinari · 2024-01-11T00:32:35Z

"preinstall": "npx only-allow pnpm",

Awesome, love to see this.

oliviertassinari · 2024-01-15T00:17:34Z

I think I found a regression with pnpm:

material-ui/packages/mui-base/package.json

Lines 43 to 47 in 9e0af1b

    
           "dependencies": { 
        
             "@babel/runtime": "^7.23.7", 
        
             "@floating-ui/react-dom": "^2.0.5", 
        
             "@mui/types": "workspace:^", 
        
             "@mui/utils": "workspace:^",

As far as I can read this: we can no longer control the minimum version dependency, we are forced to force every developer to update all their dependencies even if there are no breaking changes or reliance on new features, leading to a higher chance of package duplication in the bundle (good for our npm downloads stats, not good for end-users)

michaldudak · 2024-01-15T08:47:59Z

How is this different from the workflow we had with Yarn? We used to update the versions of internal dependencies each release to the most current one. Now it's done automatically by pnpm.

Anyway, if you want to allow earlier versions, pnpm does allow to do it (for example workspace:^5.0.0)

oliviertassinari · 2024-01-16T01:05:40Z

We used to update the versions of internal dependencies each release to the most current one.

@michaldudak It's simpler during the release process, and a lot less likely to do it wrong. But in theory, if we aren't dependent on the new feature, it's not necessary to increase the range.

Anyway, if you want to allow earlier versions, pnpm does allow to do it (for example workspace:^5.0.0)

Great to know 👍. So if we ever want to change the tradeoff, it looks like we are good.

Janpot · 2024-01-16T05:13:50Z

it's not necessary to increase the range.

I thought we kept versioning in lerna, if lerna wasn't increasing the range before pnpm migration, it won't increase it after. Or am I missing something?

oliviertassinari · 2024-01-16T20:56:11Z

I thought we kept versioning in lerna, if lerna wasn't increasing the range before pnpm migration, it won't increase it after. Or am I missing something?

@Janpot This indeed didn't change. The conclusion for me is:

[core] Use Lerna to publish #23793 came with the cons of making it hard to have optimal scope in the dependencies. It's not great, but the alternative is worse. For external dependencies, we still manage this by hand. We only bump the scope when we know we need it.
pnpm makes it harder for developers who check the source to know what's the actual version dependency (there is an indirection) but make it clear that it's from the same workspace.

Janpot · 2024-01-17T08:58:29Z

The benefit of the workspace:* version range is that it 100% guarantees that they are locally linked. pnpm will never download them from the registry. This is something that could happen in the previous setup if the versions don't match up for some reason. During publishing pnpm does all the work filling in the correct versions.

pnpm makes it harder for developers who check the source to know what's the actual version dependency (there is an indirection) but make it clear that it's from the same workspace.

Those people should instead run

npm info @mui/material
# or
pnpm info @mui/material
# or
yarn info @mui/material

It's a much more reliable source for this information as it shows the actually published package. It's also much easier to find the dependencies of a specific version of a package by adding a version range pnpm info @mui/material@5.14.8.

oliviertassinari · 2024-03-02T16:37:03Z

I was curious about the CI metric impact of this PR:

CI cost: https://docs.google.com/spreadsheets/d/1VQFM0bcX1oFrvt6tKjdqQDi5KJac_-HZcNavfvFk2tQ/edit#gid=0. I didn't see much conclusive evidence, maybe a bit more expensive but it is unclear.
CI speed: https://app.circleci.com/insights/github/mui/material-ui/workflows/pipeline/overview?reporting-window=last-90-days median duration seems also about the same.

Janpot added 2 commits February 21, 2023 19:13

Quick tryout of pnpm

ea4bfea

Update README.md

f8fe2bf

oliviertassinari added the scope: code-infra Specific to the core-infra product label Feb 21, 2023

github-actions bot added the PR: out-of-date The pull request has merge conflicts and can't be merged label Feb 22, 2023

michaldudak added 3 commits August 9, 2023 11:10

Merge remote-tracking branch 'upstream/master' into pnpm

3c28863

Fix deduplicate, propTypes, and docs:api

583df9b

Fixing pnpm typescript

9549772

github-actions bot removed the PR: out-of-date The pull request has merge conflicts and can't be merged label Aug 9, 2023

michaldudak reviewed Aug 9, 2023

View reviewed changes

packages/mui-utils/package.json Outdated Show resolved Hide resolved

Making the build pass

8f28f6e

github-actions bot added the PR: out-of-date The pull request has merge conflicts and can't be merged label Aug 10, 2023

michaldudak added 15 commits August 10, 2023 16:50

Merge remote-tracking branch 'upstream/master' into pnpm

05064f5

Add TS alias for mui-utils

8e022e1

Fix the utils self-referencing problem

8873566

Fix build:types

9a93b5c

Fix rollup build

ad76fbf

Fix prettier

5c20665

Fix eslint

4dd963d

Set the @mui/utils path resolution

4e7d972

Add tsignore to unused react imports

2c29122

Add missing dependencies

b5539ff

Remove ignore-engines

f2e6126

Update CircleCI steps

78f53c3

Update lockfile

a5bac39

Merge remote-tracking branch 'upstream/master' into pnpm

99ba2e2

sudo

53c33c4

github-actions bot removed the PR: out-of-date The pull request has merge conflicts and can't be merged label Aug 23, 2023

Update lockfile

b99d1a2

github-actions bot added the PR: out-of-date The pull request has merge conflicts and can't be merged label Jan 2, 2024

Merge remote-tracking branch 'upstream/master' into pnpm

0caa58e

michaldudak force-pushed the pnpm branch from fab2bbf to 0caa58e Compare January 2, 2024 14:33

github-actions bot removed the PR: out-of-date The pull request has merge conflicts and can't be merged label Jan 2, 2024

Add missing package

79adb48

mnajdova reviewed Jan 3, 2024

View reviewed changes

Pin next version to avoid mismatched types

ce762d3

mnajdova approved these changes Jan 3, 2024

View reviewed changes

github-actions bot added the PR: out-of-date The pull request has merge conflicts and can't be merged label Jan 3, 2024

Merge remote-tracking branch 'upstream/master' into pnpm

0c2bcdd

github-actions bot removed the PR: out-of-date The pull request has merge conflicts and can't be merged label Jan 3, 2024

michaldudak added 2 commits January 3, 2024 21:02

proptypes

bbeed4b

Use pnpm -r to build TS types on CI

d7a2af8

michaldudak force-pushed the pnpm branch from 69ff713 to d7a2af8 Compare January 3, 2024 20:46

michaldudak merged commit a1263e3 into mui:master Jan 3, 2024
22 checks passed

oliviertassinari mentioned this pull request Jan 6, 2024

[core] Migrate from yarn to pnpm mui/tech-challenge-design-engineer#10

Merged

mnajdova pushed a commit to mnajdova/material-ui that referenced this pull request Jan 9, 2024

[core] Change package manager to pnpm (mui#36287)

08d1315

Co-authored-by: Michał Dudak <michal@mui.com>

Janpot deleted the pnpm branch January 11, 2024 08:10

oliviertassinari mentioned this pull request Jan 13, 2024

docs: add Material UI to the showcase pnpm/pnpm.io#499

Merged

LukasTy mentioned this pull request Jan 31, 2024

[code-infra] Change package manager to pnpm mui/mui-x#11875

Merged

11 tasks

oliviertassinari mentioned this pull request Feb 4, 2024

[tools] Move to pnpm mui/mui-public#146

Merged

ZeeshanTamboli mentioned this pull request Feb 21, 2024

[core][base-ui] Remove @mui/base dev dependency from Base UI workspace #41216

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[core] Change package manager to pnpm #36287

[core] Change package manager to pnpm #36287

Janpot commented Feb 21, 2023 •

edited by michaldudak

oliviertassinari commented Feb 21, 2023 •

edited

Janpot commented Feb 21, 2023

mnajdova left a comment

mnajdova Jan 3, 2024

michaldudak Jan 3, 2024

mnajdova Jan 3, 2024

michaldudak Jan 3, 2024

mnajdova Jan 3, 2024

mnajdova Jan 3, 2024

michaldudak Jan 3, 2024

mnajdova Jan 3, 2024

mnajdova Jan 3, 2024

mnajdova Jan 3, 2024

michaldudak Jan 3, 2024

mnajdova left a comment

oliviertassinari commented Jan 11, 2024

oliviertassinari commented Jan 15, 2024 •

edited

michaldudak commented Jan 15, 2024

oliviertassinari commented Jan 16, 2024 •

edited

Janpot commented Jan 16, 2024

oliviertassinari commented Jan 16, 2024 •

edited

Janpot commented Jan 17, 2024 •

edited

oliviertassinari commented Mar 2, 2024 •

edited

[core] Change package manager to pnpm #36287

[core] Change package manager to pnpm #36287

Conversation

Janpot commented Feb 21, 2023 • edited by michaldudak

oliviertassinari commented Feb 21, 2023 • edited

Janpot commented Feb 21, 2023

mnajdova left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mnajdova left a comment

Choose a reason for hiding this comment

oliviertassinari commented Jan 11, 2024

oliviertassinari commented Jan 15, 2024 • edited

michaldudak commented Jan 15, 2024

oliviertassinari commented Jan 16, 2024 • edited

Janpot commented Jan 16, 2024

oliviertassinari commented Jan 16, 2024 • edited

Janpot commented Jan 17, 2024 • edited

oliviertassinari commented Mar 2, 2024 • edited

Janpot commented Feb 21, 2023 •

edited by michaldudak

oliviertassinari commented Feb 21, 2023 •

edited

oliviertassinari commented Jan 15, 2024 •

edited

oliviertassinari commented Jan 16, 2024 •

edited

oliviertassinari commented Jan 16, 2024 •

edited

Janpot commented Jan 17, 2024 •

edited

oliviertassinari commented Mar 2, 2024 •

edited