fix: support subspace in package-extractor #4655

EscapeB · 2024-04-17T10:43:33Z

Summary

Fix issue that can't extract projects dependencies correctly if there exist multiple subspaces.

Details

Now only subspace that for entry project will be passed to package extractor, but if there are multiple projects belongs to different subspaces, dependencies in other subspaces can't be extract correctly.

How it was tested

Tested with in-house production project.

Also there is an example that shows what the file structure will be like after this PR merged.
Here is the example subspace repo https://github.com/william2958/rush-subspace-demo

Impacted documentation

The internal API for PackageExtractor has been changed to support multiple subspaces

jzhang026 · 2024-04-17T13:06:38Z

libraries/rush-lib/src/cli/actions/DeployAction.ts

+    const queue: RushConfigurationProject[] = [project];
+
+    while (queue.length > 0) {
+      const _project: RushConfigurationProject = queue.shift()!;


did we forget to fill 'projects' in this function?

Yes, deleted by mistake...

chengcyber · 2024-04-17T18:36:34Z

libraries/package-extractor/src/PackageExtractor.ts

@@ -483,8 +499,13 @@ export class PackageExtractor {
          path.join(packageJsonRealFolderPath, 'package.json')
        );

+        const targetSubspace: IExtractorSubspace | undefined = subspaces?.find(
+          (p) => p.pnpmInstallFolder && Path.isUnder(packageJsonFolderPath, p.pnpmInstallFolder)


Just curious: would it be possible to have multiple "subspaces" here?

It's related if there are any nested projects in a monorepo (which is not recommended)

I don't think so, because each subspace's pnpmInstallFolder will be located at common/temp and they won't be nested right?

We can't have multiple subspaces in following logic, or it will have some issues

One dependency from one subspace might be resolved to another subspace, which is not expected

chengcyber · 2024-04-17T18:37:53Z

libraries/package-extractor/src/PackageExtractor.ts

          try {
            // The PNPM virtual store links are created in this folder.  We will resolve the current package
            // from that location and collect any additional links encountered along the way.
            // TODO: This can be configured via NPMRC. We should support that.
-            const pnpmDotFolderPath: string = path.join(pnpmInstallFolder, 'node_modules', '.pnpm');
+            const pnpmDotFolderPath: string = path.join(
+              targetSubspace.pnpmInstallFolder!,


Could we remove the "!" sign here? Instead, adding it to if-condition

if (targetSubspace && targetSubspace.pnpmInstallFolder) {

chengcyber · 2024-04-17T18:40:14Z

libraries/package-extractor/src/PackageExtractor.ts

   */
-  pnpmInstallFolder?: string;
+  subspaces?: IExtractorSubspace[];


NOTE: It's a breaking change to remove "pnpmInstallFolder" and entirely replace it with "subspaces". If we do so, it's a major version bump.

Agreed, to avoid breaking API compatibility we could allow both alternatives: Either you specify IExtractorState.transformPackageJson etc, or else you can specify IExtractorState.subspaces (but not both at the same time).

OK, I will rethink how can we make API more compatible here.

chengcyber · 2024-04-17T18:47:38Z

libraries/rush-lib/src/cli/actions/DeployAction.ts

+    const queue: RushConfigurationProject[] = [project];
+
+    while (queue.length > 0) {
+      const _project: RushConfigurationProject = queue.shift()!;


chengcyber · 2024-04-17T18:48:50Z

libraries/rush-lib/src/cli/actions/DeployAction.ts

+      const _project: RushConfigurationProject = queue.shift()!;
+
+      for (const dependency of _project.dependencyProjects) {
+        queue.push(dependency);


I wonder visited projects should be ignored.

To illustrate:

const visited: Set<RushConfigurationProject> = new Set(); ... for (const dependency of _project.dependencyProjects) { if (visited.has(dependency) { continue; } ... visited.add(dependency); }

Yes, it would have some performance improvement.

chengcyber · 2024-04-17T18:52:10Z

libraries/rush-lib/src/cli/actions/DeployAction.ts

+        if (!scenarioConfiguration.json.omitPnpmWorkaroundLinks) {
+          subspace.pnpmInstallFolder = project.subspace.getSubspaceTempFolder();
+        }
+        subspaces.push(subspace);


Assuming A, B lives in the same subspace. It feels like the current logic generates the same subspace twice. Am I understanding this correctly?

Nice catch, I might change it to a Set to avoid that case.

octogonz · 2024-04-17T20:41:57Z

How it was tested

Tested with in-house production project.

We don't need an official build test necessarily, but I'd really like to at least see a branch that illustrates how this algorithm is used. If you can't easily reproduce the error being fixed, that's okay, but it would be very helpful to have an example rush install folder that we can run the new algorithm against to see what the deployed output looks like in the context of subspaces.

@william2958 @EscapeB

octogonz · 2024-04-17T20:44:28Z

libraries/rush-lib/src/cli/actions/DeployAction.ts

+
+    while (queue.length > 0) {
+      const _project: RushConfigurationProject = queue.shift()!;
+


shift() is an O(n) operation making this loop O(n^2). If N=1,000 then N^2=1,000,000 which might be significant

Nice catch, I think I can simple convert it to be a DFS by replacing shift with pop.

Sounds good

EscapeB · 2024-04-18T04:34:46Z

How it was tested

Tested with in-house production project.

We don't need an official build test necessarily, but I'd really like to at least see a branch that illustrates how this algorithm is used. If you can't easily reproduce the error being fixed, that's okay, but it would be very helpful to have an example rush install folder that we can run the new algorithm against to see what the deployed output looks like in the context of subspaces.

@william2958 @EscapeB

@octogonz I have added the example rush repo and a screenshot of deployment output.

octogonz · 2024-04-18T05:36:29Z

How it was tested

Tested with in-house production project.

We don't need an official build test necessarily, but I'd really like to at least see a branch that illustrates how this algorithm is used. If you can't easily reproduce the error being fixed, that's okay, but it would be very helpful to have an example rush install folder that we can run the new algorithm against to see what the deployed output looks like in the context of subspaces.
@william2958 @EscapeB

@octogonz I have added the example rush repo and a screenshot of deployment output.

William also created a test branch for us here:
https://github.com/william2958/rush-subspace-demo

I had a long day today at the conference, so I will test it first thing tomorrow morning. Mainly I want to inspect the symlinks created by rush deploy to confirm that they look correct and aren't including extraneous files. (It's hard to judge from only looking at the code.)

EscapeB · 2024-04-18T07:13:38Z

How it was tested

Tested with in-house production project.

We don't need an official build test necessarily, but I'd really like to at least see a branch that illustrates how this algorithm is used. If you can't easily reproduce the error being fixed, that's okay, but it would be very helpful to have an example rush install folder that we can run the new algorithm against to see what the deployed output looks like in the context of subspaces.
@william2958 @EscapeB

@octogonz I have added the example rush repo and a screenshot of deployment output.

William also created a test branch for us here: https://github.com/william2958/rush-subspace-demo

I had a long day today at the conference, so I will test it first thing tomorrow morning. Mainly I want to inspect the symlinks created by rush deploy to confirm that they look correct and aren't including extraneous files. (It's hard to judge from only looking at the code.)

Yep, I'm also using the repo William created, I put its link in this PR description.

octogonz · 2024-04-18T17:53:19Z

libraries/package-extractor/src/PackageExtractor.ts

@@ -483,8 +499,13 @@ export class PackageExtractor {
          path.join(packageJsonRealFolderPath, 'package.json')
        );

+        const targetSubspace: IExtractorSubspace | undefined = subspaces?.find(


This find() is calling Path.isUnder() for every subspace (including paths that don't match), for every node_modules package that gets crawled. If we have n=1000 subspaces and n=1000 installed folders, that is O(n*m)=O(1,000,000) operations.

Here's a couple examples of how Rush handles these problems:

LookupByPath.ts constructs an efficient lookup tree; OR

PackageJsonLookup simply caches the results.

@EscapeB @william2958 Because PR #4655 is an important fix, we don't need to address the problem in this PR. But I think we should consider optimizing these lookups because O(n^2) loops can and do cause noticeable slowness.

OK，will try to improve and that in later PR.

octogonz · 2024-04-18T18:14:00Z

libraries/package-extractor/src/PackageExtractor.ts

        // Transform packageJson using the provided transformer, if requested
-        const packageJson: IPackageJson = transformPackageJson?.(originalPackageJson) ?? originalPackageJson;
+        const packageJson: IPackageJson =
+          targetSubspace?.transformPackageJson?.(originalPackageJson) ?? originalPackageJson;


This API design is confusing for users because it ignores mistakes. In the below example, what is the pnpmInstallFolder for the default subspace? It is unclear.

this.extractAsync({ pnpmInstallFolder: 'abc', subspaces: [ { subspaceName: 'default', pnpmInstallFolder: 'def' }, { subspaceName:'other', pnpmInstallFolder: 'ghi' } ] })

(I'm working on a fix)

D4N14L · 2024-04-18T19:44:13Z

common/reviews/api/package-extractor.api.md

@@ -36,6 +36,7 @@ export interface IExtractorOptions {
    pnpmInstallFolder?: string;
    projectConfigurations: IExtractorProjectConfiguration[];
    sourceRootFolder: string;
+    subspaces?: IExtractorSubspace[];
    targetRootFolder: string;
    terminal: ITerminal;
    transformPackageJson?: (packageJson: IPackageJson) => IPackageJson | undefined;


Suggested change

transformPackageJson?: (packageJson: IPackageJson) => IPackageJson | undefined;

transformPackageJson?: (packageJson: IPackageJson) => IPackageJson;

I think this was just typed incorrectly. I don't see any usage of this transform that could reasonably need the method to return "undefined".

Agreed. I checked and PNPM itself seems to reject that case:

if (pnpmfile?.hooks?.readPackage) { const readPackage = pnpmfile.hooks.readPackage; pnpmfile.hooks.readPackage = async function(pkg, ...args2) { pkg.dependencies = pkg.dependencies ?? {}; pkg.devDependencies = pkg.devDependencies ?? {}; pkg.optionalDependencies = pkg.optionalDependencies ?? {}; pkg.peerDependencies = pkg.peerDependencies ?? {}; const newPkg = await readPackage(pkg, ...args2); if (!newPkg) { throw new BadReadPackageHookError(pnpmFilePath, "readPackage hook did not return a package manifest object."); }

D4N14L · 2024-04-18T19:45:06Z

libraries/package-extractor/src/PackageExtractor.ts

+   * Can't be used with subspaces, if subspaces field is specified, we will use pnpmInstallFolder configured
+   * for each subspace and ignore this.


Update this to throw if both are provided instead.

Fixed in my EscapeB#1

D4N14L · 2024-04-18T19:47:16Z

libraries/package-extractor/src/PackageExtractor.ts

+          targetSubspace?.transformPackageJson?.(originalPackageJson) ??
+          transformPackageJson?.(originalPackageJson) ??
+          originalPackageJson;


We should validate that only one of the two options (subspaces vs top-level transformPackageJson/pnpmInstallFolder) is provided, and throw an error if it's not the case. We wouldn't want to hide why the top-level ones are ignored when someone specified subspaces

Fixed in my EscapeB#1

fix: support subspace in package-extractor

a975822

EscapeB requested review from iclanton, octogonz, apostolisms, D4N14L and dmichon-msft as code owners April 17, 2024 10:43

EscapeB added 2 commits April 17, 2024 20:09

chore: api change

dde74bf

fix: fix ci lint error

bc108d9

jzhang026 reviewed Apr 17, 2024

View reviewed changes

chengcyber reviewed Apr 17, 2024

View reviewed changes

octogonz reviewed Apr 17, 2024

View reviewed changes

fix: cr fix

7228c25

EscapeB force-pushed the fix/support_subspace_in_package_extractor branch from 40edee8 to 7228c25 Compare April 18, 2024 04:37

Improve change logs

075b7ce

octogonz reviewed Apr 18, 2024

View reviewed changes

octogonz added 2 commits April 18, 2024 11:15

Update documentation

bf90994

Proposed API simplification

987c4ab

octogonz mentioned this pull request Apr 18, 2024

Suggested API improvement for PR #4655 EscapeB/rushstack#1

Merged

D4N14L reviewed Apr 18, 2024

View reviewed changes

PR feedback: readPackage() should not return undefined

e1b2856

octogonz merged commit 25b89ec into microsoft:main Apr 18, 2024
5 checks passed

octogonz mentioned this pull request Apr 25, 2024

fix: issue of rush deploy of project in a subspace #4650

Closed

octogonz mentioned this pull request May 8, 2024

[rush] additionalProjectsToInclude not working anymore at rush deploy #4683

Closed


		while (queue.length > 0) {
		const _project: RushConfigurationProject = queue.shift()!;

	transformPackageJson?: (packageJson: IPackageJson) => IPackageJson \| undefined;
	transformPackageJson?: (packageJson: IPackageJson) => IPackageJson;

		* Can't be used with subspaces, if subspaces field is specified, we will use pnpmInstallFolder configured
		* for each subspace and ignore this.

fix: support subspace in package-extractor #4655

fix: support subspace in package-extractor #4655

Conversation

EscapeB commented Apr 17, 2024 • edited Loading

Summary

Details

How it was tested

Impacted documentation

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

octogonz commented Apr 17, 2024 • edited Loading

How it was tested

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

EscapeB commented Apr 18, 2024

How it was tested

octogonz commented Apr 18, 2024 • edited Loading

How it was tested

EscapeB commented Apr 18, 2024

How it was tested

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

EscapeB commented Apr 17, 2024 •

edited

Loading

octogonz commented Apr 17, 2024 •

edited

Loading

octogonz commented Apr 18, 2024 •

edited

Loading