fix #2685: preferUnplugged: true in all packages

CHANGELOG.md

@@ -43,6 +43,12 @@
 
     An aside: You may be wondering why esbuild is trying to escape the `e` at all and why it doesn't just pass through the original source code unmodified. The reason why esbuild does this is that, for robustness, esbuild's AST generally tries to omit semantically-unrelated information and esbuild's code printers always try to preserve the semantics of the underlying AST. That way the rest of esbuild's internals can just deal with semantics instead of presentation. They don't have to think about how the AST will be printed when changing the AST. This is the same reason that esbuild's JavaScript AST doesn't have a "parentheses" node (e.g. `a * (b + c)` is represented by the AST `multiply(a, add(b, c))` instead of `multiply(a, parentheses(add(b, c)))`). Instead, the printer automatically inserts parentheses as necessary to maintain the semantics of the AST, which means all of the optimizations that run over the AST don't have to worry about keeping the parentheses up to date. Similarly, the CSS AST for the dimension token stores the actual unit and the printer makes sure the unit is properly escaped depending on what value it's placed after. All of the other code operating on CSS ASTs doesn't have to worry about parsing escapes to compare units or about keeping escapes up to date when the AST is modified. Hopefully that makes sense.
 
+* Attempt to avoid creating the `node_modules/.cache` directory for people that use Yarn 2+ in Plug'n'Play mode ([#2685](https://github.com/evanw/esbuild/issues/2685))
+
+    When Yarn's PnP mode is enabled, packages installed by Yarn may or may not be put inside `.zip` files. The specific heuristics for when this happens change over time in between Yarn versions. This is problematic for esbuild because esbuild's JavaScript package needs to execute a binary file inside the package. Yarn makes extensive modifications to Node's file system APIs at run time to pretend that `.zip` files are normal directories and to make it hard to tell whether a file is real or not (since in theory it doesn't matter). But they haven't modified Node's `child_process.execFileSync` API so attempting to execute a file inside a zip file fails. To get around this, esbuild previously used Node's file system APIs to copy the binary executable to another location before invoking `execFileSync`. Under the hood this caused Yarn to extract the file from the zip file into a real file that can then be run.
+
+    However, esbuild copied its executable into `node_modules/.cache/esbuild`. This is the [official recommendation from the Yarn team](https://yarnpkg.com/advanced/rulebook/#packages-should-never-write-inside-their-own-folder-outside-of-postinstall) for where packages are supposed to put these types of files when Yarn PnP is being used. However, users of Yarn PnP with esbuild find this really annoying because they don't like looking at the `node_modules` directory. With this release, esbuild now sets `"preferUnplugged": true` in its `package.json` files, which tells newer versions of Yarn to not put esbuild's packages in a zip file. There may exist older versions of Yarn that don't support `preferUnplugged`. In that case esbuild should still copy the executable to a cache directory, so it should still run (hopefully, since I haven't tested this myself). Note that esbuild setting `"preferUnplugged": true` may have the side effect of esbuild taking up more space on the file system in the event that multiple platforms are installed simultaneously, or that you're using an older version of Yarn that always installs packages for all platforms. In that case you may want to update to a newer version of Yarn since Yarn has recently changed to only install packages for the current platform.
+
 ## 0.15.14
 
 * Fix parsing of TypeScript `infer` inside a conditional `extends` ([#2675](https://github.com/evanw/esbuild/issues/2675))

lib/npm/node-platform.ts

@@ -210,48 +210,66 @@ for your current platform.`);
     }
   }
 
-  // The esbuild binary executable can't be used in Yarn 2 in PnP mode because
-  // it's inside a virtual file system and the OS needs it in the real file
-  // system. So we need to copy the file out of the virtual file system into
-  // the real file system.
+  // This code below guards against the unlikely case that the user is using
+  // Yarn 2+ in PnP mode and that version is old enough that it doesn't support
+  // the "preferUnplugged" setting. If that's the case, then the path to the
+  // binary executable that we got above isn't actually a real path. Instead
+  // it's a path to a zip file with some extra stuff appended to it.
   //
-  // You might think that we could use "preferUnplugged: true" in each of the
-  // platform-specific packages for this instead, since that tells Yarn to not
-  // use the virtual file system for those packages. This is not done because:
+  // Yarn's PnP mode tries hard to patch Node's file system APIs to pretend
+  // that these fake paths are real. So we can't check whether it's a real file
+  // or not by using Node's file system APIs (e.g. "fs.existsSync") because
+  // they have been patched to lie. But we can't return this fake path because
+  // Yarn hasn't patched "child_process.execFileSync" to work with fake paths,
+  // so attempting to execute the binary will fail.
   //
-  // * Really early versions of Yarn don't support "preferUnplugged", so package
-  //   installation would break on those Yarn versions if we did this.
+  // As a hack, we use Node's file system APIs to copy the file from the fake
+  // path to a real path. This will cause Yarn's hacked file system to extract
+  // the binary executable from the zip file and turn it into a real file that
+  // we can execute.
   //
-  // * Earlier Yarn versions always installed all optional dependencies for all
-  //   platforms even though most of them are incompatible. To minimize file
-  //   system space, we want these useless packages to take up as little space
-  //   as possible so they should remain unzipped inside their ".zip" files.
+  // This is only done when both ".zip/" is present in the path and the
+  // "pnpapi" package is present, which is a strong indication that Yarn PnP is
+  // being used. There is no API at all for telling whether something is a real
+  // file or not as far as I can tell. Even Yarn's own code just checks for
+  // whether ".zip/" is present in the path or not.
   //
-  //   We have to explicitly pass "preferUnplugged: false" instead of leaving
-  //   it up to Yarn's default behavior because Yarn's heuristics otherwise
-  //   automatically unzip packages containing ".exe" files, and we don't want
-  //   our Windows-specific packages to be unzipped either.
-  //
-  let pnpapi: any;
-  try {
-    pnpapi = require('pnpapi');
-  } catch (e) {
-  }
-  if (pnpapi) {
-    const root = pnpapi.getPackageInformation(pnpapi.topLevel).packageLocation;
-    const binTargetPath = path.join(
-      root,
-      'node_modules',
-      '.cache',
-      'esbuild',
-      `pnpapi-${pkg}-${ESBUILD_VERSION}-${path.basename(subpath)}`,
-    );
-    if (!fs.existsSync(binTargetPath)) {
-      fs.mkdirSync(path.dirname(binTargetPath), { recursive: true });
-      fs.copyFileSync(binPath, binTargetPath);
-      fs.chmodSync(binTargetPath, 0o755);
+  // Note to self: One very hacky way to tell if a path is under the influence
+  // of Yarn's file system hacks is to stat the file and check for the "crc"
+  // property in the result. If that's present, then the file is inside a zip
+  // file. However, I haven't done that here because it's not intended to be
+  // used that way. Who knows what Yarn versions it does or does not work on
+  // (including future versions).
+  if (/\.zip\//.test(binPath)) {
+    let pnpapi: any;
+    try {
+      pnpapi = require('pnpapi');
+    } catch (e) {
+    }
+    if (pnpapi) {
+      // Copy the executable to ".cache/esbuild". The official recommendation
+      // of the Yarn team is to use the directory "node_modules/.cache/esbuild":
+      // https://yarnpkg.com/advanced/rulebook/#packages-should-never-write-inside-their-own-folder-outside-of-postinstall
+      // People that use Yarn in PnP mode find this really annoying because they
+      // don't like seeing the "node_modules" directory. These people should
+      // either work with Yarn to change their recommendation, or upgrade their
+      // version of Yarn, since newer versions of Yarn shouldn't stick esbuild's
+      // binary executables in a zip file due to the "preferUnplugged" setting.
+      const root = pnpapi.getPackageInformation(pnpapi.topLevel).packageLocation;
+      const binTargetPath = path.join(
+        root,
+        'node_modules',
+        '.cache',
+        'esbuild',
+        `pnpapi-${pkg}-${ESBUILD_VERSION}-${path.basename(subpath)}`,
+      );
+      if (!fs.existsSync(binTargetPath)) {
+        fs.mkdirSync(path.dirname(binTargetPath), { recursive: true });
+        fs.copyFileSync(binPath, binTargetPath);
+        fs.chmodSync(binTargetPath, 0o755);
+      }
+      return { binPath: binTargetPath, isWASM };
     }
-    return { binPath: binTargetPath, isWASM };
   }
 
   return { binPath, isWASM };

npm/@esbuild/android-arm/package.json

@@ -4,7 +4,7 @@
   "description": "A WebAssembly shim for esbuild on Android ARM.",
   "repository": "https://github.com/evanw/esbuild",
   "license": "MIT",
-  "preferUnplugged": false,
+  "preferUnplugged": true,
   "engines": {
     "node": ">=12"
   },

npm/@esbuild/linux-loong64/package.json

@@ -4,7 +4,7 @@
   "description": "The Linux LoongArch 64-bit binary for esbuild, a JavaScript bundler.",
   "repository": "https://github.com/evanw/esbuild",
   "license": "MIT",
-  "preferUnplugged": false,
+  "preferUnplugged": true,
   "engines": {
     "node": ">=12"
   },

npm/esbuild-android-64/package.json

@@ -4,7 +4,7 @@
   "description": "A WebAssembly shim for esbuild on Android x64.",
   "repository": "https://github.com/evanw/esbuild",
   "license": "MIT",
-  "preferUnplugged": false,
+  "preferUnplugged": true,
   "engines": {
     "node": ">=12"
   },

npm/esbuild-android-arm64/package.json

@@ -4,7 +4,7 @@
   "description": "The Android ARM 64-bit binary for esbuild, a JavaScript bundler.",
   "repository": "https://github.com/evanw/esbuild",
   "license": "MIT",
-  "preferUnplugged": false,
+  "preferUnplugged": true,
   "engines": {
     "node": ">=12"
   },

npm/esbuild-darwin-64/package.json

@@ -4,7 +4,7 @@
   "description": "The macOS 64-bit binary for esbuild, a JavaScript bundler.",
   "repository": "https://github.com/evanw/esbuild",
   "license": "MIT",
-  "preferUnplugged": false,
+  "preferUnplugged": true,
   "engines": {
     "node": ">=12"
   },

npm/esbuild-darwin-arm64/package.json

@@ -4,7 +4,7 @@
   "description": "The macOS ARM 64-bit binary for esbuild, a JavaScript bundler.",
   "repository": "https://github.com/evanw/esbuild",
   "license": "MIT",
-  "preferUnplugged": false,
+  "preferUnplugged": true,
   "engines": {
     "node": ">=12"
   },

npm/esbuild-freebsd-64/package.json

@@ -4,7 +4,7 @@
   "description": "The FreeBSD 64-bit binary for esbuild, a JavaScript bundler.",
   "repository": "https://github.com/evanw/esbuild",
   "license": "MIT",
-  "preferUnplugged": false,
+  "preferUnplugged": true,
   "engines": {
     "node": ">=12"
   },

npm/esbuild-freebsd-arm64/package.json

@@ -4,7 +4,7 @@
   "description": "The FreeBSD ARM 64-bit binary for esbuild, a JavaScript bundler.",
   "repository": "https://github.com/evanw/esbuild",
   "license": "MIT",
-  "preferUnplugged": false,
+  "preferUnplugged": true,
   "engines": {
     "node": ">=12"
   },

npm/esbuild-linux-32/package.json

@@ -4,7 +4,7 @@
   "description": "The Linux 32-bit binary for esbuild, a JavaScript bundler.",
   "repository": "https://github.com/evanw/esbuild",
   "license": "MIT",
-  "preferUnplugged": false,
+  "preferUnplugged": true,
   "engines": {
     "node": ">=12"
   },

npm/esbuild-linux-64/package.json

@@ -4,7 +4,7 @@
   "description": "The Linux 64-bit binary for esbuild, a JavaScript bundler.",
   "repository": "https://github.com/evanw/esbuild",
   "license": "MIT",
-  "preferUnplugged": false,
+  "preferUnplugged": true,
   "engines": {
     "node": ">=12"
   },

npm/esbuild-linux-arm/package.json

@@ -4,7 +4,7 @@
   "description": "The Linux ARM binary for esbuild, a JavaScript bundler.",
   "repository": "https://github.com/evanw/esbuild",
   "license": "MIT",
-  "preferUnplugged": false,
+  "preferUnplugged": true,
   "engines": {
     "node": ">=12"
   },

npm/esbuild-linux-arm64/package.json

@@ -4,7 +4,7 @@
   "description": "The Linux ARM 64-bit binary for esbuild, a JavaScript bundler.",
   "repository": "https://github.com/evanw/esbuild",
   "license": "MIT",
-  "preferUnplugged": false,
+  "preferUnplugged": true,
   "engines": {
     "node": ">=12"
   },

npm/esbuild-linux-mips64le/package.json

@@ -4,7 +4,7 @@
   "description": "The Linux MIPS 64-bit Little Endian binary for esbuild, a JavaScript bundler.",
   "repository": "https://github.com/evanw/esbuild",
   "license": "MIT",
-  "preferUnplugged": false,
+  "preferUnplugged": true,
   "engines": {
     "node": ">=12"
   },

npm/esbuild-linux-ppc64le/package.json

@@ -4,7 +4,7 @@
   "description": "The Linux PowerPC 64-bit Little Endian binary for esbuild, a JavaScript bundler.",
   "repository": "https://github.com/evanw/esbuild",
   "license": "MIT",
-  "preferUnplugged": false,
+  "preferUnplugged": true,
   "engines": {
     "node": ">=12"
   },

npm/esbuild-linux-riscv64/package.json

@@ -4,7 +4,7 @@
   "description": "The Linux RISC-V 64-bit binary for esbuild, a JavaScript bundler.",
   "repository": "https://github.com/evanw/esbuild",
   "license": "MIT",
-  "preferUnplugged": false,
+  "preferUnplugged": true,
   "engines": {
     "node": ">=12"
   },

npm/esbuild-linux-s390x/package.json

@@ -4,7 +4,7 @@
   "description": "The Linux IBM Z 64-bit Big Endian binary for esbuild, a JavaScript bundler.",
   "repository": "https://github.com/evanw/esbuild",
   "license": "MIT",
-  "preferUnplugged": false,
+  "preferUnplugged": true,
   "engines": {
     "node": ">=12"
   },

npm/esbuild-openbsd-64/package.json

@@ -4,7 +4,7 @@
   "description": "The OpenBSD 64-bit binary for esbuild, a JavaScript bundler.",
   "repository": "https://github.com/evanw/esbuild",
   "license": "MIT",
-  "preferUnplugged": false,
+  "preferUnplugged": true,
   "engines": {
     "node": ">=12"
   },

npm/esbuild-sunos-64/package.json

@@ -4,7 +4,7 @@
   "description": "The illumos 64-bit binary for esbuild, a JavaScript bundler.",
   "repository": "https://github.com/evanw/esbuild",
   "license": "MIT",
-  "preferUnplugged": false,
+  "preferUnplugged": true,
   "engines": {
     "node": ">=12"
   },

npm/esbuild-windows-32/package.json

@@ -4,7 +4,7 @@
   "description": "The Windows 32-bit binary for esbuild, a JavaScript bundler.",
   "repository": "https://github.com/evanw/esbuild",
   "license": "MIT",
-  "preferUnplugged": false,
+  "preferUnplugged": true,
   "engines": {
     "node": ">=12"
   },

npm/esbuild-windows-64/package.json

@@ -4,7 +4,7 @@
   "description": "The Windows 64-bit binary for esbuild, a JavaScript bundler.",
   "repository": "https://github.com/evanw/esbuild",
   "license": "MIT",
-  "preferUnplugged": false,
+  "preferUnplugged": true,
   "engines": {
     "node": ">=12"
   },

npm/esbuild-windows-arm64/package.json

@@ -4,7 +4,7 @@
   "description": "The Windows ARM 64-bit binary for esbuild, a JavaScript bundler.",
   "repository": "https://github.com/evanw/esbuild",
   "license": "MIT",
-  "preferUnplugged": false,
+  "preferUnplugged": true,
   "engines": {
     "node": ">=12"
   },