Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"The package "esbuild-linux-64" could not be found, and is needed by esbuild." #1819

Closed
andyburke opened this issue Dec 1, 2021 · 48 comments

Comments

@andyburke
Copy link

andyburke commented Dec 1, 2021

We are using SvelteKit, which uses Vite, which uses esbuild and we are seeing this error on some of our macs.

We also have devs on linux who are not having this issue.

I was able to find this, but it's closed with little to no explanation: vitejs/vite#5187

Could this be caused by installing things on a linux machine, package-lock.json gets some notions, and then it's a problem on other platforms? Is something else going on? Happy to be pointed in another direction if this isn't specifically an esbuild issue.

npm ls output from my linux machine indicates the following versions:

─┬ vite@2.6.14
  ├── esbuild@0.13.15 deduped
@evanw
Copy link
Owner

evanw commented Dec 1, 2021

Do you have an steps that can be followed to reproduce this issue?

@andyburke
Copy link
Author

Unfortunately, this is a difficult one to figure out a repro on.

I am suspicious it would read like:

  1. Install esbuild (or have esbuild installed as a dependency) while on a linux machine
  2. Try to npm install things on a mac
  3. Try to run esbuild on the mac and observe it complains about missing esbuild-linux-64

I can try to create a bare project on linux and point people at it, but I can't easily test this on a mac myself.

@andyburke
Copy link
Author

I created the most basic setup I could here: https://github.com/andyburke/esbuild-bug-linux-mac

I'll try to ask a colleague with a mac to pull and test it to see if they can repro the problem. If anyone here has a mac, maybe they could try as well.

@netaisllc
Copy link

Yeah, so pulling the most basic setup on a Mac and building the project fails to repro the issue.

The darwin version is installed as expected.

@evanw
Copy link
Owner

evanw commented Dec 3, 2021

Ok, good to know. Marking this as unactionable for now since there's nothing I can do about this issue at the moment. I am awaiting further input from someone who has a reproduction. It would still be good to get to the bottom of this as it may for example turn out to be a bug in npm that can be fixed. Once we have reproduction steps we can also see what happens with pnpm/yarn as well.

@xd-shadow
Copy link

Just installed Node in WSL and encountered the same problem
image

@evanw
Copy link
Owner

evanw commented Jan 6, 2022

Just saying +1 to this issue without providing specific reproduction instructions isn't adding any additional information. Please provide specific end-to-end instructions for how to reproduce the issue.

@silvenon
Copy link

silvenon commented Jan 9, 2022

I'm getting the same error when running esbuild on Vercel in a serverless function. In vercel/vercel#7287 I linked to my repository along with the deployment showing the error, but the complication is that the problem isn't happening locally, only in the deployment, so I'm not sure where it goes wrong and how to go about debugging it. It's probably a question for them, but I thought that cross-linking this issue couldn't hurt.

@geoffharcourt
Copy link

geoffharcourt commented Jan 9, 2022

I experienced this issue in some situations where we had a Yarn cache from gems installed on the wrong CPU architecture (Docker images built on a Mac with Apple Silicon then pushed up to run on Linux on x86). In every instance we encountered this, it was because we had shared node_modules between machines of different architectures (so had the wrong architecture-specific package installed). We're using Yarn and not NPM, so I'm not sure if the results will be different. When we ran yarn install from the proper architecture to start with we were unable to recreate this.

@andyburke
Copy link
Author

This could make sense, we run our containers on linux images, but for development we mount our local filesystems into the containers, sometimes those local filesystems are running on Mac, for instance.

Is there some way to have ESBuild install for multiple platforms somehow?

@zdenekgebauer
Copy link

zdenekgebauer commented Jan 20, 2022

I'm geting this error in linux based docker image. Even after npm i esbuild-linux-64

@evanw
Copy link
Owner

evanw commented Jan 21, 2022

I'm going to try improving the error message in the situation where you have installed esbuild for a different platform than the one you're currently on. The error message I added looks like this:

$ ./node_modules/.bin/esbuild
./node_modules/esbuild/bin/esbuild:106
          throw new Error(`
          ^

Error:
You installed esbuild on another platform than the one you're currently using.
This won't work because esbuild is written with native code and needs to
install a platform-specific binary executable.

Specifically the "esbuild-linux-arm64" package is present but this platform
needs the "esbuild-darwin-arm64" package instead. People often get into this
situation by installing esbuild on Windows or macOS and copying "node_modules"
into a Docker image that runs Linux, or by copying "node_modules" between
Windows and WSL environments.

If you are installing with npm, you can try not copying the "node_modules"
directory when you copy the files over, and running "npm ci" or "npm install"
on the destination platform after the copy. Or you could consider using yarn
instead which has built-in support for installing a package on multiple
platforms simultaneously.

If you are installing with yarn, you can try listing both this platform and the
other platform in your ".yarnrc.yml" file using the "supportedArchitectures"
feature: https://yarnpkg.com/configuration/yarnrc/#supportedArchitectures
Keep in mind that this means multiple copies of esbuild will be present.

Another alternative is to use the "esbuild-wasm" package instead, which works
the same way on all platforms. But it comes with a heavy performance cost and
can sometimes be 10x slower than the "esbuild" package, so you may also not
want to do that.

    at generateBinPath (./node_modules/esbuild/bin/esbuild:106:17)
    at Object.<anonymous> (./node_modules/esbuild/bin/esbuild:161:39)
    at Module._compile (node:internal/modules/cjs/loader:1101:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
    at node:internal/main/run_main_module:17:47

Edit: And it's also now documented on the website too: https://esbuild.github.io/getting-started/#simultaneous-platforms

@evanw
Copy link
Owner

evanw commented Jan 22, 2022

I'm going to close this issue because it has been marked as unactionable for a while and specific steps to reproduce the issue from scratch haven't been provided. This issue can definitely be reproduced by copying a node_modules folder from one OS to another without reinstalling, but your not supposed to do that. There should now be a specific error message when that happens with workarounds and workarounds are also described on the website. It's unclear if this is what's happening in all of these cases or if something is going on in other cases, but we won't know unless specific reproduction steps are provided for the other cases. I think these are probably best filed as new issues instead.

@colin-byrne-1
Copy link

colin-byrne-1 commented Mar 9, 2022

Just adding our experience here...

We have a shared volume containing the node_modules from our developer's mac machines to our numerous linux Docker containers. We can run the install from inside the containers to bypass this, but it's prohibitively slow due to the size of the containers compared to the host machine.

It runs fine on the few non-mac machines we have which are both running linux.

Unfortunately the outputted suggestion for the additions to .yarnrc.yml doesn't work either, as Yarn stubbornly will continue to detect the platform and not link the supportedArchitectures which are not in the current architecture/platform:

esbuild-linux-64@npm:0.14.14 The linux-x64 architecture is incompatible with this module, link skipped.

Yarn Classic used to have the option to pass the --ignore-engines flag which likely would have been a nice workaround, but in Yarn 2 this is deprecated, and I have not found another way to bypass the platform checks in Yarn 2.

@gotechnolabs
Copy link

gotechnolabs commented Mar 9, 2022

FWIW we had the same issue with yarn 2 version berry. Since with Yarn 2, you can use zero installs which makes you commit the .yarn/cache folder (which is similar to node_modules).

Turns out adding this to our .yarnrc.yml file solved it as described here (tailor to what architecture you want to support): https://yarnpkg.com/configuration/yarnrc#supportedArchitectures

supportedArchitectures:
  os:
    - "current"
    - "darwin"
    - "linux"
    - "win32"

  cpu:
    - "current"
    - "x86"
    - "x64"
    - "arm64"
    - "ia32"

@marina-mosti
Copy link

I'm having the same problem with our M1 macs.
Steps to reproduce are here: cypress-io/cypress#20768

We are not sharing node modules at all. Any chance we can give this a look over?

@evanw
Copy link
Owner

evanw commented Mar 28, 2022

One idea: are you perhaps running an x86-64 node executable? Apple lets you run x86-64 executables through Rosetta. If so, a fix for this could be to try installing the native version of node instead.

@marina-mosti
Copy link

Thank you @evanw , installing node through nvm instead of the installer from the site did the trick 👍🏻

@leonbloy
Copy link

Specifically the "esbuild-linux-arm64" package is present but this platform
needs the "esbuild-darwin-arm64" package instead. People often get into this
situation by installing esbuild on Windows or macOS and copying "node_modules"
into a Docker image that runs Linux, or by copying "node_modules" between
Windows and WSL environments.

If you are installing with npm, you can try not copying the "node_modules"
directory when you copy the files over, and running "npm ci" or "npm install"
on the destination platform after the copy.

That sounds rather wrong.

AFAIK no one in the world copies the "node_modules" directory (!!!)

What people often do (especially in CI/CD) is to rely on package_lock.json
(which is, in this scenario, commited to git) to avoid running "npm install" and
doing instead "npm ci" (the description above seem to confuse these two very different commands)

"npm ci" respects the concrete dependencies listed in package_lock.json, while "npm install" recomputes them.
The advantages of "npm ci" are speed and reproducibility.
And it's the normal way to do automated deploys in dockerized instances.

@mrmartineau
Copy link

FYI this fix solved issues that I had on Circle CI when using Yarn 2/berry PNP. thanks

@scottksmith95
Copy link

Just adding our experience here...

We have a shared volume containing the node_modules from our developer's mac machines to our numerous linux Docker containers. We can run the install from inside the containers to bypass this, but it's prohibitively slow due to the size of the containers compared to the host machine.

It runs fine on the few non-mac machines we have which are both running linux.

Unfortunately the outputted suggestion for the additions to .yarnrc.yml doesn't work either, as Yarn stubbornly will continue to detect the platform and not link the supportedArchitectures which are not in the current architecture/platform:

esbuild-linux-64@npm:0.14.14 The linux-x64 architecture is incompatible with this module, link skipped.

Yarn Classic used to have the option to pass the --ignore-engines flag which likely would have been a nice workaround, but in Yarn 2 this is deprecated, and I have not found another way to bypass the platform checks in Yarn 2.

@cobyrne09 This is the issue we are running into now. Did you ever find a solution?

@colin-byrne-1
Copy link

@cobyrne09 This is the issue we are running into now. Did you ever find a solution?

I copied the built module from inside the container into the .yarn directory then referenced it like so in my package.json.

 "esbuild": "0.14.14",
 "esbuild-linux-64": "file:./.yarn/esbuild-linux-64",

@AkshatParmar
Copy link

AkshatParmar commented Oct 12, 2022

Reinstalling the dependencies worked in my case. Specifically the vite node_modules. Also observed different outputs for different terminals. The VSCODE terminal yields the same error but the other one does not.
Screen Shot 2022-10-12 at 12 39 00 PM
Ran yarn in the source dir.

@rp-pedraza
Copy link

This also happens when yarn sets a different "node_modules" directory using .yarnrc. Esbuild looking up node_modules seems static.

@james-innes
Copy link

For anyone else like the OP who is using SvelteKit + Vite and also deploying on Vercel, I solved the following error simply by deleting package-lock.json and adding it to .gitignore

failed to load config from /vercel/path0/vite.config.ts
--
error during build:
Error: The package "@esbuild/linux-x64" could not be found, and is needed by esbuild.
 
If you are installing esbuild with npm, make sure that you don't specify the
"--no-optional" or "--omit=optional" flags. The "optionalDependencies" feature
of "package.json" is used by esbuild to install the correct binary executable
for your current platform.
at generateBinPath (/vercel/path0/node_modules/esbuild/lib/main.js:1840:15)
at esbuildCommandAndArgs (/vercel/path0/node_modules/esbuild/lib/main.js:1907:33)
at ensureServiceIsRunning (/vercel/path0/node_modules/esbuild/lib/main.js:2072:25)
at build (/vercel/path0/node_modules/esbuild/lib/main.js:1963:26)
at bundleConfigFile (file:///vercel/path0/node_modules/vite/dist/node/chunks/dep-6305614c.js:63019:26)
at loadConfigFromFile (file:///vercel/path0/node_modules/vite/dist/node/chunks/dep-6305614c.js:62995:31)
at resolveConfig (file:///vercel/path0/node_modules/vite/dist/node/chunks/dep-6305614c.js:62617:34)
at doBuild (file:///vercel/path0/node_modules/vite/dist/node/chunks/dep-6305614c.js:45474:26)
at build (file:///vercel/path0/node_modules/vite/dist/node/chunks/dep-6305614c.js:45463:22)
at CAC.<anonymous> (file:///vercel/path0/node_modules/vite/dist/node/cli.js:808:15)
Error: Command "npm run build" exited with 1

@foxxyz
Copy link

foxxyz commented Jan 17, 2023

@james-innes's workaround works great, but the real question is...

Why is esbuild adding the platform-specific package to package-lock.json? Isn't package-lock.json supposed to be version-controlled, and hence supposed to be platform-independent?

If I'm on MacOS, and my colleague is on Windows, we will be generating different package-lock.json files, and our CI system, which might be some flavor of Linux, won't run npm ci because the wrong platform-specific version of esbuild is getting installed.

Is there any way to avoid the platform-specific esbuild package from ending up in package-lock.json in the first place?

@seangwright
Copy link

I noticed this issue when I ran an npm prune locally.

I'm on Windows, so the prune removed the cross platform packages from my package-lock.json. Then when the project was installed/built in GitHub actions (running on Linux) the build blew up because the linux binaries were missing.

Removing my node_modules, deleting package-lock.json and re-installing the dependency got everything working for me.

If you run into this issue, look at the diff of the package-lock.json compared to the last time the build worked and also check if you are building on a different OS than you updated the package-lock.json on.

@electrovir
Copy link

electrovir commented Mar 22, 2023

This error suddenly started consistently happening to me in Netlify deploys using Vite without me having made any config or dependency changes.

Re-generating package-lock.json fixed it, and I have no idea why 🤷

edit
Here's a full command since I keep coming back here:

rm -rf node_modules && rm package-lock.json && npm i

# in a mono-repo, make sure to remove each package's node_modules directory as well:
rm -rf node_modules && rm -rf packages/*/node_modules && rm package-lock.json && npm i

@barbalex
Copy link

barbalex commented Apr 5, 2023

This has just started destroying my vercel builds in a project that has worked for years. Very surprising and hard to solve as removing package-lock.json and the node_modules folder has not helped.

Edit: After another try it works.

@snake-py
Copy link

I simply had to delete the node_modules folder and reinstall

@luixo
Copy link

luixo commented Apr 15, 2023

This happened to me while having two different versions of esbuild in package-lock.json.
The old one used to have packages named esbuild-<os>-<arch> while the new one has @esbuild/<os>-<arch>.

The new one is installed as a nested package (node_modules/@esbuild-kit/core-utils/node_modules/esbuild) which apparently makes it skip postinstall script (npm v9.6.0) which installs the missing dependencies.

I have this error while running CI on linux OS and developing (hence generating package lock) on darwin OS.
The simplest solution I found so far is to copy the module description that is already in package-lock.json (in my case "node_modules/@esbuild/linux-arm64") and populate it with values for @esbuild/linux-x64.

@calexander3
Copy link

calexander3 commented Apr 17, 2023

This happened to me because I had --ignore-scripts on my npm command

kerny3d added a commit to KnowageLabs/Knowage-Server that referenced this issue Apr 26, 2023
@mxfactorial
Copy link

mxfactorial commented May 16, 2023

for when youre switching between a mac and devcontainer

  1. install.sh
#!/bin/bash

# test for node_modules availability
if [[ -d node_modules ]]; then
	# darwin, linux, etc
	PLATFORM="$(echo `uname -s` | tr '[:upper:]' '[:lower:]')"
	# test for absent esbuild platform install
	if [[ "$(npm list --depth=2 | grep -e "─ @esbuild/$PLATFORM" | wc -l | xargs)" -eq 0 ]]; then
		# delete node_modules
		rm -rf node_modules;
	fi
fi

# install when missing
if ! [ -d node_modules ]; then npm install; fi
  1. Makefile adapted
install:
	@if [ -d node_modules ]; then \
		PLATFORM="$$(echo `uname -s` | tr '[:upper:]' '[:lower:]')"; \
		if [ "$$(npm list --depth=2 | grep -e "─ @esbuild/$$PLATFORM" | wc -l | xargs)" -eq 0 ]; then \
			rm -rf node_modules; \
		fi; \
	fi
	if ! [ -d node_modules ]; then npm install; fi

edit: support initially absent node_modules

@vanvix
Copy link

vanvix commented Aug 7, 2023

Solution for me was installing npm i @esbuild/linux-x64 --legacy-peer-deps via the postinstall script.
I have monorepo project so the structure looks like this:

root
   - packages
   - platforms
     - web
       - package.json 
   - package.json // root
      "scripts": {
          ... ,
         "postinstall": "npm i myPrivatePackage --registry http://myPrivateRegistry && npm i @esbuild/linux-x64 --legacy-peer-deps"
      }

When running on windows i'm running:

yarn

and as you can guess the yarn fails, but because it's in the end it fails when everything is already installed :) a little hack but i guess this bug is in the library.
I tried solution wth the .yarnrc.yml and it didn't work for me. I tried adding it in root/package.json and in the platforms/web/package.json and it didnt work.

@romanslonov
Copy link

romanslonov commented Aug 11, 2023

Two options that works on gitlab's docker compose (node:20-buster)

№1

    # Build frontend
    - rm -rf node_modules/.cache/
    - npm install
    - npm i @esbuild/linux-x64 --save-dev
    - npm run test:unit
    - npm run build

№2

    # Build frontend
    - rm -rf node_modules
    - npm install
    - npm run test:unit
    - npm run build

@jonnylink
Copy link

Happened to me using yarn. The different arches were listed in the lock file, but they weren't in the yarn cache.

adding this to .yarnrc.yml fixed it.

supportedArchitectures:
    os:
        - 'darwin'
        - 'linux'
        - 'win32'
    cpu:
        - 'arm'
        - 'arm64'
        - 'x64'

@jsnanigans
Copy link

To anyone who wants to fix this, but does not want to remove --ignore-scripts for all packages,
Running npm install esbuild after npm ci --ignore-scripts fixes the issue.

@Gabuardi
Copy link

Gabuardi commented Sep 12, 2023

For those which runs this issue on mac with NPM can fix it running npm ci instead of npm install on the Dockerfile.

@raythurnevoid
Copy link

raythurnevoid commented Sep 28, 2023

Also running npm i -f solves the issue, there's no need to delete the node_modules folder.

We have some devs working on Windows and others working on Linux. Reproduction steps are easy:

  1. Delete node_modules and package-lock.json
  2. Run npm i on one platform (windows perhaps)
  3. Commit the package-lock.json
  4. Pull and run npm i on the other platform (the error shows up)

@Congee
Copy link

Congee commented Sep 30, 2023

This could make sense, we run our containers on linux images, but for development we mount our local filesystems into the containers, sometimes those local filesystems are running on Mac, for instance.

Is there some way to have ESBuild install for multiple platforms somehow?

Yarn provides --ignore-platform to have all architecture specific dependencies installed. If you do yarn add/install --ignore-platform, your mapped node_modules/ will be able to choose one of the dependencies in your host

@jonnylink
Copy link

I'd recommend using this instead of the ignore platforms flag if you just have two OSs you're worried about. The flag will create unnecessary bloat.

adding this to .yarnrc.yml fixed it.


supportedArchitectures:

    os:

        - 'darwin'

        - 'linux'

        - 'win32'

    cpu:

        - 'arm'

        - 'arm64'

        - 'x64'

@Congee
Copy link

Congee commented Sep 30, 2023

I'd recommend using this instead of the ignore platforms flag if you just have two OSs you're worried about. The flag will create unnecessary bloat.

adding this to .yarnrc.yml fixed it.


supportedArchitectures:

    os:

        - 'darwin'

        - 'linux'

        - 'win32'

    cpu:

        - 'arm'

        - 'arm64'

        - 'x64'

Thanks. That did not work for me. I have two OSs, but node_modules is mapped from one OS to another. The guest OS has to use the host OS's node_modules where yarn only gives me the package of one platform instead of multiple platforms. supportedArchitectures has no effect in this case for me

@yuriy-yarosh
Copy link

yuriy-yarosh commented Dec 21, 2023

yarn set version 3.7.0 fixes things for yarn rn.
Downgrading from 4.0.2 helps.

@Sameerthe-Big-O
Copy link

Sameerthe-Big-O commented Mar 1, 2024

As of 2024
I'm having the same error while deploying my frontend on Vercel
for the local it works fine but vercel try to build for the production it gets crashes
i tried some of the upper solutions but didn't and most of them are either docker-related or platform independent issues
I'd love to hear why this error happened on vercel i mean on the local it makes sense as above told if we manually makes the copies the

@Annukul
Copy link

Annukul commented Mar 20, 2024

I was having the same issue. I am using Remix which uses Vite (vite uses esbuild to transpile typescript to javascript) - while deploying this on Vercel it resulted in this error The package "@esbuild/linux-x64" could not be found, and is needed by esbuild..
How I resolved it:

  • Deleted node_modules and package-lock.json
  • Reinstalled all the packages using npm i
    And it worked!

@Gugustinette
Copy link

Also getting this error on Vercel, when trying to use the VueEmail library with Nuxt. I use npm and node 20.

Somehow related to vue-email/nuxt#40

Very weird behaviour from esbuild or npm or vercel or nitro or I don't know who's responsible for that but this seems like we have a tough problem here.

white0ut added a commit to dudasaus/tierlist that referenced this issue Apr 27, 2024
I had to wipe node_modules and package-log.json and do a fresh install
to get frontend running on my machine.

I ran into this issue: evanw/esbuild#1819
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests