Using esbuild in serverless functions fails due to "esbuild-linux-64" not being found

[silvenon]

Discussed in #6865

^{Originally posted by hassanzadeh October 19, 2021}
Hey guys,
Starting form about a week ago, all my deployments fail at Run time with this error (copied form functions log):
"2021-10-19T17:58:53.710Z f7284007-d328-48a7-b4e3-ecf669500227 ERROR Error: The package "esbuild-linux-64" could not be found, and is needed by esbuild.
If you are installing esbuild with npm, make sure that you don't specify the
"--no-optional" flag. The "optionalDependencies" package.json feature is used
by esbuild to install the correct binary executable for your current platform.
at generateBinPath (/var/task/node_modules/esbuild/lib/main.js:1643:15)"

I'm not sure what has changed under the hood that is causing everything to break. I also noticed that I'm getting these warnings during the build time;
Info esbuild-darwin-arm64@0.13.8: The platform "linux" is incompatible with this module.

Any thoughts?
thanks

Hello!

It appears that using esbuild in serverless functions doesn't work, I've been getting the same error described in the discussion above, i.e. esbuild cannot find esbuild-linux-64. I created a Remix project from Vercel's template and ran esbuild in the home page's loader to demonstrate this issue, it's deployed at https://silvenon-repro.vercel.app, where you can clearly see the error. The project builds fine, so the esbuild that Remix itself uses under the hood works fine in the build step, but using esbuild separately later, in the application itself, doesn't work.

Any help with this would be highly appreciated ❤️

[jodafm]

Hi @silvenon, do you find a solution?

[silvenon]

No 😕

[schickling]

@styfle thanks for looking into this. Is your "fix" already live? I'm still running into this issue.

[styfle]

We need to land #7349 for node.

Or if you’re using Next.js you could try next@canary

[styfle]

You can try this out today by adding an environment variable to your project FORCE_RUNTIME_TAG=canary, thanks!

[pakatagoh]

Thanks for all the hard work! For me, I still seem to be getting an error

2022-02-13T02:49:08.776Z	ad9543f4-9c73-4af4-9764-15aa52369cf3	ERROR	Error: The package "esbuild-linux-64" could not be found, and is needed by esbuild.
If you are installing esbuild with npm, make sure that you don't specify the
"--no-optional" flag. The "optionalDependencies" package.json feature is used
by esbuild to install the correct binary executable for your current platform.
    at generateBinPath (/var/task/output/server/pages/node_modules/esbuild/lib/main.js:1806:15)
    at esbuildCommandAndArgs (/var/task/output/server/pages/node_modules/esbuild/lib/main.js:1862:11)
    at ensureServiceIsRunning (/var/task/output/server/pages/node_modules/esbuild/lib/main.js:2019:25)
    at Object.build (/var/task/output/server/pages/node_modules/esbuild/lib/main.js:1912:26)
    at bundleMDX (/var/task/output/server/pages/node_modules/mdx-bundler/dist/index.js:242:33)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at async getOneBlogPost (/var/task/output/server/pages/api/_build/index.js:670:35)
    at async loader (/var/task/output/server/pages/api/_build/index.js:712:40)
    at async Object.callRouteLoader (/var/task/output/server/pages/node_modules/@remix-run/server-runtime/data.js:73:14)

After adding FORCE_RUNTIME_TAG=canary

Vercel build logs the following:

Running `vercel build`
--
10:36:49.600 | Vercel CLI 23.1.3-canary.71 build (beta) — https://vercel.com/feedback

What should the run canary version be?

build passes for my remix application but fails during run time due to the esbuild issue

I'm using mdx-bundler and esbuild with the following versions

    "esbuild": "^0.14.21",
    "mdx-bundler": "^8.0.1",

Error still happens on my blog detail route:
https://pakatagoh-com-o2ickks5c-pakata.vercel.app/blog/switching-to-software-development-as-a-career

For repo details : https://github.com/pakatagoh/pakatagoh.com/tree/pakatagoh-dot-com/v2

[styfle]

I see now. Remix is using the experimental Filesystem API https://github.com/silvenon/repro/blob/2e26ea8f10ef90e46bb3102bfb90633b273ab84e/vercel.json#L7

We'll have to update that separately.

[pakatagoh]

@styfle

thanks for looking into this.

Not sure if it helps, but when downgrading the version of esbuild to v0.12.9 everything works fine for me

[bshzzhsb]

@styfle

thanks for looking into this.

Not sure if it helps, but when downgrading the version of esbuild to v0.12.9 everything works fine for me

I have downgraded esbuild to v0.12.9. It sometimes works fine, but sometimes cannot work. So I think this bug is related to vercel.

[jescalan]

Hey @silvenon - want to let you know we are still tracking this issue and plan to resolve it, but I am curious if you'd be able to provide a little more context around your use case here. Why is it that you need to run esbuild inside the loader?

Also, would you be able to run esbuild inside of an api function and call that function to get the same result? We have tested the code below inside something like /api/esbuild.js and confirmed that it does work.

const esbuild = require('esbuild')

module.exports = async (req, res) => {
  const result = await esbuild.transform(`{ "title": "Hello world!" }`, { loader: 'json' })
  res.end(result.code)
}

[silvenon]

Hey @jescalan! My use case is compiling MDX blog posts upon request using mdx-bundler, which uses esbuild.

Considering that Remix compiles to /api/index.js, is calling esbuild from there different than your api/esbuild.js example? Also, based on what @styfle said I assumed that compiling strings works now, but not files. But I didn't try it out.

[jescalan]

Is there any reason you don't want to compile the blog posts at build time, rather than on request? This seems like it would lead to slower response times with this approach.

[silvenon]

They are indeed slow, you're right, I'm not sure how to solve that without building ahead of time as I'm very inexperienced with SSR apps, maybe build them all and cache them before the user visits, but then I'd feel like I still have a static site, and slower deploys. The main point is that I also wanted to build a CMS where I can edit posts as MDX. In that case I'd need to have it build on save and cached right away maybe, and it means that I'd need to have mdx-bundler in runtime, no?

[jescalan]

Typically in this case running a static site with slower deploys is the right tradeoff, since it's more important that users can see your page render as quickly as possible than that it takes an extra minute to deploy.

What you can do with the CMS is have it hit a wekhook that triggers a re-deployment of the site - this is possible on vercel. If you use next.js you can also leverage ISR to make the deployments faster, but I do not think remix supports this feature at the moment.

[silvenon]

Yeah, I could do something like that, I was using a fully SSG solution with Vite before, but I wanted to see if I could build a fullstack blog that's fast enough without static components. I wonder what Kent C. Dodds is doing to makes his blog so fast. He's also using mdx-bundler, but I guess he's doing some sophisticated prefetching and caching that is beyond my skillset at the moment. He documented it well, though, so I'll figure it out eventually.

I understand why you want to talk me out of this 😄 it does sound kind of unnecessary, but I would prefer continuing in this direction because I'm learning valuable performance concepts that I wasn't learning with SSG frameworks.

[brattonross]

I'm in somewhat the same boat as you, @silvenon. I wanted to try out Remix, so I built a blog that uses mdx-bundler, deployed to vercel, and came across this issue. In my case I am fetching the MDX content from an API instead of reading from the filesystem.

Remix purposely does not support SSG, so mdx-bundler seemed like the best solution, plus it gets a mention in the docs.

[SandroMaglione]

Same issue here. I am using esbuild and mdx-bundler in a api function:

"dependencies": {
    "esbuild": "^0.14.31",
    "mdx-bundler": "^9.0.0",

The error is the following:

2022-04-04T15:30:12.576Z	674dfae9-36e0-4d08-9b4a-43023fdf14bf	ERROR	Error: The package "esbuild-linux-64" could not be found, and is needed by esbuild.
If you are installing esbuild with npm, make sure that you don't specify the
"--no-optional" flag. The "optionalDependencies" package.json feature is used
by esbuild to install the correct binary executable for your current platform.
    at generateBinPath (/var/task/node_modules/esbuild/lib/main.js:1813:15)
    at esbuildCommandAndArgs (/var/task/node_modules/esbuild/lib/main.js:1869:31)
    at ensureServiceIsRunning (/var/task/node_modules/esbuild/lib/main.js:2031:25)
    at Object.build (/var/task/node_modules/esbuild/lib/main.js:1924:26)
    at bundleMDX (/var/task/node_modules/mdx-bundler/dist/index.js:242:33)
    at async handler (/var/task/.next/server/pages/api/bundler.js:28:24)
    at async Object.apiResolver (/var/task/node_modules/next/dist/server/api-utils.js:102:9)
    at async NextNodeServer.handleApiRequest (/var/task/node_modules/next/dist/server/base-server.js:1076:9)
    at async Object.fn (/var/task/node_modules/next/dist/server/base-server.js:963:37)
    at async Router.execute (/var/task/node_modules/next/dist/server/router.js:222:32)

I tried adding the FORCE_RUNTIME_TAG variable but without success.

In my case the client sends a request to the api which should run mdx-bundler and return the formatted code.

Any news or suggestions?

Note: I am using NextJs and not remix "next": "12.0.8"

---

Update

So, I upgraded NextJs to "next": "^12.1.4", and now it works 💁🏼‍♂️.

"dependencies": {
  "mdx-bundler": "^9.0.0",
  "esbuild": "^0.14.31",
  "next": "^12.1.4",

[fwouts]

I have a use case where I'd want to use esbuild at runtime in the context of an online developer tool, ideally hosted on Vercel.

I'm running into the same issue. Note that I'm also using Remix.

Repro:

• Vercel deployment: https://vercel-esbuild-repro.vercel.app/
• Source: https://github.com/fwouts/vercel-esbuild-repro

[jescalan]

The best solution at the moment is to put esbuild inside of an api function and call to the api function from your loader

[silvenon]

What do you mean by "API function"? My repro-remix-esbuild-vercel is a somewhat minimal demo of the issue. The error that appears in the app is not the one described in this issue because I didn't know how to make that messsage appear in the frontend, but the error is visible in the function logs:

Would you mind describing what exactly I would need to do to make this work? I would like to edit the issue description with that info so that the comments in this issue hopefully stop repeating themselves.

[jescalan]

Sure - vercel as a platform has a concept of "functions" that are created by putting some code in the /api folder at the root of your project (more docs here). If instead of adding the code that requires esbuild to a loader, you add the code to a function, then fetch the function from your loader, as far as we have tested, this should work. Hopefully this helps?

[silvenon]

Remix already compiles down to a serverless function, how would using esbuild in a separate function (created manually) be different than using it in a Remix loader? Did I understand your suggestion correctly?

[jescalan]

@silvenon fishing for more updates here - it looks like the root cause of the error should be resolved if remix is able to update their deploy adapter to the latest version of the build output api - I have been in contact with their team about this but haven't got a response yet. Hopefully it's something that will happen soon!

[silvenon]

Thanks so much for the info! 👌 Waiting patiently then 😌

[Infi-Knight]

@jescalan any updates on this issue?

[silvenon]

@Infi-Knight it turns out that all you need to do is add an environment variable in your vercel.json file, Remix team doesn't need to do anything.

{
  "build": {
    "env": {
      "ENABLE_VC_BUILD": "1"
    }
  }
}

You can also turn on ENABLE_FILE_SYSTEM_API the same way if you need filesystem access. My example app doesn't because it's compiling strings, not files.

Check out the working deployment of my app: https://repro-remix-esbuild-vercel-72x4eh7fe-silvenon.vercel.app/

I'll close this because apparently the problem is solved 🎉