With webpack 5 and once deployed, local files are not readable from APIs anymore #24700

QuentinRoy · 2021-05-02T03:18:15Z

What version of Next.js are you using?

10.2.0

What version of Node.js are you using?

14.0.0

What browser are you using?

Chrome,Firefox,Safari

What operating system are you using?

macOS

How are you deploying your application?

Vercel

Describe the Bug

Since updating to 10.2.0, local files are not accessible from apis anymore once deployed on Vercel. I do not know if it was on purpose, but it used to work fine before upgrading.

In both case, an internal server error is returned. After closer inspection of the logs, the error message is as follows:

[Error: ENOENT: no such file or directory, open '/app/data/metadata.yml'] {

errno: -2,

code: 'ENOENT',

syscall: 'open',

path: '/app/data/metadata.yml'

}

It went back to normal after downgrading to next 10.1.3.

EDIT: It seems the problem comes from webpack 5, it is working fine if webpack5 is disabled in next.config.js.

EDIT: It does not fail if there is only one API trying to read only one local file. It is possible the problem occurs when multiple files have to be available for reading.

EDIT: According to @jacksonhardaker, this also fails when reading files from getServerSideProps.

Expected Behavior

Local files should be readable from apis. If these are not embedded by default anymore, maybe an option could be used to indicate what files should be available?

To Reproduce

Here is an example showing APIs failing: https://github.com/QuentinRoy/vercel-local-min-example deployed at https://vercel-local-min-example.vercel.app.

Here is another example from @jacksonhardaker showing getServerSideProps failing: https://github.com/jacksonhardaker/next-webpack-5-fs-bug deployed at https://next-webpack-5-fs-bug-jwegk8q1o-jacksonhardaker1.vercel.app/.

The text was updated successfully, but these errors were encountered:

QuentinRoy · 2021-05-02T14:19:36Z

Following @mmazzarolo suggestion in #8251, I can attest the problem actually comes with webpack 5. Disabling webpack 5 after updating fixes it:

next.config.js

module.exports = {
  future: {
    webpack5: false,
  },
};

EDIT:
With newer Next versions, switching back to webpack 4 is:

module.exports = {
  webpack5: false,
};

joe-bell · 2021-05-02T17:44:01Z

I manually opted into webpack5 and can confirm I'm also seeing this behaviour inside API routes

Example

API Route

const CONTENT_DIR = path.join(process.cwd(), "src", "content");

export const getContentByDir = async (
  dir: string,
  options?: IGetContentOptions
) => {
  const srcDir = path.join(CONTENT_DIR, dir);

  if (!fs.lstatSync(srcDir)?.isDirectory()) {
    // Error thrown to Vercel
    throw Error(`${srcDir} is not a valid directory`);
  }
  // other unrelated code
};

Vercel Logs

Error: ENOENT: no such file or directory, lstat '/var/task/src/content/blog'
    at Object.lstatSync (fs.js:1077:3)
    at getContentByDir (/var/task/.next/server/pages/api/sitemap.js:283:49)
    at getBlogPaths (/var/task/.next/server/pages/api/sitemap.js:292:23)
    at getAllPaths (/var/task/.next/server/pages/api/sitemap.js:313:67)
    at getSitemapPaths (/var/task/.next/server/pages/api/sitemap.js:314:43)
    at exports.modules.3692.__webpack_exports__.default (/var/task/.next/server/pages/api/sitemap.js:331:96)
    at apiResolver (/var/task/node_modules/next/dist/next-server/server/api-utils.js:8:7)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async Server.handleApiRequest (/var/task/node_modules/next/dist/next-server/server/next-server.js:66:462)
    at async Object.fn (/var/task/node_modules/next/dist/next-server/server/next-server.js:58:580) {
  errno: -2,
  syscall: 'lstat',
  code: 'ENOENT',
  path: '/var/task/src/content/blog'
}

Additional Context

I tried adding this to my next.config.js, but had no luck:

  webpack: (config, { isServer }) => {
    if (!isServer) {
      config.resolve.fallback.fs = false;
    }

    return config;
  },

This behaviour doesn't occur when I opt-out of webpack5

ijjk · 2021-05-05T14:39:34Z

Hi, can someone who's run into this provide a repo with a minimal reproduction, we aren't able to reproduce this with the above provided information

QuentinRoy · 2021-05-05T17:06:16Z

Hi @ijjk, thank you for looking into this.
I investigated the problem a bit more. It seems it is working fine when there is only one api reading one local file. However, if there are two apis reading two local files, it breaks.
Here is a minimal reproduction repository: https://github.com/QuentinRoy/vercel-local-min-example.
This is deployed here: https://vercel-local-min-example.vercel.app

jacksonhardaker · 2021-05-05T18:02:37Z

Good tip @QuentinRoy I can confirm that the addition of a second read appears to reveal the issue.
Repo: https://github.com/jacksonhardaker/next-webpack-5-fs-bug
Deployment: ~~https://next-webpack-5-fs-bug-mfyukza5m-jacksonhardaker1.vercel.app/~~ https://next-webpack-5-fs-bug-jwegk8q1o-jacksonhardaker1.vercel.app/

edit: updated the deployment url

wworrall · 2021-05-06T09:06:37Z

This is a major issue causing Vercel deployments to break silently (build completing successfully). Rolling back to webpack 4 for now - please fix asap, Vercel 😍. Also, perhaps add something to your tests as to which projects are automatically compatible with webpack5 https://nextjs.org/docs/messages/webpack5

joe-bell · 2021-05-06T15:35:09Z

@wworrall your tone is super unproductive and contributes nothing to this issue

QuentinRoy · 2021-05-06T17:22:23Z

@jacksonhardaker, does your deployment matches the code in your repo? The deployed app appears functional.

jacksonhardaker · 2021-05-06T19:12:13Z

@QuentinRoy Thanks for the ping, I must have pasted the incorrect url. Updated.

marbiano · 2021-05-25T01:13:12Z

We have a slightly similar case. We can read files but only if one API endpoint is doing the fs.readFile. The moment we add a second endpoint it fails!

Thanks for taking a look and providing help, Vercel crew. 🙏

EDIT: just realized it's the same issue as @QuentinRoy described above. Good to know we are not the only ones!

carl0s · 2021-10-02T00:56:37Z

Any update on this?

ijjk · 2021-10-02T03:47:28Z

@carl0s did you try the experimental.nftTracing config already and experienced issues still? If so you can try the latest version of Next.js v11.1.3-canary.45 with the now renamed experimental.outputFileTracing.

carl0s · 2021-10-05T19:48:37Z

I will totally try that!

carl0s · 2021-10-06T08:06:52Z

I tried that with the mentioned version of Next.js, v11.1.3-canary.45.
I'm trying to have a next build with next-18n and I have this error.

Error occurred prerendering page "/de/article/test-article". Read more: https://nextjs.org/docs/messages/prerender-error Error: Cannot find module '[redacted]/next-i18next.config.js' at webpackEmptyContext ([redacted]/.next/serverless/pages/[...slug].js:11:10) at [redacted]/.next/serverless/chunks/709.js:245:72 at processTicksAndRejections (node:internal/process/task_queues:96:5)

I removed some reference to local paths, as they might be confusing.
Without I18n works fine, and with webpack 4 it builds fine. Though we need webpack 5 to work correctly.

adrai · 2021-10-06T08:25:15Z

I tried that with the mentioned version of Next.js, v11.1.3-canary.45. I'm trying to have a next build with next-18n and I have this error.

Error occurred prerendering page "/de/article/test-article". Read more: https://nextjs.org/docs/messages/prerender-error Error: Cannot find module '[redacted]/next-i18next.config.js' at webpackEmptyContext ([redacted]/.next/serverless/pages/[...slug].js:11:10) at [redacted]/.next/serverless/chunks/709.js:245:72 at processTicksAndRejections (node:internal/process/task_queues:96:5)

I removed some reference to local paths, as they might be confusing. Without I18n works fine, and with webpack 4 it builds fine. Though we need webpack 5 to work correctly.

@carl0s This is a "known" issue... read about it here: i18next/next-i18next#1202 (comment) and here: i18next/next-i18next#1384

eric-burel · 2021-10-15T08:01:16Z

The update to 11.3 with experimental.outputFileTracing seems to stop this error message from happening but leads to new issues with Material UI v5 / Emotion, I get this:

/node/index.js',
    '/var/task/node_modules/@mui/system/index.js',
    '/var/task/node_modules/@mui/material/node/styles/adaptV4Theme.js',
    '/var/task/node_modules/@mui/material/node/styles/index.js',
    '/var/task/.next/server/pages/_document.js',
    '/var/task/node_modules/next/dist/server/require.js',
    '/var/task/node_modules/next/dist/server/load-components.js',
    '/var/task/node_modules/next/dist/server/api-utils.js',
    '/var/task/node_modules/next/dist/server/next-server.js',
    '/var/task/___next_launcher.js',
    '/var/runtime/UserFunction.js',
    '/var/runtime/index.js'
  ]
}
2021-10-15T07:57:31.238Z	f5ecb81a-7cb9-49fa-9f66-4595e6f427d0	ERROR	Error: Cannot find module 'stylis'
Require stack:
- /var/task/node_modules/@emotion/cache/dist/emotion-cache.cjs.prod.js
- /var/task/node_modules/@emotion/cache/dist/emotion-cache.cjs.js
... // etc.

Seems similar to this ticket: emotion-js/emotion#850 and this old SO question (https://stackoverflow.com/questions/62526318/nextjs-emotionjs-and-vercel-error-err-require-esm-must-use-import-to-load-es). Installing stylis explicitely doesn't fix the issue.

Only one route of my app is affected I don't really get why specifically, probably because it's an MDX file, with @next/mdx.
It also affects API routes despite them not using @emotion at all.

I can reproduce only on Vercel so far, not locally. I can share a non-minimal repro if needed, this affect an open source project.

ijjk · 2021-10-19T19:31:26Z

@eric-burel the experimental.outputFileTracing has been updated in the latest version of Next.js v11.1.3-canary.86, please update and give it a try! If you are still seeing issues please provide the mentioned repo for further investigation.

adrai · 2021-10-20T07:12:57Z

v11.1.3-canary.86 behaves still the same for next-i18next issue: #24700 (comment)

eric-burel · 2021-10-20T15:26:07Z

@ijjk First try after this update canary.86 gives me ModuleNotFoundError: Module not found: Error: Can't resolve 'querystring' in '/vercel/path0/node_modules/next/dist/compiled/native-url' during Vercel build and locally as well.
It's probably just an unrelated regression in this version, it's fixed by installing querystring, explicitely.

Then after a second try, it seems to partially fix the issue: I don't have trouble with API routes anymore.

But my MDX page still doesn't work.

2021-10-20T15:24:05.039Z	2f2bc57a-9453-45f1-8b7c-785d883c2e14	ERROR	TypeError: Cannot destructure property 'components' of 'object null' as it is null.
    at Server.renderToResponseWithComponents (/var/task/node_modules/next/dist/server/next-server.js:985:90)
    at Server.renderErrorToResponse (/var/task/node_modules/next/dist/server/next-server.js:1428:35)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at async Server.pipe (/var/task/node_modules/next/dist/server/next-server.js:866:25)
    at async Object.fn (/var/task/node_modules/next/dist/server/next-server.js:671:21)
    at async Router.execute (/var/task/node_modules/next/dist/server/router.js:205:32)
    at async Server.run (/var/task/node_modules/next/dist/server/next-server.js:843:29)
    at async Server.handleRequest (/var/task/node_modules/next/dist/server/next-server.js:293:20)
    at async Server.<anonymous> (/var/task/___next_launcher.js:32:9)

This page works locally but still not in Vercel.

You can get the code here: https://github.com/eric-burel/vulcan-next/tree/main
Faulty page is there: https://vulcan-next.vercel.app/learn

ijjk · 2021-10-20T15:27:58Z

@adrai it looks like the config is being included correctly in the latest canary while deploying the example, test deployment done here it seems there is a different issue which is responded about on the initial issue here.

ijjk · 2021-10-20T16:59:55Z

@eric-burel it looks like your issue is related to next-i18next as well so the above comment is relevant for your case as well.

adrai · 2021-10-20T17:24:24Z

@adrai it looks like the config is being included correctly in the latest canary while deploying the example, test deployment done here it seems there is a different issue which is responded about on the initial issue here.

@ijjk I still get: error - Error: Cannot find module '/Users/usr/Projects/next-i18next/examples/simple/next-i18next.config.js'

steps to reproduce:

clone https://github.com/isaachinman/next-i18next
add experimental flag here: https://github.com/isaachinman/next-i18next/blob/master/examples/simple/next.config.js
use new canary version of next here: https://github.com/isaachinman/next-i18next/blob/master/examples/simple/package.json#L12
npm run run-example and visit http://localhost:3000

ijjk · 2021-10-20T18:29:14Z

@adrai that appears to be an issue with running the package from the monorepo, the package.json for the example should look like this:

{
  "name": "v10",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "scripts": {
    "dev": "next",
    "build": "next build",
    "start": "next start"
  },
  "dependencies": {
    "next": "canary",
    "next-i18next": "latest",
    "react": "^17.0.2",
    "react-dom": "^17.0.2"
  },
  "devDependencies": {
    "prop-types": "^15.7.2"
  }
}

and the modules should be installed specific to the example along with running yarn next build inside of the example folder. The reason it isn't finding the config is because an incorrect current working directory is being used.

adrai · 2021-10-20T18:41:57Z

@ijjk you're right

eric-burel · 2021-10-20T19:01:31Z

Much thanks for the help @ijjk I've made some progress.

I ended up with the config described there in the relevant next-i18next ticket i18next/next-i18next#661 (comment)

However, I have a new issue :/

2021-10-20T18:55:03.846Z	63b14614-4efe-45fb-877f-d362013ab373	ERROR	Error: Cannot find module '@mdx-js/react'
Require stack:
- /var/task/.next/server/pages/learn/intro-online.js
- /var/task/node_modules/next/dist/server/require.js
- /var/task/node_modules/next/dist/server/load-components.js
- /var/task/node_modules/next/dist/server/api-utils.js
- /var/task/node_modules/next/dist/server/next-server.js
- /var/task/___next_launcher.js
- /var/runtime/UserFunction.js
- /var/runtime/index.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:889:15)
    at Function.Module._load (internal/modules/cjs/loader.js:745:27)
    at Module.require (internal/modules/cjs/loader.js:961:19)
    at require (internal/modules/cjs/helpers.js:92:18)
    at Object.7425 (/var/task/.next/server/pages/learn/intro-online.js:144:18)
    at __webpack_require__ (/var/task/.next/server/webpack-runtime.js:25:42)
    at Object.8373 (/var/task/.next/server/pages/learn/intro-online.js:19:71)
    at __webpack_require__ (/var/task/.next/server/webpack-runtime.js:25:42)
    at __webpack_exec__ (/var/task/.next/server/pages/learn/intro-online.js:399:39)
    at /var/task/.next/server/pages/learn/intro-online.js:400:91 {
  code: 'MODULE_NOT_FOUND',

this happens if I refresh the page https://vulcan-next.vercel.app/learn/intro-online
this doesn't happen if I go to another page, and just add /learn in the URL => as far as I can tell Next is doing a kind of client page transition in this case. So basically only the server render fails.
no repro locally, it works ok

The package is correctly listed in the dependencies.

Sorry I'd like to help more but I don't fully understand what happens, and the error message is kinda vague here. That's maybe a different issue. Could there be some server-rendering happening? This is an MDX page using the row @next/mdx setup and MDXProvider from @mdx-js/react to pass some components.

ijjk · 2021-10-25T21:18:44Z

@eric-burel you are adding getInitialProps in _app here so any page that doesn't use getStaticProps will be server rendered for every request. It seems the above shared URL is working correctly currently, did you need to change something related to mdx for this to work?

ijjk · 2021-10-25T21:20:31Z

For anyone else who ran into the initial issue, please try upgrading to v11.1.3-canary.104 of Next.js and remove the experimental.outputFileTracing as it is no longer needed and confirm the issue is resolved fully 🙏

eric-burel · 2021-10-27T07:25:45Z

@ijjk Nope it doesn't actually work, if you reload the page again, you'll get a 500 error. After updating to canary 104 I still get the server error Cannot find module '@mdx-js/react'. Very weird, I could not reproduce locally ever, it's really specific to the Vercel deployment.

Thanks for the tip though, I think getting rid of getInitialProps will fix it since it will prevent a rerender, I need to update next-i18next to the latest version. However it also means I'll lose next export, but I can live without it.
I'll do that and open a separate issue if I'm still stuck.

ijjk · 2021-11-17T16:30:25Z

I'm going to close this as this should be working properly in the latest version of Next.js v12.0.4, if you are still encountering issues with this in the latest version please open a fresh issue with reproduction.

balazsorban44 · 2022-01-27T08:05:53Z

This issue has been automatically locked due to no recent activity. If you are running into a similar issue, please create a new issue with the steps to reproduce. Thank you.

QuentinRoy added the bug Issue was opened via the bug report template. label May 2, 2021

QuentinRoy changed the title ~~Local files not readable from APIs anymore~~ With webpack 5 and once deployed, local files not readable from APIs anymore May 2, 2021

QuentinRoy changed the title ~~With webpack 5 and once deployed, local files not readable from APIs anymore~~ With webpack 5 and once deployed, local files are not readable from APIs anymore May 2, 2021

timneutkens added kind: bug and removed bug Issue was opened via the bug report template. labels May 5, 2021

timneutkens added this to the Iteration 20 milestone May 5, 2021

timneutkens assigned sokra and ijjk May 5, 2021

ijjk added the please add a complete reproduction The issue lacks information for further investigation label May 5, 2021

QuentinRoy mentioned this issue May 7, 2021

Next.js API routes (and pages) should support reading files #8251

Closed

ijjk mentioned this issue May 11, 2021

Tracing fails to detect fs when used in multiple entries with webpack 5 vercel/nft#201

Closed

This was referenced May 16, 2021

Not working with Next.js >= 10.2.x (probably because of webpack 5, dynamic imports are handled differently) i18next/next-i18next#1202

Closed

yarn build with Webpack 5 + next-18next fails with Error: ENOENT: no such file or directory #24729

Closed

timneutkens modified the milestones: Iteration 20, Iteration 21 May 22, 2021

ijjk mentioned this issue Jun 16, 2021

Add entrypoint tracing #25538

Merged

2 tasks

timneutkens modified the milestones: Iteration 21, Iteration 22, Iteration 23 Jun 21, 2021

ijjk removed the please add a complete reproduction The issue lacks information for further investigation label Sep 19, 2021

isaachinman mentioned this issue Sep 27, 2021

feat: add possibility to define next-i18next.config as json file i18next/next-i18next#1384

Closed

eric-burel mentioned this issue Oct 15, 2021

Cannot find module '@emotion/stylis' from 'create-emotion/dist/index.cjs.js emotion-js/emotion#850

Closed

belgattitude mentioned this issue Oct 17, 2021

Trying out outputFileTracing in latest next 11.1.3.canary.93 belgattitude/nextjs-monorepo-example#613

Closed

styfle modified the milestones: 11.1.x, 12.0.4 Nov 5, 2021

timneutkens added the Output (export/standalone) Related to the the output option in `next.config.js`. label Nov 16, 2021

timneutkens removed this from the 12.0.5 milestone Nov 17, 2021

ijjk closed this as completed Nov 17, 2021

vercel locked as resolved and limited conversation to collaborators Jan 27, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

With webpack 5 and once deployed, local files are not readable from APIs anymore #24700

With webpack 5 and once deployed, local files are not readable from APIs anymore #24700

QuentinRoy commented May 2, 2021 •

edited

Loading

QuentinRoy commented May 2, 2021 •

edited

Loading

joe-bell commented May 2, 2021 •

edited

Loading

ijjk commented May 5, 2021

QuentinRoy commented May 5, 2021 •

edited

Loading

jacksonhardaker commented May 5, 2021 •

edited

Loading

wworrall commented May 6, 2021 •

edited

Loading

joe-bell commented May 6, 2021 •

edited

Loading

QuentinRoy commented May 6, 2021 •

edited

Loading

jacksonhardaker commented May 6, 2021

marbiano commented May 25, 2021 •

edited

Loading

carl0s commented Oct 2, 2021

ijjk commented Oct 2, 2021 •

edited

Loading

carl0s commented Oct 5, 2021

carl0s commented Oct 6, 2021

adrai commented Oct 6, 2021

eric-burel commented Oct 15, 2021 •

edited

Loading

ijjk commented Oct 19, 2021

adrai commented Oct 20, 2021

eric-burel commented Oct 20, 2021

ijjk commented Oct 20, 2021

ijjk commented Oct 20, 2021

adrai commented Oct 20, 2021

ijjk commented Oct 20, 2021 •

edited

Loading

adrai commented Oct 20, 2021

eric-burel commented Oct 20, 2021 •

edited

Loading

ijjk commented Oct 25, 2021

ijjk commented Oct 25, 2021 •

edited

Loading

eric-burel commented Oct 27, 2021

ijjk commented Nov 17, 2021

balazsorban44 commented Jan 27, 2022

With webpack 5 and once deployed, local files are not readable from APIs anymore #24700

With webpack 5 and once deployed, local files are not readable from APIs anymore #24700

Comments

QuentinRoy commented May 2, 2021 • edited Loading

What version of Next.js are you using?

What version of Node.js are you using?

What browser are you using?

What operating system are you using?

How are you deploying your application?

Describe the Bug

Expected Behavior

To Reproduce

QuentinRoy commented May 2, 2021 • edited Loading

joe-bell commented May 2, 2021 • edited Loading

Example

Additional Context

ijjk commented May 5, 2021

QuentinRoy commented May 5, 2021 • edited Loading

jacksonhardaker commented May 5, 2021 • edited Loading

wworrall commented May 6, 2021 • edited Loading

joe-bell commented May 6, 2021 • edited Loading

QuentinRoy commented May 6, 2021 • edited Loading

jacksonhardaker commented May 6, 2021

marbiano commented May 25, 2021 • edited Loading

carl0s commented Oct 2, 2021

ijjk commented Oct 2, 2021 • edited Loading

carl0s commented Oct 5, 2021

carl0s commented Oct 6, 2021

adrai commented Oct 6, 2021

eric-burel commented Oct 15, 2021 • edited Loading

ijjk commented Oct 19, 2021

adrai commented Oct 20, 2021

eric-burel commented Oct 20, 2021

ijjk commented Oct 20, 2021

ijjk commented Oct 20, 2021

adrai commented Oct 20, 2021

ijjk commented Oct 20, 2021 • edited Loading

adrai commented Oct 20, 2021

eric-burel commented Oct 20, 2021 • edited Loading

ijjk commented Oct 25, 2021

ijjk commented Oct 25, 2021 • edited Loading

eric-burel commented Oct 27, 2021

ijjk commented Nov 17, 2021

balazsorban44 commented Jan 27, 2022

QuentinRoy commented May 2, 2021 •

edited

Loading

QuentinRoy commented May 2, 2021 •

edited

Loading

joe-bell commented May 2, 2021 •

edited

Loading

QuentinRoy commented May 5, 2021 •

edited

Loading

jacksonhardaker commented May 5, 2021 •

edited

Loading

wworrall commented May 6, 2021 •

edited

Loading

joe-bell commented May 6, 2021 •

edited

Loading

QuentinRoy commented May 6, 2021 •

edited

Loading

marbiano commented May 25, 2021 •

edited

Loading

ijjk commented Oct 2, 2021 •

edited

Loading

eric-burel commented Oct 15, 2021 •

edited

Loading

ijjk commented Oct 20, 2021 •

edited

Loading

eric-burel commented Oct 20, 2021 •

edited

Loading

ijjk commented Oct 25, 2021 •

edited

Loading