Next.js API routes (and pages) should support reading files

[Timer]

Feature request

Is your feature request related to a problem? Please describe.

It's currently not possible to read files from API routes or pages.

Describe the solution you'd like

I want to be able to call fs.readFile with a __dirname path and have it "just work".

This should work in Development and Production mode.

Describe alternatives you've considered

This may need to integrate with @zeit/webpack-asset-relocator-loader in some capacity. This plugin handles these types of requires.

However, it's not a necessity. I'd be OK with something that only works with __dirname and __filename (no relative or cwd-based paths).

Additional context

Example:

// pages/api/test.js
import fs from 'fs'
import path from 'path'

export default (req, res) => {
  const fileContent = fs.readFileSync(
    path.join(__dirname, '..', '..', 'package.json'), 
    'utf8'
  )
  // ...
}

Note: I know you can cheat the above example ☝️ with require, but that's not the point. 😄

[amytych]

Just wanted to second that, trying to implement file uploading using API routes. I can get the file to upload but then need to be able to access it again to upload it to S3 bucket.

[ScottSmith95]

I second this! Also, being able to read directories is very important for my company's usage as we keep our data like team members and blog posts in a content directory so we're looking for a way to require all files in the directory.

[Timer]

The above PR will fix this! ☝️ 🙏

[marlonmarcello]

How about fs.writeFile is that possible? For example, create and save a JSON file based on a webhook that was posted on an /api/route

[huv1k]

Hey @marlonmarcello, this is going to be possible. Stay tuned 😊

[NicolasHz]

It's this already solved?

[huv1k]

Not yet, you can subscribe for #8334

[NicolasHz]

@huv1k Many thanks!

[BrunoBernardino]

Is there a way to help this move forward more quickly?

[bitjson]

Worth noting: if you're using TypeScript, you can already import a JSON file as a module directly (make sure resolveJsonModule is true in tsconfig.json). E.g.:

import myJson from '../../../some/path/my.json';

The shape of the JSON object is also automatically used as its type, so autocomplete is really nice.

[jkjustjoshing]

Workaround I'm using:

# next.config.js
module.exports = {
    serverRuntimeConfig: {
        PROJECT_ROOT: __dirname
    }
}

and in the location you need the path

import fs from 'fs'
import path from 'path'
import getConfig from 'next/config'
const { serverRuntimeConfig } = getConfig()

fs.readFile(path.join(serverRuntimeConfig.PROJECT_ROOT, './path/to/file.json'))

I know this doesn't solve the need to reference files with paths relative to the current file, but this solves my very related use case (reading image files from a /public/images folder).

[lancejpollard]

I just tried this (adding my call to load a file from within getServerSideProps in a page), but it's still not finding the file:

{ PROJECT_BASE: '/vercel/path1' }
Error: ENOENT: no such file or directory, open '/vercel/path1/data/name/list.json'
    at Object.openSync (node:fs:600:3)
    at Object.readFileSync (node:fs:468:35)
    at loadJSON (/var/task/.next/server/chunks/1336.js:27:71)
    at renderNameList (/var/task/.next/server/pages/name/list.js:403:25)

I have a data folder in the base of the project, and data/name/list.json has some JSON data, but it's not loading. Does this still work in 2023? (3.5 years after you posted this @jkjustjoshing)?

I also tried just doing:

fs.readFileSync(`${process.cwd()}/data/name/list.json`)

But same error, however that time process.cwd() showed a different folder than this config solution, it showed:

Error: ENOENT: no such file or directory, open '/var/task/data/name/list.json'

Any ideas how to solve this in today's day and age? I don't know if I'm misconfiguring anything, but it seems all right, so not sure what's going wrong yet.

I will say, it works fine either way locally (with the config solution, or process.cwd(), or just leaving it off altogether). But doesn't work on Vercel in any of the 3 cases.

[lancejpollard]

Alright so I tried a few other things.

• I tried putting path.resolve(process.cwd(), './some/path.json') inside of a utility module, but that didn't work.
• I tried putting path.resolve('./some/path.json') directly in a getServerSideProps call (hardcoded path), and it seems to have worked!

It looks like what everyone recently is saying is a working hack: Vercel with Next.js looks at static path.resolve calls directly in your key files like the pages file like in getServerSideProps to determine what to include in the serverless build. 🎉

Apparently you also need to make sure to import path in the top-level module scope, like this:

import path from 'path'

export async function getServerSideProps(ctx) {
  const fs = (await import('fs')).default
  const content = fs.readFileSync(path.resolve('./some/path.json'), 'utf-8')
}

This won't work:

export async function getServerSideProps(ctx) {
  const path = (await import('path')).default
  const fs = (await import('fs')).default
  const content = fs.readFileSync(path.resolve('./some/path.json'), 'utf-8')
}

[IanMitchell]

Saw in the PR this has changed a bit - any update on what the current plans are (or aren't)? Sounds like there are some strategies you don't want pursued, mind listing them + why so contributors can give this a shot?

[timsuchanek]

This is blocking the usage of nexus with Next.js. It would be great to see this prioritized again.

[karthhhi]

Workaround I'm using:

# next.config.js
module.exports = {
    serverRuntimeConfig: {
        PROJECT_ROOT: __dirname
    }
}

and in the location you need the path

import fs from 'fs'
import path from 'path'
import getConfig from 'next/config'
const { serverRuntimeConfig } = getConfig()

fs.readFile(path.join(serverRuntimeConfig.PROJECT_ROOT, './path/to/file.json'))

I know this doesn't solve the need to reference files with paths relative to the current file, but this solves my very related use case (reading image files from a /public/images folder).

Awesome Man. Worked for me.

[ScottSmith95]

I've been using the new getStaticProps method for this (in #9524). The method is currently marked as unstable but there seems to be good support from the Next.js team on shipping it officially.

e.g.:

export async function unstable_getStaticProps() {
  const siteData = await import("../data/pages/siteData.json");
  const home = await import("../data/pages/home.json");

  return {
    props: { siteData, home }
  };
}

[lancejpollard]

I got some strange bugs when importing dozens of large JSON files in this way, so I'm staying away 😅 .

[activenode]

it's been 3 years and the best answer imo is still a cheat/workaround. Is vercel giving up on this feature?

I don't get how this is still a thing. I mean I now installed webpack raw-loader to be able to import SomeFile from 'whatever.html' but now I'm obviously trapped in webpack which I don't get honestly.

What's the problem that this still is a problem? Would really love to know more.

[revskill10]

The universal solution, is we store blob in S3, and read file directly from S3.

[activenode]

Which works but is very inconvenient for many people including myself. For several reasons but first and foremost I don't wanna create a S3 bucket for loading one file.

[Janpot]

😄 The universal solution is "we use the proprietary API of a single cloud provider"?

[the-retreater]

For anyone having file issues using the @sentry/nextjs package I spent too much time trying to resolve a issue where email template .hbs files were not being copied over when deployed. You need to change your next.config.js in the sentry section of your config:

autoInstrumentServerFunctions: false

After you've done this to make Sentry work on your API routes you'll need to wrap the handler like so:

export default withSentry(handler);

There is a open PR to address this issue vercel/nft#322

[spicybackend]

Thank you so much for linking this back 🙌🏼 I've spent the last few days panicking about the regression in my app and wouldn't have ever thought the Sentry integration was the cause!

To add to this, if you only have a handful of endpoints that need fs access, you can exclude auto-instrumentation for those particular endpoints with the excludeServerRoutes option added in @sentry/nextjs 7.20.0. This way you don't have to manually wrap all of your endpoints with the withSentry wrapper.

const moduleExports = {
  sentry: {
    excludeServerRoutes: ['/api/endpoint/reading/files']
  }
}

[jakobo]

Can confirm we ran into this issue as well. Here's the related Sentry issue getsentry/sentry-javascript#5998

[nbouvrette]

Just spent a whole day troubleshooting this very weird issue... details on this thread: https://github.com/orgs/vercel/discussions/1180

The only workaround that worked for me is (you need to install raw-loader):

I added a type declaration for my template files in a d.ts file in my project:

declare module '*.mjml' {
  const content: string
  export default content
}

Then I added a new Webpack configuration to load these files using the raw-loader plugin which just mimics a readFile (this is in next.config.js):

webpack(config, context) {
  config.module.rules.push(
    {
      test: /\.mjml$/,
      use: 'raw-loader',
    }
  )
  return config
}

And now in my API file, I simply import the file so that instead of trying to use the static analysis to guess which files are used, it's just a normal Node dependency:

import emailTemplate from '../../emails/default.mjml'

[activenode]

That's essentially what I was pointing out here: #32236 (comment)

I feel your pain. I did the same. I just don't get how this has such low priority.

[nbouvrette]

Yes, I found your comment at about the same time that I had the same idea! It took so long to figure out what was going on and find anything on the topic. Hopefully they fix this bug soon but it didn't feel this was very high on the list based on my other Vercel discussion

[georgiosd]

Bump on this. This is the only solution that worked for me while trying to render some PDFs with pdfkit and it just wouldn't copy the fonts over...

[thomasvalera]

This discussion has been opened on Aug 5 2019 and today, 16th of Feb 2023 it is still an issue.

It's been 3,5 years, how is this still a thing?! @$#!&

[natterstefan]

@thomasvalera I just wanted to post the same. I do not see a path to fix this yet... Do you?

[thomasvalera]

Got a workaround working!
#32236 (comment)

[thomasvalera]

So, I've been at this for 3 days now and I have come across a few dirty solutions for it.

What we know is that api's run as "serverless functions" and to optimize them Vercel bundles the strict minimum which is why we don't have access to files, they are simply not bundled.

What seems to work most reliably is forcing Vercel to bundle the files by resolving the path to the files themselves with path.resolve()

In my case I'm generating an image with background images and custom font that are in a resources folder in the root.
My api looks like this

import { NextApiRequest, NextApiResponse } from "next";
import path from "path";

// Force files to be bundled
const BACKGROUND_IMAGE_PATH = path.resolve("./resources/bg_image.png");
const FONT_PATH = path.resolve("./resources/font.otf");

const handler = async (req: NextApiRequest, res: NextApiResponse) => {
...
}

export default handler;

[phbernard]

Thank you for this workaround, which totally makes sense. Unfortunately, it doesn't work for me. I think I followed your example carefully (path.resolve, main scope, the file appears in the corresponding .nft.json log file...), to no avail.

[estevaz12]

Works for me on both api routes and non-api serverless functions!

[MiguelTrinca]

This worked for me! I was trying to import an html file to send as an email. Might be the cleanest solution. Instead of using path.join(__dirname, "your-path-here") just do path.resolve("your-path-here")

[MrMegaMango]

doesn't work for dynamic route

[nmitic]

Not working for me :( I am doing the same, the only difference it that I have .json files, and have third party lib writing and reading those files. However, that should not matter, as all I need it to see those json files being bundled under .next dir....

[rsxdalv]

This might be useful to some, who just want to get to the "root" (I might update this with more testing).

const __next__base__dirname = __dirname.split(".next")[0]

then, for example

path.join(__next__base__dirname, "public")

[nmitic]

Unfortunately none of the noted workarounds works for me. In my case I have a third party lib which is both reading and writing to a file system inside route handler, ofc it works absolutely fine locally... but not when deployed on Vercel, I am experiencing the same error as the rest of the bunch.

[sroussey]

Can you read from your public folder?

[nmitic]

I am not able to to do that from route handler.

[nmitic]

Hm so I had an idea, very naive one I guess, that I can just use webpack plugin to copy file from source to build and that somehow read and write to it will be possible in api route. That does not seem to be the case 😖

[nmitic]

I got it to work by using /tmp dir. Most likely not something that would be of a use to most of you guys, but it work like we would expect it to work. I however have no idea how is this tmp being handled if it is being removed after a while.

Docs here >> https://vercel.com/guides/how-can-i-use-files-in-serverless-functions

[nightmarelie]

Perhaps this information could benefit someone.

We use serverless functions and the task was to switch to an app router. After the transition, a problem appeared with fonts. We have used page routers in the past and had no problems. We also didn't have any problems with performance and dev experience at that time, but that’s another story...🥲

    "next": "^14.1.0",
    "pdfkit": "0.14.0",

structure

├── app
│   ├── api
│   │   ├── reports
│   │   │   └── export-report
├── public
│   ├── fonts
│   │   ├── helvetica-bold.woff
│   │   └── helvetica.woff
├── next.config.js
├── libs
│   ├── documents
│   │   ├── document-helpers.ts
│   │   └── pdf-document.ts

1. Modify next.config.js according to https://nextjs.org/docs/app/api-reference/next-config-js/output#caveats

const nextConfig = {
  experimental: {
    outputFileTracingIncludes: {
      '/api/reports/export-report': ['./public/fonts/**/*'],
    },
    ...
}

2. Created document-helpers.ts according to https://vercel.com/guides/how-can-i-use-files-in-serverless-functions

import PDFKit from 'pdfkit';
import path from 'path';

export const FONT_FAMILY = 'Helvetica';
export const FONT_FAMILY_BOLD = 'Helvetica-Bold';

const PUBLIC_PATH = './public';
const FONTS_PATH = 'fonts';

export const fonts = {
  [FONT_FAMILY]: 'helvetica.woff',
  [FONT_FAMILY_BOLD]: 'helvetica-bold.woff',
};

export const registerFonts = (doc: PDFKit.PDFDocument) => {
  Object.entries(fonts).forEach(([name, fontFileName]) => {
    doc.registerFont(name, getFontPath(fontFileName));
  });
};

const resolveFontFolderPath = () => {
  return path.resolve(process.cwd(), PUBLIC_PATH, FONTS_PATH);
};

export const getFontPath = (fontFileName: string) => {
  const folderPath = resolveFontFolderPath();

  return path.join(folderPath, fontFileName);
};

[Paras-Jain-dev]

"Thanks, it proved effective with some adjustments. Referring to your project structure, I made necessary modifications in the existing next.config.mjs file. Consequently, I successfully accessed and utilized the files and images located within the public folder. This approach streamlined the process and ensured smooth integration within my project."

[laurent512]

I tried everything to allow my server actions ( "use server) to read a file stored in public folder without any success.
If someone achieved it and could share some explanations 🙏

[gmotta-dev]

4 years btw

[cmdruid]

Just ran into this issue.

So basically, vercel is caching my JSON files in a completely unrelated path, which is why all my JSON imports break upon deployment (and work inside a dev environment).

The solution seems to be to create some sort of 'hack' import step that tricks the vercel compiler to bundle the json files against its will. And this code has to be duplicated across all API endpoints.

It was foolish for me to assume that importing a JSON file (which has been standard esm spec for years) would be respected within vercel.

My solution has been to get rid of JSON entirely within my project. I am lucky that I can afford to do this. Others seem not so lucky. Judging by the age of this problem, vercel has no intention of supporting JSON imports.

Longer term, we are forced to migrate away from vercel, as this is one of many basic issues that they refuse to deal with.