Next build fails with empty .mdx-data folder

[leerob]

Hey! Love the project. Trying to convert my MDX blog over to this. Hit a snag while trying to deploy. I'm seeing this error when I run next build.

➜  next build
yarn run v1.12.3
$ next build
> Using external babel configuration
> Location: "/repo/.babelrc"
Creating an optimized production build  

Failed to compile.

./pages/blog.js
Module not found: Can't resolve '/repo/.mdx-data/01bcfb81f06ba1b626478d4440916332.json' in '/repo/pages'


> Build error occurred
Error: > Build failed because of webpack errors
    at build (/repo/node_modules/next/dist/build/index.js:12:900)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

If I then immediately retry the build, it works as expected. Any idea what this could be? It seems like if the .mdx-data folder does not exist at all (which is what happens when I try to deploy) then it errors out.

I can try and get a reproducible example up if that would be helpful.

[leerob]

Large reproducible example -> leerob/leerob.io#159

After bringing that down to the absolute minimum working example, which I had originally started with, I started to piece it back together. By the process of elimination of deleting code and modules, I found the smoking gun to be Firebase.

I think by adding that dependency it's bumping some Babel things, which is breaking this. Here is the absolute bare minimum reproducible example. First commit is in a working state, second commit the build fails after adding firebase as a dependency.

https://github.com/leerob/mdx-enhanced-issue-repro

[leerob]

This seems related -> vercel/next.js#296. Maybe the JSON file isn't available on the server, which is why then once .mdx-data folder is created, the build passes the second time.

[leerob]

@jescalan @joe-bell Sorry for the rambling of messages here. Any chance you could provide me with any additional info to debug or any ideas you might have? 🙏

[joe-bell]

@leerob hey Lee! I'm not actually a maintainer but I've just hit a similar issue to you. My solution was a quick flush of the build directories:

rm -rf .next out .mdx-data

[leerob]

@joe-bell My bad, thanks for chiming in, though! I believe this issue is the exact opposite of that one. The root problem is the .mdx-data folder does not exist before we run next build. Subsequent runs of next build work as expected because the folder is already there.

[leerob]

This isn't the greatest workaround, but it works. I just run the build script twice 🤷‍♂

"build": "next build ; next build"

[NateRadebaugh]

I'm also hitting this, causing my "zeit now" deployments to fail.

11:20:06.612  Module not found: Can't resolve '/zeit/4756b631/.mdx-data/030ee89fb6f7f9ddf334677dba207de9.json' in '/zeit/4756b631/src/layouts'

I'd hate double my build-time to work around this bug, but I guess it's better than broken deployments...

[jescalan]

Hi! I'm so sorry I was behind on notifications and just seeing this message. The error messages I am seeing above, I do not think are happening because the .mdx-data folder isn't present, I would suspect they are based on something different.

Thanks so much for providing that repro though, this is a life saver. I'm going to take a look and get back to you all here. Sorry again for the late response!

[jescalan]

Confirmed there's an issue here, working on tracking it down. It may have been some changes to webpack? When I run this, the file that it says is not present does seem to be present, and all the file writes are set to happen before any nextjs processing begins, so I'm confused by how this is possible.

[leerob]

@jescalan No worries at all, thrilled you're willing to take a look. I appreciate all the hard work that went into this library. It's been a lifesaver for me!

[leerob]

Confirmed there's an issue here, working on tracking it down. It may have been some changes to webpack?

That was my hunch as well. Very strange. I tried to look at the yarn.lock differences after updating Firebase, but nothing stood out to me.

[jescalan]

Ok @leerob I think I have tracked down the issue here, it was pretty nasty.

The root problem is that the layout file was not returning the right thing. Layouts need to return a function, the way it shows in the readme for this repo and for mdx itself. This is not at all intuitive and I can't blame you for missing this though, so I do think I should probably make sure this library looks for the export format of the layout and flags it with a more clear error when this type of thing happens.

Changing the layout file in your repro to the following:

export default function Layout(frontMatter) {
  return ({ children: content }) => {
    return <div>{content}</div>;
  };
}

...then clearing the .mdx-data and .next folders is giving me a clean build consistently. I tried deleting just the .mdx-data folder and not .next intermittently along with making random changes to various files, and I was able to reproduce that same error a couple times in my tinkering, but every time I did, deleting the cache (which can be done by removing .next, or more specifically .next/cache/next-babel-loader) would get me back to a clean build.

The error above was coming up not because it was the actual error, but because of a strange babel caching problem. Clearing the cache entirely and running a fresh build would turn up an alternate error straight out of react that eventually led me back to the layout problem.

I think overall the lesson here is that I need to be stronger in my recommendation that using this plugin means clearing the babel cache on build. There are a variety of babel cache issues that root from the unusual nature of the babel plugin manipulation we need to do to make this work, and its dependence on filesystem changes (more detail in this issue).

If it makes any difference, I modded npm start to remove the cache on all my team's projects a while ago. We have about 5 separate projects that rely entirely on this plugin with over 100 distinct people committing to them, several thousand pages overall, and are pulling down 1m+/month views in production, and we haven't had one report of this issue. I think based on this I can say fairly confidently that clearing the cache on build and formatting the layout properly will ensure this doesn't happen.

Before I take this on myself, want to open up the opportunity to add the enhanced "layout return format mistake" error message to the library via PR if anyone here is interested in contributing. If I get no bites, I'll add something in here maybe later this week.

While I'm here want to also mention that we are cooking up an alternative to this library that we are pretty excited by, and there will be more info on this soon 😀

[leerob]

I'm still seeing the issue after switching the layout to this:

import React from 'react';

export default function Layout(frontMatter) {
  return ({ children: content }) => {
    return <div>{content}</div>;
  };
}

1. On my example repro, I deleted node_modules, .next, and .mdx-data
2. Ran yarn and yarn build
3. Same issue

Are you not seeing that?

[jescalan]

Hmm, I changed a few other things as well and dropped a bunch of unused dependencies in the debugging process, this was the step that ultimately did it. Let me clone a fresh copy and try again

[jescalan]

@leerob it was indeed some of my dependency changes, pretty crazy. I'm not sure which dependency it was that was causing the interference, but stripping out unneeded dependencies is giving me a clean build, where with the dependencies you originally had listed, I was seeing the error.

I made a PR to your reproduction repo with my changes - hopefully this helps!

[jescalan]

Hey hey, going to close this one out as I haven't heard anything back. Let me know if there's anything that still needs to be addressed here and I'd be happy to re-open!

[QuanTumli]

Hi @jescalan I can confirm the same issue as @leerob.

See my repo for reproduction: https://github.com/QuanTumli/mdx-enhanced-issue

"dependencies": {
    "next": "9.3.6",
    "next-mdx-enhanced": "^2.5.0",
    "react": "16.13.1",
    "react-dom": "16.13.1"
  }

(Project was initialized with yarn create next-app, then I added the minimal example of next-mdx-enhanced)

Steps to reproduce the error:

1. Clone my repo (or create minimal app with yarn create next-app and add next-mdx-enhanced example)
2. Run yarn to install deps
3. Run yarn build -> There the error appears. (Exactly the same the one of @leerob)

Failed to compile.

./pages/index.js
Module not found: Can't resolve '/my-app/.mdx-data/540043c61436029cd7e7ce19630eeb32.json' in '/my-app/pages'

The .mdx-data/ folder is created during yarn build, so any subsequent yarn build will run successfully.

[leerob]

Sorry for my slow response!

While your PR did have a fix, as soon as I added back the dependencies, I saw the same error as before. Going back to my initial message, that was why I pinned down updating Firebase specifically that seemed to mess it up.

[franky47]

I'm hitting the same issue (see CI build), something interesting is that the "missing" JSON file does exist in .mdx-data at the moment the error is thrown.

The issue seems to come from the front matter import feature:

• import { frontMatter as blogPosts } from './**/*.mdx => clean build fails
• import { frontMatter } from './a-single-blog-post.mdx => clean build fails
• No frontMatter imports => clean build passes

[evanfrawley]

Confirming this as well -- clean build locally & on vercel fail with either import style of frontMatter and passes without. Same errors occur as above.

However, I have found that if I specify the exact version in package.json the clean build seems to work. This was after removing node_modules yarn.lock .next and .mdx-data and running yarn and yarn build

"dependencies": {
    ...
    "next": "9.4.0",
    "next-mdx-enhanced": "3.0.0",
    ...
}

[franky47]

Pinning next-mdx-enhanced to 3.0.0 solves it for me too, thanks @evanfrawley !

Edit: it has more to do with regenerating the lockfile than pinning the version, when changing it back to ^3.0.0 the clean build still passes on my local machine, but not on CI (that's probably a separate issue though).

[jurecuhalev]

I'm seeing the same issue as @franky47 . I can get local version building correctly after removing local cached folders, but rebuilding from clear cache on Netlify still produces clean build error.

[franky47]

I also get a race-condition kind of error, which does not happen all the time, and could be related to this issue:

Failed to compile.

./.mdx-data/8c439756be33782fad8d921d7a445501.json
Module parse failed: Unexpected end of JSON input while parsing near ''
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
SyntaxError: Unexpected end of JSON input while parsing near ''
    at JSON.parse (<anonymous>)

Looks like the JSON file is being written while Next.js/Webpack is trying to load it. This happens on non-clean builds, and it takes up to 4 or 5 retries to get the build to pass sometimes.

[talentlessguy]

Getting the similar to @franky47's error:

2020-06-05T20:24:14.989Z  Failed to compile.
2020-06-05T20:24:14.989Z  
2020-06-05T20:24:14.989Z  ./pages/blog/index.tsx
2020-06-05T20:24:14.989Z  Module not found: Can't resolve '/vercel/183922fb/.mdx-data/05d247cc14968ece54ac0fa1c4c4a5f4.json' in '/vercel/183922fb/pages/blog'
2020-06-05T20:24:14.989Z  
2020-06-05T20:24:14.989Z  
2020-06-05T20:24:14.989Z  > Build error occurred
2020-06-05T20:24:14.990Z  Error: > Build failed because of webpack errors
2020-06-05T20:24:14.990Z      at build (/vercel/183922fb/node_modules/next/dist/build/index.js:13:917)
2020-06-05T20:24:15.016Z  error Command failed with exit code 1.
2020-06-05T20:24:15.016Z  info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
2020-06-05T20:24:15.027Z  Error: Command "yarn run build" exited with 1
2020-06-05T20:24:16.614Z  Done with "package.json"

I tried these hacks:

• next build; next build in build script - doesn't work
• 3.0.0 version - doesn't work either

[daneden]

FWIW I’m experiencing this issue as well on my site’s repo. I hated the idea of always running the build step twice so changed the build step to next build || next build which will only run build a second time if the first one fails (i.e. if the build cache isn’t carried over from previous builds).

[jescalan]

This should be resolved by the latest release!

[franky47]

Hey, sorry I didn't report it back sooner, but yes the issue has been solved for a few weeks in my case. Thanks !

[ivannovazzi]

Interesting enough, version 5.0.0 does not fix the issue for me 😔

Notes

I have to be clear on a number of things though:

• versions 3.0.0, 4.0.0, 5.0.0 all cause the same issue
• I am experiencing the problem not only with the build but also during dev mode - given I am experiencing this locally so I didn't even try on CI
• I have been trying with multiple next@9.5.6 canary versions, haven't tried with next@10.0.0
• preparing a clean or not clean context doesn't make any difference, nor reinstalling the node_modules
• sequential builds did not work

Findings

• The .mdx-data folder is being populated with JSON files for all the .mdx files, except the catch-them-all one

import { frontMatter as blogPosts } from 'pages/posts/**/*.mdx';

• I can see the logs showing the event - compiled successfully sometime after the previous error log line is produced

event - build page: /blog
wait  - compiling...
error - ./pages/blog.js
Module not found: Can't resolve '/Users/ivan/Projects/personal/ivan-website/.next/cache/mdx-data/85a9f12bff72bcf2394b051245538228.json' in '/Users/ivan/Projects/personal/ivan-website/pages'
event - compiled successfully

My project has a number of dependencies, played a bit with some of them but can't really tell anything did any difference.

[jescalan]

@ivannovazzi - the 5.0 release does not use .mdx-data at all, it can be removed as noted in the release notes

If you can make me a minimal reproduction of the issue here I would be happy to look!

[ivannovazzi]

@jescalan well, the folder is moved under .next/cache and the same exact behaviour is observed. I will put together a minimal reproducible scenario asap.

[jescalan]

Thanks so much @ivannovazzi - looking forward to it!

[ivannovazzi]

My apologies @jescalan , the plugin is behaving perfectly and congrats for building this!

In case someone else runs into this exact issue, the import shown above was causing troubles because of the missing babel plugin babel-plugin-import-glob-array; I didn't have it installed and didn't have a .babelrc either.

I should have smelled something was wrong on my end since I had noticed the **/*.mdx and it appeared strange to me at first sight since I had never used the glob array in any project earlier.

[jescalan]

Thats great to hear - so glad you figured it out!