New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Import d3 causes webpack dependency errors #2107

Closed
nwshane opened this Issue Sep 13, 2017 · 29 comments

Comments

Projects
None yet
@nwshane
Contributor

nwshane commented Sep 13, 2017

Adding the line import * as d3 from 'd3' to any JS file in my src code causes the following errors:

ERROR  Failed to compile with 2 errors

These dependencies were not found:

* child_process in ./~/xmlhttprequest/lib/XMLHttpRequest.js
* fs in ./~/xmlhttprequest/lib/XMLHttpRequest.js

To install them, you can run: npm install --save child_process fs

Could this be a bug caused by Gatsby's webpack config? I don't understand what's going on here: Why is XMLHttpRequest throwing an error when it tries to require built in node modules like fs and child_process, and why would these errors be triggered by including d3?

More info

gatsby-config.js

module.exports = {
  siteMetadata: {
    title: 'NathanShane.me',
    author: 'Nathan Shane',
    description: require('./package.json').description
  },
  plugins: [
    'gatsby-plugin-react-helmet',
    'gatsby-plugin-catch-links',
    'gatsby-transformer-sharp',
    'gatsby-plugin-offline',
    {
      resolve: `gatsby-source-filesystem`,
      options: {
        path: `${__dirname}/src/pages/projects`,
        name: 'projects',
      },
    },
    {
      resolve: `gatsby-source-filesystem`,
      options: {
        path: `${__dirname}/src/pages/blog`,
        name: 'blogPosts',
      },
    },
    {
      resolve: 'gatsby-transformer-remark',
      options: {
        plugins: [
          {
            resolve: `gatsby-remark-images`,
            options: {
              maxWidth: 590
            }
          },
          'gatsby-remark-copy-linked-files'
        ]
      }
    }
  ]
}

gatsby-node.js:

const {resolve} = require('path')

exports.modifyBabelrc = ({ babelrc }) => {
  return {
    plugins: babelrc.plugins.concat([
      ['babel-plugin-root-import']
    ]),
  }
}

exports.createPages = ({boundActionCreators, graphql}) => {
  const {createPage} = boundActionCreators

  return graphql(
    `
    {
      allMarkdownRemark(limit: 1000) {
        edges {
          node {
            fileAbsolutePath
            frontmatter {
              slug
            }
          }
        }
      }
    }
    `
  ).then((result) => {
    if (result.errors) {
      return Promise.reject(result.errors)
    }
    const nodes = result.data.allMarkdownRemark.edges

    const filterByPathIncludes = (testStr, nodes) => (
      nodes.filter(
        ({node: {fileAbsolutePath}}) => (fileAbsolutePath.includes(testStr))
      )
    )

    const projects = filterByPathIncludes('/pages/projects/', nodes)

    projects.forEach(({node}) => {
      const {slug} = node.frontmatter
      createPage({
        path: `/projects/${slug}`,
        component: resolve('./src/templates/project.js'),
        context: {
          slug
        }
      })
    })

    const blogPosts = filterByPathIncludes('/pages/blog/', nodes)

    blogPosts.forEach(({node}) => {
      const {slug} = node.frontmatter
      createPage({
        path: `/blog/${slug}`,
        component: resolve('./src/templates/blogPost.js'),
        context: {
          slug
        }
      })
    })
  }).catch((error) => {
    console.log(error)
  })
}

package.json:

{
  "name": "nathan_shane_site",
  "description": "Nathan Shane's portfolio website",
  "repository": "https://github.com/nwshane/nwshane.github.io.git",
  "license": "MIT",
  "scripts": {
    "dev": "gatsby develop",
    "build": "gatsby build",
    "start": "gatsby serve"
  },
  "dependencies": {
    "babel-plugin-root-import": "^5.1.0",
    "d3": "^4.10.2",
    "gatsby": "^1.8.11",
    "gatsby-link": "^1.4.1",
    "gatsby-plugin-catch-links": "^1.0.8",
    "gatsby-plugin-offline": "^1.0.9",
    "gatsby-plugin-react-helmet": "^1.0.6",
    "gatsby-remark-copy-linked-files": "^1.5.7",
    "gatsby-remark-images": "^1.5.10",
    "gatsby-source-filesystem": "^1.4.12",
    "gatsby-transformer-remark": "^1.7.6",
    "gatsby-transformer-sharp": "^1.6.5",
    "normalize.css": "^7.0.0",
    "ramda": "^0.24.1",
    "styled-components": "^2.1.2",
    "webfontloader": "^1.6.28"
  }
}

Repo and branch in which error is occuring: https://github.com/nwshane/nwshane.github.io/tree/blog

Gatsby version: 1.9.21
NodeJS version: 8.4.0
OS version: Mac OS X 10.12

@KyleAMathews

This comment has been minimized.

Contributor

KyleAMathews commented Sep 14, 2017

Probably you installed the d3-node module by accident which then webpack is choking on trying to import as it's trying to import a bunch of node only modules.

@jquense

This comment has been minimized.

Collaborator

jquense commented Sep 14, 2017

its interesting that the error is so misleading, maybe we can fix it. e.g. if the target is not Node in webpack we shouldn't recommend installing core node modules.

@nwshane

This comment has been minimized.

Contributor

nwshane commented Sep 14, 2017

@KyleAMathews I don't think that's the case. The d3 documentation itself recommends using import * as d3 from "d3";, which is what I'm doing. I don't have d3-node in my package.json dependencies and I'm not requiring it anywhere.

If you have any ideas about how to debug this issue, that'd be very helpful.

@revolunet

This comment has been minimized.

revolunet commented Sep 15, 2017

looks like this xmlhttprequest module is not webpack friendly :/

related : driverdan/node-XMLHttpRequest#127

@nwshane

This comment has been minimized.

Contributor

nwshane commented Sep 15, 2017

@revolunet Why is xmlhttprequest getting required by d3 in the first place?

@revolunet

This comment has been minimized.

revolunet commented Sep 16, 2017

no idea , could be some other module

@haroldtreen

This comment has been minimized.

haroldtreen commented Sep 17, 2017

Some findings:

  • d3 requires d3-request code
  • d3-request requires xmlhttprequest code
  • xmlhttprequest requires child-process and fs code
  • d3-request builds two version - one for node and one min.
  • The default export is the node version.
  • Changing the main module of d3-request from build/d3-request.node.js --> build/d3-request.min.js fixes the import.
  • Using the modifyWebpackConfig to include: node: { fs: 'empty', child_process: 'empty' } fixes the issue.

Strange though because this issue seems to not appear when using a very plain webpack.config.js. Trying to find out what webpack differences are triggering the error.

@haroldtreen

This comment has been minimized.

haroldtreen commented Sep 17, 2017

The plain webpack was version 3. Using the gatsby webpack breaks it.

It would be interesting to talk to d3-request and find out why the node version if the main module...

@haroldtreen

This comment has been minimized.

haroldtreen commented Sep 17, 2017

Known d3-request issue. That module is expecting bundlers to use the module field of package.json instead of main.

d3/d3-request#24

@haroldtreen

This comment has been minimized.

haroldtreen commented Sep 17, 2017

d3-request deleted a field called browser that used to direct webpack to the correct non-node index file. Now d3-request expects the module field to be used.

Commit: d3/d3-request@d635b89#diff-b9cfc7f2cdf78a7f4b91a753d10865a2L19

Solution:

Add this to the webpack config:

{
   resolve: {
      packageMains: [
        'module', // adds check for 'module'
        'webpack',
        'browser',
        'web',
        'browserify',
        ['jam', 'main'],
        'main',
    ]
   }
}

See packageMains config documentation

@hermionewy

This comment has been minimized.

hermionewy commented Oct 30, 2017

I added the packageMains, but still not working. I use Webpack1. Anyone know what to do to get rid of the XMLHttpRequest and fs error?

@mattjstar

This comment has been minimized.

mattjstar commented Nov 8, 2017

yep, getting the same issue as @hermionewy when using @haroldtreen 's solution with packageMains. iTerm output:

$ gatsby develop
success delete html files from previous builds — 0.010 s
success open and validate gatsby-config.js — 0.003 s
info One or more of your plugins have changed since the last time you ran Gatsby. As
a precaution, we're deleting your site's cache to ensure there's not any stale
data
success copy gatsby files — 0.013 s
success onPreBootstrap — 0.004 s
success source and transform nodes — 0.015 s
success building schema — 0.064 s
success createLayouts — 0.024 s
success createPages — 0.004 s
success createPagesStatefully — 0.011 s
success onPreExtractQueries — 0.001 s
success update schema — 0.044 s
success extract queries from components — 0.057 s
success run graphql queries — 0.013 s
success write out page data — 0.003 s
success write out redirect data — 0.001 s
success onPostBootstrap — 0.000 s

info bootstrap finished - 1.505 s

error There was an error compiling the html.js component for the development server.

See our docs page on debugging HTML builds for help https://goo.gl/yL9lND


  WebpackError: Unexpected token import






  - reactProdInvariant.js:34 Compilation.<anonymous>
    ~/react/lib/reactProdInvariant.js:34:1

  - develop-static-entry.js:3 Compilation.next
    .cache/develop-static-entry.js:3:1



  - reactProdInvariant.js:30 Compilation.<anonymous>
    ~/react/lib/reactProdInvariant.js:30:1

  - develop-static-entry.js:3 Compilation.next
    .cache/develop-static-entry.js:3:1

  - React.js:127 ExtractTextPlugin.<anonymous>
    ~/react/lib/React.js:127:1


  - React.js:78 Object.async.forEachOf.async.eachOf
    ~/react/lib/React.js:78:1

  - React.js:51 Object.async.forEach.async.each
    ~/react/lib/React.js:51:1

  - React.js:79 ExtractTextPlugin.<anonymous>
    ~/react/lib/React.js:79:1


error Command failed with exit code 1.

I was able to get it to work with this, but I'm afraid that will have unintended consequences down the line.

exports.modifyWebpackConfig = ({ config, stage }) => {
  config.merge({
     node: { fs: 'empty', child_process: 'empty' },
  })

  return config;
};
@jquense

This comment has been minimized.

Collaborator

jquense commented Nov 8, 2017

No i think that's the right way to handle this @mattjstar we should make those the defaults for v2 ( #2641)

@richardwestenra

This comment has been minimized.

richardwestenra commented Nov 9, 2017

I'm getting a similar error with fs being required by dotenv, which I presume showed up after reinstalling my node modules a couple of days ago, but I just noticed it today:

 ERROR  Failed to compile with 1 errors 

This dependency was not found:

* fs in ./~/dotenv/lib/main.js

To install it, you can run: npm install --save fs

@mattjstar's fix works for me too.

EDIT: No it didn't - it stopped the error messages from appearing, but then dotenv stopped working.

@KyleAMathews

This comment has been minimized.

Contributor

KyleAMathews commented Nov 9, 2017

@richardwestenra is this error happening in the browser? If so, know that dotenv is only useful in node.js code so shouldn't be required in the browser.

@richardwestenra

This comment has been minimized.

richardwestenra commented Nov 9, 2017

@KyleAMathews Nope, this error is happening in the terminal, confusingly.

I've tried moving it to devDependencies, as well as setting

  config.merge({
    target: 'node'
  });

in the webpack config, but have had no success 😕

@KyleAMathews

This comment has been minimized.

Contributor

KyleAMathews commented Nov 9, 2017

Webpack is only for browser code so unless you're requiring dotenv in browser code modifying the webpack config isn't going to help with node code. Confusing I know :-)

@richardwestenra

This comment has been minimized.

richardwestenra commented Nov 9, 2017

ah okay, that helps narrow things down a bit. Does that mean it's unlikely to be a Gatsby-related issue, then? I'm pretty much stumped for ideas.

I'm using dotenv to populate Contentful/Workable API keys used in plugins in gatsby-config.js, having initialised dotenv at the start of that file.

@KyleAMathews

This comment has been minimized.

Contributor

KyleAMathews commented Nov 9, 2017

Perhaps try rm -r node_modules && yarn? :-)

But yeah, this is either an install problem or a dotenv problem. Node.js is somehow failing to require the built-in fs module.

@richardwestenra

This comment has been minimized.

richardwestenra commented Nov 11, 2017

Have tried that several times, and it doesn't look like dotenv has published any releases in a year. I'll keep looking. Thanks for the assistance! :)

@richardwestenra

This comment has been minimized.

richardwestenra commented Nov 13, 2017

Okay I figured it out! I'm unsure whether it's relevant to others' issues posted above, but I'll post details of my solution here in case it helps anyone.

It turns out, the issue showed up when I started including siteMetadata.title from gatsby-config in my index layout file. I'm not sure what the siteMetadata prop is supposed to be used for - I found a reference to page components having access to it with props.data.site.siteMetadata, but that props.data isn't showing up for me. Anyway it seemed like using that property for the site title would likely be best practice, so I included it in the layout template with import { siteMetadata } from '../../gatsby-config', then used it to populate <title> with React Helmet. However I guess this caused gatsby-config started to be included in Webpack browser code, which doesn't have access to fs, so it started throwing errors.

Is there a recommended way of including gatsby-config's siteMetadata prop in browser code? If not, what is its purpose?

@KyleAMathews

This comment has been minimized.

Contributor

KyleAMathews commented Nov 13, 2017

@richardwestenra

This comment has been minimized.

richardwestenra commented Nov 13, 2017

ah okay, great, thanks! I didn't spot that, but I guess it'll be included in the documentation when the GraphQL docs page is done.

Thanks again for your help!

@parvezk

This comment has been minimized.

parvezk commented Nov 19, 2017

You can also follow this approach to fix the XMLHTTPRequest issue with module bundling:
webpack/webpack-dev-server#66

@kwelch

This comment has been minimized.

kwelch commented Mar 27, 2018

I am seeing the same fs error post-installing dotenv. Does anyone have version 1.9.223 working with dotenv?

@ohenrik

This comment has been minimized.

ohenrik commented Apr 2, 2018

@haroldtreen where should i add this? If i add packageMains to resolve i get an error telling me that packageMains is unknown.

@jhlabs

This comment has been minimized.

jhlabs commented May 10, 2018

I am facing exactly the same error as @richardwestenra when installing the gatsby-plugin-mailchimp and using dotenv. Is there a way to resolve it in that case?

@kakadiadarpan

This comment has been minimized.

Contributor

kakadiadarpan commented Sep 28, 2018

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub issues, we have to clean some of the old issues as many of them have already been resolved with the latest updates.

Please make sure to update to the latest Gatsby version and check if that solves the issue. Let us know if that works for you by adding a comment 👍

@kakadiadarpan

This comment has been minimized.

Contributor

kakadiadarpan commented Oct 10, 2018

This issue is being closed because there hasn't been any activity for at least 30 days. Feel free to open a new one if you still experience this problem 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment