Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Import d3 causes webpack dependency errors #2107

Closed
nwshane opened this issue Sep 13, 2017 · 29 comments
Closed

Import d3 causes webpack dependency errors #2107

nwshane opened this issue Sep 13, 2017 · 29 comments

Comments

@nwshane
Copy link
Collaborator

@nwshane nwshane commented Sep 13, 2017

Adding the line import * as d3 from 'd3' to any JS file in my src code causes the following errors:

ERROR  Failed to compile with 2 errors

These dependencies were not found:

* child_process in ./~/xmlhttprequest/lib/XMLHttpRequest.js
* fs in ./~/xmlhttprequest/lib/XMLHttpRequest.js

To install them, you can run: npm install --save child_process fs

Could this be a bug caused by Gatsby's webpack config? I don't understand what's going on here: Why is XMLHttpRequest throwing an error when it tries to require built in node modules like fs and child_process, and why would these errors be triggered by including d3?

More info

gatsby-config.js

module.exports = {
  siteMetadata: {
    title: 'NathanShane.me',
    author: 'Nathan Shane',
    description: require('./package.json').description
  },
  plugins: [
    'gatsby-plugin-react-helmet',
    'gatsby-plugin-catch-links',
    'gatsby-transformer-sharp',
    'gatsby-plugin-offline',
    {
      resolve: `gatsby-source-filesystem`,
      options: {
        path: `${__dirname}/src/pages/projects`,
        name: 'projects',
      },
    },
    {
      resolve: `gatsby-source-filesystem`,
      options: {
        path: `${__dirname}/src/pages/blog`,
        name: 'blogPosts',
      },
    },
    {
      resolve: 'gatsby-transformer-remark',
      options: {
        plugins: [
          {
            resolve: `gatsby-remark-images`,
            options: {
              maxWidth: 590
            }
          },
          'gatsby-remark-copy-linked-files'
        ]
      }
    }
  ]
}

gatsby-node.js:

const {resolve} = require('path')

exports.modifyBabelrc = ({ babelrc }) => {
  return {
    plugins: babelrc.plugins.concat([
      ['babel-plugin-root-import']
    ]),
  }
}

exports.createPages = ({boundActionCreators, graphql}) => {
  const {createPage} = boundActionCreators

  return graphql(
    `
    {
      allMarkdownRemark(limit: 1000) {
        edges {
          node {
            fileAbsolutePath
            frontmatter {
              slug
            }
          }
        }
      }
    }
    `
  ).then((result) => {
    if (result.errors) {
      return Promise.reject(result.errors)
    }
    const nodes = result.data.allMarkdownRemark.edges

    const filterByPathIncludes = (testStr, nodes) => (
      nodes.filter(
        ({node: {fileAbsolutePath}}) => (fileAbsolutePath.includes(testStr))
      )
    )

    const projects = filterByPathIncludes('/pages/projects/', nodes)

    projects.forEach(({node}) => {
      const {slug} = node.frontmatter
      createPage({
        path: `/projects/${slug}`,
        component: resolve('./src/templates/project.js'),
        context: {
          slug
        }
      })
    })

    const blogPosts = filterByPathIncludes('/pages/blog/', nodes)

    blogPosts.forEach(({node}) => {
      const {slug} = node.frontmatter
      createPage({
        path: `/blog/${slug}`,
        component: resolve('./src/templates/blogPost.js'),
        context: {
          slug
        }
      })
    })
  }).catch((error) => {
    console.log(error)
  })
}

package.json:

{
  "name": "nathan_shane_site",
  "description": "Nathan Shane's portfolio website",
  "repository": "https://github.com/nwshane/nwshane.github.io.git",
  "license": "MIT",
  "scripts": {
    "dev": "gatsby develop",
    "build": "gatsby build",
    "start": "gatsby serve"
  },
  "dependencies": {
    "babel-plugin-root-import": "^5.1.0",
    "d3": "^4.10.2",
    "gatsby": "^1.8.11",
    "gatsby-link": "^1.4.1",
    "gatsby-plugin-catch-links": "^1.0.8",
    "gatsby-plugin-offline": "^1.0.9",
    "gatsby-plugin-react-helmet": "^1.0.6",
    "gatsby-remark-copy-linked-files": "^1.5.7",
    "gatsby-remark-images": "^1.5.10",
    "gatsby-source-filesystem": "^1.4.12",
    "gatsby-transformer-remark": "^1.7.6",
    "gatsby-transformer-sharp": "^1.6.5",
    "normalize.css": "^7.0.0",
    "ramda": "^0.24.1",
    "styled-components": "^2.1.2",
    "webfontloader": "^1.6.28"
  }
}

Repo and branch in which error is occuring: https://github.com/nwshane/nwshane.github.io/tree/blog

Gatsby version: 1.9.21
NodeJS version: 8.4.0
OS version: Mac OS X 10.12

@KyleAMathews
Copy link
Contributor

@KyleAMathews KyleAMathews commented Sep 14, 2017

Probably you installed the d3-node module by accident which then webpack is choking on trying to import as it's trying to import a bunch of node only modules.

@jquense
Copy link
Collaborator

@jquense jquense commented Sep 14, 2017

its interesting that the error is so misleading, maybe we can fix it. e.g. if the target is not Node in webpack we shouldn't recommend installing core node modules.

@nwshane
Copy link
Collaborator Author

@nwshane nwshane commented Sep 14, 2017

@KyleAMathews I don't think that's the case. The d3 documentation itself recommends using import * as d3 from "d3";, which is what I'm doing. I don't have d3-node in my package.json dependencies and I'm not requiring it anywhere.

If you have any ideas about how to debug this issue, that'd be very helpful.

@revolunet
Copy link

@revolunet revolunet commented Sep 15, 2017

looks like this xmlhttprequest module is not webpack friendly :/

related : driverdan/node-XMLHttpRequest#127

@nwshane
Copy link
Collaborator Author

@nwshane nwshane commented Sep 15, 2017

@revolunet Why is xmlhttprequest getting required by d3 in the first place?

@revolunet
Copy link

@revolunet revolunet commented Sep 16, 2017

no idea , could be some other module

@haroldtreen
Copy link

@haroldtreen haroldtreen commented Sep 17, 2017

Some findings:

  • d3 requires d3-request code
  • d3-request requires xmlhttprequest code
  • xmlhttprequest requires child-process and fs code
  • d3-request builds two version - one for node and one min.
  • The default export is the node version.
  • Changing the main module of d3-request from build/d3-request.node.js --> build/d3-request.min.js fixes the import.
  • Using the modifyWebpackConfig to include: node: { fs: 'empty', child_process: 'empty' } fixes the issue.

Strange though because this issue seems to not appear when using a very plain webpack.config.js. Trying to find out what webpack differences are triggering the error.

@haroldtreen
Copy link

@haroldtreen haroldtreen commented Sep 17, 2017

The plain webpack was version 3. Using the gatsby webpack breaks it.

It would be interesting to talk to d3-request and find out why the node version if the main module...

@haroldtreen
Copy link

@haroldtreen haroldtreen commented Sep 17, 2017

Known d3-request issue. That module is expecting bundlers to use the module field of package.json instead of main.

d3/d3-request#24

@haroldtreen
Copy link

@haroldtreen haroldtreen commented Sep 17, 2017

d3-request deleted a field called browser that used to direct webpack to the correct non-node index file. Now d3-request expects the module field to be used.

Commit: d3/d3-request@d635b89#diff-b9cfc7f2cdf78a7f4b91a753d10865a2L19

Solution:

Add this to the webpack config:

{
   resolve: {
      packageMains: [
        'module', // adds check for 'module'
        'webpack',
        'browser',
        'web',
        'browserify',
        ['jam', 'main'],
        'main',
    ]
   }
}

See packageMains config documentation

@hermionewy
Copy link

@hermionewy hermionewy commented Oct 30, 2017

I added the packageMains, but still not working. I use Webpack1. Anyone know what to do to get rid of the XMLHttpRequest and fs error?

@mattjstar
Copy link

@mattjstar mattjstar commented Nov 8, 2017

yep, getting the same issue as @hermionewy when using @haroldtreen 's solution with packageMains. iTerm output:

$ gatsby develop
success delete html files from previous builds — 0.010 s
success open and validate gatsby-config.js — 0.003 s
info One or more of your plugins have changed since the last time you ran Gatsby. As
a precaution, we're deleting your site's cache to ensure there's not any stale
data
success copy gatsby files — 0.013 s
success onPreBootstrap — 0.004 s
success source and transform nodes — 0.015 s
success building schema — 0.064 s
success createLayouts — 0.024 s
success createPages — 0.004 s
success createPagesStatefully — 0.011 s
success onPreExtractQueries — 0.001 s
success update schema — 0.044 s
success extract queries from components — 0.057 s
success run graphql queries — 0.013 s
success write out page data — 0.003 s
success write out redirect data — 0.001 s
success onPostBootstrap — 0.000 s

info bootstrap finished - 1.505 s

error There was an error compiling the html.js component for the development server.

See our docs page on debugging HTML builds for help https://goo.gl/yL9lND


  WebpackError: Unexpected token import






  - reactProdInvariant.js:34 Compilation.<anonymous>
    ~/react/lib/reactProdInvariant.js:34:1

  - develop-static-entry.js:3 Compilation.next
    .cache/develop-static-entry.js:3:1



  - reactProdInvariant.js:30 Compilation.<anonymous>
    ~/react/lib/reactProdInvariant.js:30:1

  - develop-static-entry.js:3 Compilation.next
    .cache/develop-static-entry.js:3:1

  - React.js:127 ExtractTextPlugin.<anonymous>
    ~/react/lib/React.js:127:1


  - React.js:78 Object.async.forEachOf.async.eachOf
    ~/react/lib/React.js:78:1

  - React.js:51 Object.async.forEach.async.each
    ~/react/lib/React.js:51:1

  - React.js:79 ExtractTextPlugin.<anonymous>
    ~/react/lib/React.js:79:1


error Command failed with exit code 1.

I was able to get it to work with this, but I'm afraid that will have unintended consequences down the line.

exports.modifyWebpackConfig = ({ config, stage }) => {
  config.merge({
     node: { fs: 'empty', child_process: 'empty' },
  })

  return config;
};
@jquense
Copy link
Collaborator

@jquense jquense commented Nov 8, 2017

No i think that's the right way to handle this @mattjstar we should make those the defaults for v2 ( #2641)

@richardwestenra
Copy link
Collaborator

@richardwestenra richardwestenra commented Nov 9, 2017

I'm getting a similar error with fs being required by dotenv, which I presume showed up after reinstalling my node modules a couple of days ago, but I just noticed it today:

 ERROR  Failed to compile with 1 errors 

This dependency was not found:

* fs in ./~/dotenv/lib/main.js

To install it, you can run: npm install --save fs

@mattjstar's fix works for me too.

EDIT: No it didn't - it stopped the error messages from appearing, but then dotenv stopped working.

@KyleAMathews
Copy link
Contributor

@KyleAMathews KyleAMathews commented Nov 9, 2017

@richardwestenra is this error happening in the browser? If so, know that dotenv is only useful in node.js code so shouldn't be required in the browser.

@richardwestenra
Copy link
Collaborator

@richardwestenra richardwestenra commented Nov 9, 2017

@KyleAMathews Nope, this error is happening in the terminal, confusingly.

I've tried moving it to devDependencies, as well as setting

  config.merge({
    target: 'node'
  });

in the webpack config, but have had no success 😕

@KyleAMathews
Copy link
Contributor

@KyleAMathews KyleAMathews commented Nov 9, 2017

Webpack is only for browser code so unless you're requiring dotenv in browser code modifying the webpack config isn't going to help with node code. Confusing I know :-)

@richardwestenra
Copy link
Collaborator

@richardwestenra richardwestenra commented Nov 9, 2017

ah okay, that helps narrow things down a bit. Does that mean it's unlikely to be a Gatsby-related issue, then? I'm pretty much stumped for ideas.

I'm using dotenv to populate Contentful/Workable API keys used in plugins in gatsby-config.js, having initialised dotenv at the start of that file.

@KyleAMathews
Copy link
Contributor

@KyleAMathews KyleAMathews commented Nov 9, 2017

Perhaps try rm -r node_modules && yarn? :-)

But yeah, this is either an install problem or a dotenv problem. Node.js is somehow failing to require the built-in fs module.

@richardwestenra
Copy link
Collaborator

@richardwestenra richardwestenra commented Nov 11, 2017

Have tried that several times, and it doesn't look like dotenv has published any releases in a year. I'll keep looking. Thanks for the assistance! :)

@richardwestenra
Copy link
Collaborator

@richardwestenra richardwestenra commented Nov 13, 2017

Okay I figured it out! I'm unsure whether it's relevant to others' issues posted above, but I'll post details of my solution here in case it helps anyone.

It turns out, the issue showed up when I started including siteMetadata.title from gatsby-config in my index layout file. I'm not sure what the siteMetadata prop is supposed to be used for - I found a reference to page components having access to it with props.data.site.siteMetadata, but that props.data isn't showing up for me. Anyway it seemed like using that property for the site title would likely be best practice, so I included it in the layout template with import { siteMetadata } from '../../gatsby-config', then used it to populate <title> with React Helmet. However I guess this caused gatsby-config started to be included in Webpack browser code, which doesn't have access to fs, so it started throwing errors.

Is there a recommended way of including gatsby-config's siteMetadata prop in browser code? If not, what is its purpose?

@richardwestenra
Copy link
Collaborator

@richardwestenra richardwestenra commented Nov 13, 2017

ah okay, great, thanks! I didn't spot that, but I guess it'll be included in the documentation when the GraphQL docs page is done.

Thanks again for your help!

@parvezk
Copy link

@parvezk parvezk commented Nov 19, 2017

You can also follow this approach to fix the XMLHTTPRequest issue with module bundling:
webpack/webpack-dev-server#66

@kwelch
Copy link

@kwelch kwelch commented Mar 27, 2018

I am seeing the same fs error post-installing dotenv. Does anyone have version 1.9.223 working with dotenv?

@ohenrik
Copy link

@ohenrik ohenrik commented Apr 2, 2018

@haroldtreen where should i add this? If i add packageMains to resolve i get an error telling me that packageMains is unknown.

@jhlabs
Copy link

@jhlabs jhlabs commented May 10, 2018

I am facing exactly the same error as @richardwestenra when installing the gatsby-plugin-mailchimp and using dotenv. Is there a way to resolve it in that case?

@kakadiadarpan
Copy link
Collaborator

@kakadiadarpan kakadiadarpan commented Sep 28, 2018

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub issues, we have to clean some of the old issues as many of them have already been resolved with the latest updates.

Please make sure to update to the latest Gatsby version and check if that solves the issue. Let us know if that works for you by adding a comment 👍

@kakadiadarpan
Copy link
Collaborator

@kakadiadarpan kakadiadarpan commented Oct 10, 2018

This issue is being closed because there hasn't been any activity for at least 30 days. Feel free to open a new one if you still experience this problem 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet