New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

webpack & google-cloud #1821

Closed
brandonmp opened this Issue Nov 26, 2016 · 12 comments

Comments

Projects
None yet
10 participants
@brandonmp

brandonmp commented Nov 26, 2016

Environment details

  • OS: ubuntu 16.10
  • Node.js version: 4.3.2
  • npm version: 2.14.12
  • google-cloud-node version: 0.44

Steps to reproduce

I'm using a webpack setup to build a nodejs project, the dependencies of which include google-cloud

Webpack sort of loses its mind w/ google-cloud, though, so hoping someone has some insight on how to fix.

The first error, ironically, appeared to be a deeply nested aws-sdk dependency, b/c I guess aws-sdk doesn't play nicely w/ webpack. I was able to fix that by including a pre-packaged AWS in my webpack-config.js, which is here:

// webpack.config.js
var fs = require('fs');
var path = require("path");
var webpack = require('webpack')
var prePackagedAwsPath = path.join('aws-sdk', 'dist', 'aws-sdk');

var nodeModules = {};
fs.readdirSync('node_modules')
    .filter(function (x) {
        return ['.bin'].indexOf(x) === -1;
    })
    .forEach(function (mod) {
        nodeModules[mod] = 'commonjs ' + mod;
    });

module.exports = {
    entry: path.join(__dirname, '/src/index.js'),
    output: {
        path: path.join(__dirname, "/dist/"),
        filename: "[name].js"
    },
    resolve: {
        alias: {
            'aws-sdk': prePackagedAwsPath
        }
    },
    target: "node",
    externals: nodeModules,
    module: {
        loaders: [
            {
                test: /\.node/,
                loader: 'node'
            },
            {
                test: require.resolve(prePackagedAwsPath),
                loader: 'imports?window=>null'
            },
            {
                test: /\.js$/,
                exclude: /node_modules/,
                loader: 'babel'
            },
            {
                test: /\.json$/,
                loader: 'json'
            }
        ]
    }
};

Now, however, the problem lies with the JSONStream dependency. The file apparently includes a # character at the top of its index.js, and that's breaking the build process.

Here's the error:

ERROR in ./~/@wayhome/wayhome-utils/~/google-cloud/~/JSONStream/index.js
Module parse failed: /home/bmp/code/scraping/node_modules/@wayhome/wayhome-utils/node_modules/google-cloud/node_modules/JSONStream/index.js Unexpected character '#' (1:0
)
You may need an appropriate loader to handle this file type.
SyntaxError: Unexpected character '#' (1:0)
    at Parser.pp$4.raise (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:2221:15)
    at Parser.pp$7.getTokenFromCode (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:2756:10)
    at Parser.pp$7.readToken (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:2477:17)
    at Parser.pp$7.nextToken (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:2468:15)
    at Parser.parse (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:515:10)
    at Object.parse (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:3098:39)
    at Parser.parse (/home/bmp/code/scraping/node_modules/webpack/lib/Parser.js:902:15)
    at DependenciesBlock.<anonymous> (/home/bmp/code/scraping/node_modules/webpack/lib/NormalModule.js:104:16)
    at DependenciesBlock.onModuleBuild (/home/bmp/code/scraping/node_modules/webpack/node_modules/webpack-core/lib/NormalModuleMixin.js:310:10)
    at nextLoader (/home/bmp/code/scraping/node_modules/webpack/node_modules/webpack-core/lib/NormalModuleMixin.js:275:25)
    at /home/bmp/code/scraping/node_modules/webpack/node_modules/webpack-core/lib/NormalModuleMixin.js:259:5
    at Storage.finished (/home/bmp/code/scraping/node_modules/webpack/node_modules/enhanced-resolve/lib/CachedInputFileSystem.js:38:16)
    at /home/bmp/code/scraping/node_modules/graceful-fs/graceful-fs.js:78:16
    at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:380:3)
 @ ./~/@wayhome/wayhome-utils/~/google-cloud/~/@google-cloud/prediction/src/model.js 25:17-38

I'd normally call this a JSONStream issue and file it there, but google-cloud is throwing warnings everywhere (below) as well, so I assume there's some overarching issue w/ webpack.

WARNING in ./~/@wayhome/wayhome-utils/~/google-cloud/~/hash-stream-validation/index.js
Module not found: Error: Cannot resolve module 'fast-crc32c' in /home/bmp/code/scraping/node_modules/@wayhome/wayhome-utils/node_modules/google-cloud/node_modules/hash-s
tream-validation
 @ ./~/@wayhome/wayhome-utils/~/google-cloud/~/hash-stream-validation/index.js 5:8-30

WARNING in ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/src/node/src/grpc_extension.js
Critical dependencies:
38:14-35 the request of a dependency is an expression
 @ ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/src/node/src/grpc_extension.js 38:14-35

WARNING in ./~/@wayhome/wayhome-utils/~/google-cloud/~/bytebuffer/dist/bytebuffer-node.js
Module not found: Error: Cannot resolve module 'memcpy' in /home/bmp/code/scraping/node_modules/@wayhome/wayhome-utils/node_modules/google-cloud/node_modules/bytebuffer/
dist
 @ ./~/@wayhome/wayhome-utils/~/google-cloud/~/bytebuffer/dist/bytebuffer-node.js 29:38-55

WARNING in ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/~/node-pre-gyp/lib/pre-binding.js
Critical dependencies:
19:22-48 the request of a dependency is an expression
 @ ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/~/node-pre-gyp/lib/pre-binding.js 19:22-48
WARNING in ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/~/node-pre-gyp/lib/util/versioning.js
Critical dependencies:
15:20-67 the request of a dependency is an expression
 @ ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/~/node-pre-gyp/lib/util/versioning.js 15:20-67

WARNING in ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/~/node-pre-gyp/lib/util/nw-pre-gyp/index.html
Module parse failed: /home/bmp/code/scraping/node_modules/@wayhome/wayhome-utils/node_modules/google-cloud/node_modules/grpc/node_modules/node-pre-gyp/lib/util/nw-pre-gy
p/index.html Unexpected token (1:0)
You may need an appropriate loader to handle this file type.
SyntaxError: Unexpected token (1:0)
    at Parser.pp$4.raise (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:2221:15)
    at Parser.pp.unexpected (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:603:10)
    at Parser.pp$3.parseExprAtom (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:1822:12)
    at Parser.pp$3.parseExprSubscripts (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:1715:21)
    at Parser.pp$3.parseMaybeUnary (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:1692:19)
    at Parser.pp$3.parseExprOps (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:1637:21)
    at Parser.pp$3.parseMaybeConditional (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:1620:21)
    at Parser.pp$3.parseMaybeAssign (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:1597:21)
    at Parser.pp$3.parseExpression (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:1573:21)
    at Parser.pp$1.parseStatement (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:727:47)
    at Parser.pp$1.parseTopLevel (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:638:25)
    at Parser.parse (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:516:17)
    at Object.parse (/home/bmp/code/scraping/node_modules/webpack/node_modules/acorn/dist/acorn.js:3098:39)
    at Parser.parse (/home/bmp/code/scraping/node_modules/webpack/lib/Parser.js:902:15)
    at DependenciesBlock.<anonymous> (/home/bmp/code/scraping/node_modules/webpack/lib/NormalModule.js:104:16)
    at DependenciesBlock.onModuleBuild (/home/bmp/code/scraping/node_modules/webpack/node_modules/webpack-core/lib/NormalModuleMixin.js:310:10)
    at nextLoader (/home/bmp/code/scraping/node_modules/webpack/node_modules/webpack-core/lib/NormalModuleMixin.js:275:25)
    at /home/bmp/code/scraping/node_modules/webpack/node_modules/webpack-core/lib/NormalModuleMixin.js:259:5
    at Storage.finished (/home/bmp/code/scraping/node_modules/webpack/node_modules/enhanced-resolve/lib/CachedInputFileSystem.js:38:16)
    at /home/bmp/code/scraping/node_modules/graceful-fs/graceful-fs.js:78:16
    at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:380:3)
 @ ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/~/node-pre-gyp/lib ^\.\/.*$

WARNING in ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/~/node-pre-gyp/lib/util/compile.js
Module not found: Error: Cannot resolve module 'node-gyp' in /home/bmp/code/scraping/node_modules/@wayhome/wayhome-utils/node_modules/google-cloud/node_modules/grpc/node
_modules/node-pre-gyp/lib/util
 @ ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/~/node-pre-gyp/lib/util/compile.js 29:28-55

WARNING in ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/~/node-pre-gyp/lib/util/compile.js
Module not found: Error: Cannot resolve module 'npm' in /home/bmp/code/scraping/node_modules/@wayhome/wayhome-utils/node_modules/google-cloud/node_modules/grpc/node_modu
les/node-pre-gyp/lib/util
 @ ./~/@wayhome/wayhome-utils/~/google-cloud/~/grpc/~/node-pre-gyp/lib/util/compile.js 39:27-49
@stephenplusplus

This comment has been minimized.

Member

stephenplusplus commented Nov 28, 2016

We haven't been testing against webpack, so I'm quite confident it would be pure luck if all of our dependencies would be compatible. I'm going to play around with it as time allows, but feel free to dig into it and google around for issues if you want to help out. After a quick search, I found this issue which might get you past the JSONStream error: webpack/webpack#2168

If you do dig into this, please keep me updated!

@brandonmp

This comment has been minimized.

brandonmp commented Dec 1, 2016

thanks @stephenplusplus ! that helped get around the !# error, though i've hit a few more I'll detail here for posterity. if anyone has ideas on how to address the unsolved problem, would be grateful :) note that this sort of stuff is a little outta my league, so some of these fixes may be suboptimal

!# error

error: wepback won't parse .js files that start with a !# arg
solution: install a special loader to handle

$ npm install shebang-loader

// in webpack.config.js, add:
module: {
  loaders: [
...
     {
      // several gcloud dependencies have !#, so we'll apply loader broadly
      test: /node_modules\/.*\.js$/,
      loaders: ['shebang', 'babel']
     }
...
  ]
}

source: webpack/webpack#2168

node-forge

error: node-forge throws Invalid left-hand side in assignment
solution: this one feels a bit janky. we'll minimize and sideload it into our webpack.config.js:

npm i node-forge
cd node_modules/node-forge
npm i
node node_modules\requirejs\bin\r.js -o minify.js optimize=none out=js/forge.bundle.js
// copy forge.bundle.js elsewhere if you like. mine is in src/webpack/

// webpack.config.js:
var forge = require('node-forge')

//then modify your babel-loader module loader to add forge.bundle.js, and add the file to `noParse`:
module: {
    loaders: [
...
          {
                test: /\.js$/,
                exclude: /(node_modules|forge\.bundle\.js)/,
                loader: 'babel-loader'
            }
...
    ],
     noParse: [
            path.resolve(__dirname, './whatever/path/you/chose/for/forge.bundle.js')
        ]

source: digitalbazaar/forge#198

node-pre-gyp

error: haven't researched this one yet. here's error text:

ERROR in ./~/@google-cloud/storage/~/@google-cloud/common/~/grpc/~/node-pre-gyp/~/tar/tar.js
Module build failed: SyntaxError: Invalid number (108:10)

  106 | // values for the mode field
  107 | var modes =
> 108 |   { suid: 04000 // set uid on extraction
      |           ^
  109 |   , sgid: 02000 // set gid on extraction
  110 |   , svtx: 01000 // set restricted deletion flag on dirs on extraction
  111 |   , uread:  0400

 @ ./~/@google-cloud/storage/~/@google-cloud/common/~/grpc/~/node-pre-gyp/lib/install.js 56:24-38
@stephenplusplus

This comment has been minimized.

Member

stephenplusplus commented Dec 6, 2016

I'm going to close this issue, since we don't have any plans to support being run through webpack at this time. Please subscribe to #531 and if we end up carving out some time to spend towards this goal, you can follow along. Sorry I can't be of more help at this time.

@brandonmp

This comment has been minimized.

brandonmp commented Dec 7, 2016

thanks @stephenplusplus , though my interests were more on the-server-side--specifically for aws-lambda/google-cloud functions. webpack is pretty clutch in that workflow, so I'm hopeful the plan to support it changes in the near future.

@mikelambert

This comment has been minimized.

mikelambert commented Dec 13, 2016

@brandonmp , I believe the node-pre-gyp code would require a change:
{ suid: parseInt("4000", 8)
and so on.
(Or if node-pre-gyp is relying on ES6, which I'm not sure is safe...one could write 0o4000)

Seems webpack is enforcing the fact that 0400 is a non-standard way to write an octal number, which I'd classify as a bug, except that maybe node-pre-gyp is assuming it only ever runs in a node environment.

@jonwoodring

This comment has been minimized.

jonwoodring commented Feb 25, 2017

@olegccc

This comment has been minimized.

olegccc commented Apr 30, 2017

based on the comments above I assume the only safe way is to exclude this module from bundling using externals webpack option or webpack-node-externals module.

@slootzky

This comment has been minimized.

slootzky commented Jun 8, 2017

any news with this issue? trying to use Google's pubsub from Azure functions & Firebase functions so we're using webpack which fails (currently at node-pre-gyp / pre-binding.js in this part :
if (!existsSync(package_json_path)) { throw new Error("package.json does not exist at " + package_json_path); }

@gvillenave

This comment has been minimized.

gvillenave commented Jan 19, 2018

I'm seeing the same error when building a React website with Firebase using Gatsby. Webpack is THE bundling standard nowadays, it seems quite odd that the Google Cloud Platform SDK would not be able to work with it...
Why was that issue closed in the first place?

@ksnyde

This comment has been minimized.

ksnyde commented Feb 14, 2018

I've tried bundling with webpack, rollup, and browserify (all targeting node) and they all have problems with the @google-cloud/* libraries. Everything else works flawlessly.

@dundas

This comment has been minimized.

dundas commented Apr 10, 2018

I dont imagine anyone has come up with an answer for this. this bug is making me crazy right now. any work around?

@ohabash

This comment has been minimized.

ohabash commented Dec 2, 2018

@dundas did you find a solution?

How do use firebase-admin in angular unversal server?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment