Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

import tfjs-node and build via webpack,babel - error node-pre-gyp #1772

Closed
jakubfrydrych opened this issue Jul 25, 2019 · 7 comments
Closed

import tfjs-node and build via webpack,babel - error node-pre-gyp #1772

jakubfrydrych opened this issue Jul 25, 2019 · 7 comments
Assignees
@kangyizhang
Labels
stat:awaiting response type:bug Something isn't working type:build/install

Comments

@jakubfrydrych
Copy link

Hi, I'm trying to break tfjs-node in vain the next day. When compiling using a webpack, I get errors from the node-pre-gyp package. I tried all possible ways I found in the forums, but nothing helps. I tried downgrading tfjs-node to version 1.2.1, but also without effect. I also tried the node in 8. I import the package according to the example. Is there a bug in building with a webpack with a babel?

TensorFlow.js version

1.2.5

Node version

v10.16.0 (for test try v8.11.3)

OS

macOS Mojave 10.14.5 (18F132)
processor: 2,7 GHz Intel Core i5

BUILD ERRORS

> adsplayertf@1.0.0 try /Users/jf/Documents/Work/adsPlayer TF/tf
> webpack && node ./build/main.js

Hash: ee4f65476382096ae2f8
Version: webpack 3.12.0
Time: 2852ms
        Asset     Size  Chunks                    Chunk Names
      main.js  2.85 MB       0  [emitted]  [big]  main
normalizer.js  47.7 kB       1  [emitted]         normalizer
   creator.js  5.73 kB       2  [emitted]         creator
 [147] ./node_modules/node-pre-gyp/lib ^\.\/.*$ 1.03 kB {0} [built]
 [170] ./dataset/creator.js 2.5 kB {2} [built]
 [171] ./src/index.js 4.62 kB {0} [built]
 [247] ./node_modules/node-pre-gyp/lib/util 160 bytes {0} [built]
 [248] ./node_modules/node-pre-gyp/lib 160 bytes {0} [built]
 [302] ./node_modules/@tensorflow/tfjs-node/dist 160 bytes {0} [built]
 [306] ./src/class/Normalize.js 1.54 kB {0} [built]
 [312] ./data/normalizer_values.json 392 bytes {0} [built]
 [313] ./data/normalized_dataset.json 645 kB {0} [built]
 [314] ./dataset/normalizer.js 2.04 kB {1} [built]
 [328] ./data/raw_data.json 1.13 kB {1} [built]
    + 318 hidden modules

WARNING in ./node_modules/node-pre-gyp/lib/util/versioning.js
17:20-67 Critical dependency: the request of a dependency is an expression
 @ ./node_modules/node-pre-gyp/lib/util/versioning.js
 @ ./node_modules/node-pre-gyp/lib ^\.\/.*$
 @ ./node_modules/node-pre-gyp/lib/node-pre-gyp.js
 @ ./node_modules/@tensorflow/tfjs-node/dist/index.js
 @ ./src/index.js

WARNING in ./node_modules/node-pre-gyp/lib/pre-binding.js
20:22-48 Critical dependency: the request of a dependency is an expression
 @ ./node_modules/node-pre-gyp/lib/pre-binding.js
 @ ./node_modules/node-pre-gyp/lib/node-pre-gyp.js
 @ ./node_modules/@tensorflow/tfjs-node/dist/index.js
 @ ./src/index.js

WARNING in ./node_modules/@tensorflow/tfjs-node/dist/index.js
44:15-35 Critical dependency: the request of a dependency is an expression
 @ ./node_modules/@tensorflow/tfjs-node/dist/index.js
 @ ./src/index.js

WARNING in ./node_modules/node-fetch/lib/index.es.js
Module not found: Error: Can't resolve 'encoding' in '/Users/jakubfrydrych/Documents/Work/adsPlayer TF/tf/node_modules/node-fetch/lib'
 @ ./node_modules/node-fetch/lib/index.es.js 141:11-30
 @ ./node_modules/@tensorflow/tfjs-core/dist/tf-core.esm.js
 @ ./node_modules/@tensorflow/tfjs-node/dist/io/file_system.js
 @ ./node_modules/@tensorflow/tfjs-node/dist/index.js
 @ ./src/index.js

WARNING in ./node_modules/node-pre-gyp/lib/util/compile.js
Module not found: Error: Can't resolve 'node-gyp' in '/Users/jakubfrydrych/Documents/Work/adsPlayer TF/tf/node_modules/node-pre-gyp/lib/util'
 @ ./node_modules/node-pre-gyp/lib/util/compile.js 29:28-55
 @ ./node_modules/node-pre-gyp/lib ^\.\/.*$
 @ ./node_modules/node-pre-gyp/lib/node-pre-gyp.js
 @ ./node_modules/@tensorflow/tfjs-node/dist/index.js
 @ ./src/index.js

WARNING in ./node_modules/node-pre-gyp/lib/util/compile.js
Module not found: Error: Can't resolve 'npm' in '/Users/jakubfrydrych/Documents/Work/adsPlayer TF/tf/node_modules/node-pre-gyp/lib/util'
 @ ./node_modules/node-pre-gyp/lib/util/compile.js 39:27-49
 @ ./node_modules/node-pre-gyp/lib ^\.\/.*$
 @ ./node_modules/node-pre-gyp/lib/node-pre-gyp.js
 @ ./node_modules/@tensorflow/tfjs-node/dist/index.js
 @ ./src/index.js

WARNING in ./node_modules/node-pre-gyp/lib/install.js
Module not found: Error: Can't resolve 'request' in '/Users/jakubfrydrych/Documents/Work/adsPlayer TF/tf/node_modules/node-pre-gyp/lib'
 @ ./node_modules/node-pre-gyp/lib/install.js 28:18-36
 @ ./node_modules/node-pre-gyp/lib ^\.\/.*$
 @ ./node_modules/node-pre-gyp/lib/node-pre-gyp.js
 @ ./node_modules/@tensorflow/tfjs-node/dist/index.js
 @ ./src/index.js

WARNING in ./node_modules/node-pre-gyp/lib/util/nw-pre-gyp/index.html
Module parse failed: Unexpected token (1:0)
You may need an appropriate loader to handle this file type.
| <!doctype html>
| <html>
| <head>
 @ ./node_modules/node-pre-gyp/lib ^\.\/.*$
 @ ./node_modules/node-pre-gyp/lib/node-pre-gyp.js
 @ ./node_modules/@tensorflow/tfjs-node/dist/index.js
 @ ./src/index.js

ERROR in ./node_modules/node-pre-gyp/lib/info.js
Module not found: Error: Can't resolve 'aws-sdk' in '/Users/jf/Documents/Work/adsPlayer TF/tf/node_modules/node-pre-gyp/lib'
 @ ./node_modules/node-pre-gyp/lib/info.js 14:14-32
 @ ./node_modules/node-pre-gyp/lib ^\.\/.*$
 @ ./node_modules/node-pre-gyp/lib/node-pre-gyp.js
 @ ./node_modules/@tensorflow/tfjs-node/dist/index.js
 @ ./src/index.js

ERROR in ./node_modules/node-pre-gyp/lib/publish.js
Module not found: Error: Can't resolve 'aws-sdk' in '/Users/jf/Documents/Work/adsPlayer TF/tf/node_modules/node-pre-gyp/lib'
 @ ./node_modules/node-pre-gyp/lib/publish.js 18:14-32
 @ ./node_modules/node-pre-gyp/lib ^\.\/.*$
 @ ./node_modules/node-pre-gyp/lib/node-pre-gyp.js
 @ ./node_modules/@tensorflow/tfjs-node/dist/index.js
 @ ./src/index.js

ERROR in ./node_modules/node-pre-gyp/lib/unpublish.js
Module not found: Error: Can't resolve 'aws-sdk' in '/Users/jf/Documents/Work/adsPlayer TF/tf/node_modules/node-pre-gyp/lib'
 @ ./node_modules/node-pre-gyp/lib/unpublish.js 16:14-32
 @ ./node_modules/node-pre-gyp/lib ^\.\/.*$
 @ ./node_modules/node-pre-gyp/lib/node-pre-gyp.js
 @ ./node_modules/@tensorflow/tfjs-node/dist/index.js
 @ ./src/index.js

package.json

{
  "name": "adsplayertf",
  "version": "1.0.0",
  "description": "",
  "main": "webpack.config.js",
  "scripts": {
    "build": "webpack",
    "create-data": "rimraf data && node ./build/creator.js",
    "normalize-data": "node ./build/normalizer.js",
    "prepare-dataset": "rimraf data && webpack && node ./build/creator.js 100 && node ./build/normalizer.js",
    "try": "webpack && node ./build/main.js",
    "watch:build": "webpack --watch"
  },
  "author": "",
  "license": "MIT",
  "devDependencies": {
    "babel-core": "^6.26.0",
    "babel-loader": "^7.1.2",
    "babel-plugin-transform-runtime": "^6.23.0",
    "babel-preset-env": "^1.6.1",
    "regenerator-runtime": "^0.12.1",
    "webpack": "^3.8.1"
  },
  "dependencies": {
    "@tensorflow/tfjs-node": "^1.2.5",
    "rimraf": "^2.6.3"
  }
}

top of main.js document, import tfjs-node

import * as tf from '@tensorflow/tfjs-node'
@rthadur rthadur added type:bug Something isn't working type:build/install labels Jul 25, 2019
@rthadur rthadur self-assigned this Jul 25, 2019
@kangyizhang
Copy link
Contributor

hi @jakubfrydrych

It seems like webpack is trying to import some objects that are only used during development from node-pre-gyp. 'aws-sdk' in node-pre-gyp is a devDependencies and should not be invoked in your process. Can you try webpack-node-externals and ignore the modules?

@jakubfrydrych
Copy link
Author

Hi @kangyizhang
I have installed webpack-node-externals, but I don't know if you meant this configuration. Or somehow exclude that node-pre-gyp in the webpack settings?

webpack.config.js

const path = require('path');

var nodeExternals = require('webpack-node-externals');

module.exports = {
    entry: {
      //'regenerator-runtime/runtime',
      creator: './dataset/creator.js',
      main: './src/index.js',
      normalizer: './dataset/normalizer.js'
    },
    target: 'node',
    //devtool: 'source-map',
    output: {
        path: path.resolve(__dirname, 'build'),
        filename: '[name].js',
        publicPath: 'build/'
    },
    module: {
        rules: [
            {
                use: 'babel-loader',
                exclude: /(node_modules)/,
                test: /\.js$/
            }
        ]
    },
    target: 'node', 
    externals: [nodeExternals()]
}

build error

> adsplayertf@1.0.0 try /Users/jakubfrydrych/Documents/Work/adsPlayer TF/tf
> webpack && node ./build/main.js

Hash: a75ba2c09973765a46b4
Version: webpack 3.12.0
Time: 1201ms
        Asset     Size  Chunks                    Chunk Names
      main.js   655 kB       0  [emitted]  [big]  main
normalizer.js  6.19 kB       1  [emitted]         normalizer
   creator.js   5.2 kB       2  [emitted]         creator
   [3] ./dataset/creator.js 2.5 kB {2} [built]
   [4] ./src/index.js 4.62 kB {0} [built]
   [8] ./src/class/Normalize.js 1.54 kB {0} [built]
  [11] ./data/normalizer_values.json 392 bytes {0} [built]
  [12] ./data/normalized_dataset.json 645 kB {0} [built]
  [13] ./dataset/normalizer.js 2.04 kB {1} [built]
  [15] ./data/raw_data.json 1.13 kB {1} [built]
    + 9 hidden modules
node-pre-gyp info This Node instance does not support builds for N-API version 4
node-pre-gyp info This Node instance does not support builds for N-API version 4
module.js:549
    throw err;
    ^

Error: Cannot find module '/Users/jf/Documents/Work/adsPlayer TF/tf/node_modules/@tensorflow/tfjs-node/lib/napi-v3/tfjs_binding.node'
    at Function.Module._resolveFilename (module.js:547:15)
    at Function.Module._load (module.js:474:25)
    at Module.require (module.js:596:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/Users/jakubfrydrych/Documents/Work/adsPlayer TF/tf/node_modules/@tensorflow/tfjs-node/dist/index.js:44:16)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)
npm ERR!

@rthadur rthadur assigned kangyizhang and unassigned rthadur Jul 25, 2019
@kangyizhang
Copy link
Contributor

@jakubfrydrych

The tfjs-node module requires tensorflow C library and node native addon. I don't think webpack would include these artifacts as static assets.

@jakubfrydrych
Copy link
Author

@kangyizhang

Thank you very much for your advice. I reinstalled the native TF library, updated packages, cleared the cache, and mainly removed some of the code that the webpack needed (es6 features) to build, and now I run all the logic through node.js and everything works.

@kangyizhang
Copy link
Contributor

Glad it works.

@rthadur rthadur mentioned this issue Sep 8, 2020
Closed
@rfourdinier
Copy link

rfourdinier commented Oct 9, 2020 •
edited
Loading

@kangyizhang @jakubfrydrych hello , i am struggling with the exact same error

I do not understand what do you mean when you say that you have "installed tensorflow library", should i install it from pip or put the Tensorflow C library folder somewhere ? how does tfjs-node know where my tensorflow C library is located ?

thank you very much

@playground
Copy link

I'm getting the same errors. webpack-node-externals didn't help.

"devDependencies": {
"@types/node": "^15.0.2",
"@typescript-eslint/parser": "^4.22.1",
"awesome-typescript-loader": "^5.2.1",
"dotenv": "^9.0.0",
"jsonfile": "^6.1.0",
"ts-loader": "^9.1.2",
"ts-node": "^9.1.1",
"typescript": "^4.2.4",
"webpack": "^5.36.2",
"webpack-cli": "^4.6.0",
"webpack-node-externals": "^3.0.0"
},
"dependencies": {
"@tensorflow-models/mobilenet": "^2.1.0",
"@tensorflow/tfjs": "^3.6.0",
"@tensorflow/tfjs-node": "^3.6.1",
"rxjs": "^7.0.0"
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kangyizhang kangyizhang

Labels
stat:awaiting response type:bug Something isn't working type:build/install
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@playground @kangyizhang @jakubfrydrych @rthadur @rfourdinier