Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

yarn remove <package> results in Error: Cannot find module './build/Release/re2' #69

Closed
niftylettuce opened this issue Jun 1, 2020 · 47 comments
Assignees
Labels
bug A reported bug. help wanted

Comments

@niftylettuce
Copy link

After running yarn remove <package>, re2 errors out on app reboot with this error:

Error: Cannot find module './build/Release/re2'

Using latest Node LTS stable and latest re2. This is probably a node-gyp + yarn issue though.

yarn@1.22.4
node@12.16.1
re2@1.14.0

Ref: google/re2#261

@uhop uhop self-assigned this Jun 2, 2020
@uhop uhop added the bug A reported bug. label Jun 2, 2020
@uhop
Copy link
Owner

uhop commented Jun 2, 2020

errors out on app reboot with this error

What is "app reboot"? How can I repro it?

@uhop
Copy link
Owner

uhop commented Jun 4, 2020

This is my session:

Creating the test project
$ mkdir test
$ cd test
$ npm init -y
Wrote to /media/raid/Work/test/package.json:

{
  "name": "test",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}

Installing RE2
$ yarn add re2
yarn add v1.12.3
info No lockfile found.
[1/4] Resolving packages...
warning re2 > node-gyp > request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
warning Your current version of Yarn is out of date. The latest version is "1.22.4", while you're on "1.12.3".
info To upgrade, run the following command:
$ sudo apt-get update && sudo apt-get install yarn
success Saved 98 new dependencies.
info Direct dependencies
└─ re2@1.14.0
info All dependencies
├─ abbrev@1.1.1
├─ ajv@6.12.2
├─ ansi-regex@2.1.1
├─ aproba@1.2.0
├─ are-we-there-yet@1.1.5
├─ asn1@0.2.4
├─ asynckit@0.4.0
├─ aws-sign2@0.7.0
├─ aws4@1.10.0
├─ balanced-match@1.0.0
├─ bcrypt-pbkdf@1.0.2
├─ brace-expansion@1.1.11
├─ caseless@0.12.0
├─ chownr@1.1.4
├─ code-point-at@1.1.0
├─ combined-stream@1.0.8
├─ concat-map@0.0.1
├─ console-control-strings@1.1.0
├─ core-util-is@1.0.2
├─ dashdash@1.14.1
├─ delayed-stream@1.0.0
├─ delegates@1.0.0
├─ ecc-jsbn@0.1.2
├─ env-paths@2.2.0
├─ extend@3.0.2
├─ extsprintf@1.3.0
├─ fast-deep-equal@3.1.1
├─ fast-json-stable-stringify@2.1.0
├─ forever-agent@0.6.1
├─ form-data@2.3.3
├─ fs-minipass@1.2.7
├─ fs.realpath@1.0.0
├─ gauge@2.7.4
├─ getpass@0.1.7
├─ glob@7.1.6
├─ graceful-fs@4.2.4
├─ har-schema@2.0.0
├─ har-validator@5.1.3
├─ has-unicode@2.0.1
├─ http-signature@1.2.0
├─ inflight@1.0.6
├─ inherits@2.0.4
├─ is-fullwidth-code-point@1.0.0
├─ is-typedarray@1.0.0
├─ isarray@1.0.0
├─ isexe@2.0.0
├─ isstream@0.1.2
├─ json-schema-traverse@0.4.1
├─ json-schema@0.2.3
├─ json-stringify-safe@5.0.1
├─ jsprim@1.4.1
├─ mime-db@1.44.0
├─ mime-types@2.1.27
├─ minimatch@3.0.4
├─ minimist@1.2.5
├─ minipass@2.9.0
├─ minizlib@1.3.3
├─ mkdirp@0.5.5
├─ nan@2.14.1
├─ node-gyp@6.1.0
├─ nopt@4.0.3
├─ npmlog@4.1.2
├─ number-is-nan@1.0.1
├─ oauth-sign@0.9.0
├─ object-assign@4.1.1
├─ os-homedir@1.0.2
├─ os-tmpdir@1.0.2
├─ osenv@0.1.5
├─ path-is-absolute@1.0.1
├─ performance-now@2.1.0
├─ process-nextick-args@2.0.1
├─ psl@1.8.0
├─ punycode@2.1.1
├─ qs@6.5.2
├─ re2@1.14.0
├─ readable-stream@2.3.7
├─ request@2.88.2
├─ rimraf@2.7.1
├─ safe-buffer@5.2.1
├─ safer-buffer@2.1.2
├─ semver@5.7.1
├─ set-blocking@2.0.0
├─ signal-exit@3.0.3
├─ sshpk@1.16.1
├─ string_decoder@1.1.1
├─ string-width@1.0.2
├─ strip-ansi@3.0.1
├─ tar@4.4.13
├─ tough-cookie@2.5.0
├─ tunnel-agent@0.6.0
├─ tweetnacl@0.14.5
├─ uri-js@4.2.2
├─ util-deprecate@1.0.2
├─ uuid@3.4.0
├─ verror@1.10.0
├─ which@1.3.1
├─ wide-align@1.1.3
└─ yallist@3.1.1
Done in 40.26s.
Checking that it works
$ node
Welcome to Node.js v14.3.0.
Type ".help" for more information.
> const RE2 = require('re2');
undefined
> RE2('a', 'i').test('bac')
true
> RE2('a', 'i').exec('bac')
[ 'a', index: 1, input: 'bac', groups: undefined ]
> RE2('a', 'i').exec('BAC')
[ 'A', index: 1, input: 'BAC', groups: undefined ]
> 
Removing RE2
$ yarn remove re2
yarn remove v1.12.3
[1/2] Removing module re2...
[2/2] Regenerating lockfile and installing missing dependencies...
success Uninstalled packages.
Done in 34.27s.
Checking again that it doesn't work
$ node
Welcome to Node.js v14.3.0.
Type ".help" for more information.
> const RE2 = require('re2');
Uncaught Error: Cannot find module 're2'
Require stack:
- <repl>
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:1029:15)
    at Function.Module._load (internal/modules/cjs/loader.js:898:27)
    at Module.require (internal/modules/cjs/loader.js:1089:19)
    at require (internal/modules/cjs/helpers.js:73:18) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [ '<repl>' ]
}
> 

As far as I can tell everything works as expected. Did I miss anything?

@niftylettuce
Copy link
Author

@uhop thanks for looking at this - I noticed you aren't doing the correct reproduction steps though - my instructions should have clarified the issue is reproducible when you remove a different package (other than re2).

@niftylettuce
Copy link
Author

@uhop see previous comment - also if you want to test on the exact repository I'm having issues with, you can do so by cloning https://github.com/forwardemail/forwardemail.net

@niftylettuce
Copy link
Author

@uhop one final comment - there is a CVE (I discovered) that this package saved me with - so thank you for publishing this!! https://snyk.io/vuln/SNYK-JS-URLREGEX-569472

@uhop
Copy link
Owner

uhop commented Jun 4, 2020

I've noticed that my yarn is not up-to-date, and I used Node 14, not 12. So I upgraded yarn and downgraded Node. The results were exactly the same.

@uhop
Copy link
Owner

uhop commented Jun 4, 2020

OK, I added one more package:

Installing two packages
$ yarn add re2 stream-json
yarn add v1.22.4
[1/4] Resolving packages...
warning re2 > node-gyp > request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
success Saved 100 new dependencies.
info Direct dependencies
├─ re2@1.14.0
└─ stream-json@1.5.0
info All dependencies
├─ abbrev@1.1.1
├─ ajv@6.12.2
├─ ansi-regex@2.1.1
├─ aproba@1.2.0
├─ are-we-there-yet@1.1.5
├─ asn1@0.2.4
├─ asynckit@0.4.0
├─ aws-sign2@0.7.0
├─ aws4@1.10.0
├─ balanced-match@1.0.0
├─ bcrypt-pbkdf@1.0.2
├─ brace-expansion@1.1.11
├─ caseless@0.12.0
├─ chownr@1.1.4
├─ code-point-at@1.1.0
├─ combined-stream@1.0.8
├─ concat-map@0.0.1
├─ console-control-strings@1.1.0
├─ core-util-is@1.0.2
├─ dashdash@1.14.1
├─ delayed-stream@1.0.0
├─ delegates@1.0.0
├─ ecc-jsbn@0.1.2
├─ env-paths@2.2.0
├─ extend@3.0.2
├─ extsprintf@1.3.0
├─ fast-deep-equal@3.1.1
├─ fast-json-stable-stringify@2.1.0
├─ forever-agent@0.6.1
├─ form-data@2.3.3
├─ fs-minipass@1.2.7
├─ fs.realpath@1.0.0
├─ gauge@2.7.4
├─ getpass@0.1.7
├─ glob@7.1.6
├─ graceful-fs@4.2.4
├─ har-schema@2.0.0
├─ har-validator@5.1.3
├─ has-unicode@2.0.1
├─ http-signature@1.2.0
├─ inflight@1.0.6
├─ inherits@2.0.4
├─ is-fullwidth-code-point@1.0.0
├─ is-typedarray@1.0.0
├─ isarray@1.0.0
├─ isexe@2.0.0
├─ isstream@0.1.2
├─ json-schema-traverse@0.4.1
├─ json-schema@0.2.3
├─ json-stringify-safe@5.0.1
├─ jsprim@1.4.1
├─ mime-db@1.44.0
├─ mime-types@2.1.27
├─ minimatch@3.0.4
├─ minimist@1.2.5
├─ minipass@2.9.0
├─ minizlib@1.3.3
├─ mkdirp@0.5.5
├─ nan@2.14.1
├─ node-gyp@6.1.0
├─ nopt@4.0.3
├─ npmlog@4.1.2
├─ number-is-nan@1.0.1
├─ oauth-sign@0.9.0
├─ object-assign@4.1.1
├─ os-homedir@1.0.2
├─ os-tmpdir@1.0.2
├─ osenv@0.1.5
├─ path-is-absolute@1.0.1
├─ performance-now@2.1.0
├─ process-nextick-args@2.0.1
├─ psl@1.8.0
├─ punycode@2.1.1
├─ qs@6.5.2
├─ re2@1.14.0
├─ readable-stream@2.3.7
├─ request@2.88.2
├─ rimraf@2.7.1
├─ safe-buffer@5.2.1
├─ safer-buffer@2.1.2
├─ semver@5.7.1
├─ set-blocking@2.0.0
├─ signal-exit@3.0.3
├─ sshpk@1.16.1
├─ stream-chain@2.2.1
├─ stream-json@1.5.0
├─ string_decoder@1.1.1
├─ string-width@1.0.2
├─ strip-ansi@3.0.1
├─ tar@4.4.13
├─ tough-cookie@2.5.0
├─ tunnel-agent@0.6.0
├─ tweetnacl@0.14.5
├─ uri-js@4.2.2
├─ util-deprecate@1.0.2
├─ uuid@3.4.0
├─ verror@1.10.0
├─ which@1.3.1
├─ wide-align@1.1.3
└─ yallist@3.1.1
Done in 37.49s.
Testing that RE2 works
$ node
Welcome to Node.js v12.16.3.
Type ".help" for more information.
> const RE2 = require('re2');
undefined
> RE2('a', 'i').exec('BAC')
[ 'A', index: 1, input: 'BAC', groups: undefined ]
> 
Removing the other package
$ yarn remove stream-json
yarn remove v1.22.4
[1/2] Removing module stream-json...
[2/2] Regenerating lockfile and installing missing dependencies...
success Uninstalled packages.
Done in 5.22s.
Testing that RE2 still works
$ node
Welcome to Node.js v12.16.3.
Type ".help" for more information.
> const RE2 = require('re2');
undefined
> RE2('a', 'i').exec('BAC')
[ 'A', index: 1, input: 'BAC', groups: undefined ]
> 

Still WFM. Must be some bizarre configuration or an obscure yarn bug.

@uhop
Copy link
Owner

uhop commented Jun 4, 2020

I tried to use the original project to debug. It takes almost 30 minutes to build on my computer. After that, I tried to start it with npm start and got:

npm start output
$ npm start

> forwardemail-net@0.1.41 start /media/raid/Work/forwardemail.net
> nps

Usage: nps [options] <script>...

Commands:
  nps init        automatically migrate from npm scripts to nps
  nps completion  generate completion script

Options:
  --config, -c      Config file to use (defaults to nearest package-scripts.yml
                    or package-scripts.js)
               [default: "/media/raid/Work/forwardemail.net/package-scripts.js"]
  --silent, -s      Silent nps output                 [boolean] [default: false]
  --log-level, -l   The log level to use
                   [choices: "error", "warn", "info", "debug"] [default: "info"]
  --require, -r     Module to preload
  --scripts         Log command text for script        [boolean] [default: true]
  --help-style, -y  Choose the level of detail displayed by the help command
                           [choices: "all", "scripts", "basic"] [default: "all"]
  -v, --version     Show version number                                [boolean]

Examples:
  nps test build                     Runs the `test` script then the `build`
                                     script
  nps "test --cover" "build --prod"  Runs the `test` script and forwards the
                                     "--cover" flag then the `build` script and
                                     forwards the "--prod" flag
Available scripts (camel or kebab case accepted)

all - nps build && nps apps-and-watch
appsAndWatch - node node_modules/concurrently/src/main.js --kill-others-on-fail --prefix-colors "bgBlue.bold,bgMagenta.bold" --prefix "[{name}]" --names "apps,watch" 'nps apps' 'nps watch'
apps - node node_modules/concurrently/src/main.js --kill-others-on-fail --prefix-colors "bgBlue.bold,bgMagenta.bold,bgGreen.bold" --prefix "[{name}]" --names "bull,api,web" 'nps bull' 'nps api' 'nps web'
bull - nodemon bull.js
api - nodemon api.js
web - nodemon web.js
watch - gulp watch
clean - gulp clean
build - gulp build
publishAssets - gulp publish
lint - gulp xo && gulp remark && gulp pug
pretest - node node_modules/concurrently/src/main.js --kill-others-on-fail --prefix-colors "bgBlue.bold,bgMagenta.bold,bgGreen.bold,bgBlack.bold" --prefix "[{name}]" --names "lint,build,pretest-mongo,pretest-redis" 'nps lint' 'nps build' 'nps pretest-mongo' 'nps pretest-redis'
pretestMongo - mongo forwardemail_test --eval 'db.dropDatabase()'
pretestRedis - redis-cli EVAL "return redis.call('del', 'defaultKey', unpack(redis.call('keys', ARGV[1])))" 0 limit_test:*
test - ava
testCoverage - nps pretest && nyc ava
testUpdateSnapshots - nps pretest && ava --update-snapshots
coverage - nyc report --reporter=text-lcov > coverage.lcov && codecov

npm test didn't run. Something about Redis, Mongo, and other infrastructure I do not posses:

npm test output
$ npm test

> forwardemail-net@0.1.41 pretest /media/raid/Work/forwardemail.net
> nps pretest

nps is executing `pretest` : node node_modules/concurrently/src/main.js --kill-others-on-fail --prefix-colors "bgBlue.bold,bgMagenta.bold,bgGreen.bold,bgBlack.bold" --prefix "[{name}]" --names "lint,build,pretest-mongo,pretest-redis" 'nps lint' 'nps build' 'nps pretest-mongo' 'nps pretest-redis'
[build] nps is executing `build` : gulp build
[lint] nps is executing `lint` : gulp xo && gulp remark && gulp pug
[pretest-redis] nps is executing `pretest-redis` : redis-cli EVAL "return redis.call('del', 'defaultKey', unpack(redis.call('keys', ARGV[1])))" 0 limit_test:*
[pretest-mongo] nps is executing `pretest-mongo` : mongo forwardemail_test --eval 'db.dropDatabase()'
[pretest-mongo] /bin/sh: 1: mongo: not found
[pretest-mongo] The script called "pretest-mongo" which runs "mongo forwardemail_test --eval 'db.dropDatabase()'" failed with exit code 127 https://github.com/sezna/nps/blob/master/other/ERRORS_AND_WARNINGS.md#failed-with-exit-code
[pretest-redis] /bin/sh: 1: redis-cli: not found
[pretest-redis] The script called "pretest-redis" which runs "redis-cli EVAL "return redis.call('del', 'defaultKey', unpack(redis.call('keys', ARGV[1])))" 0 limit_test:*" failed with exit code 127 https://github.com/sezna/nps/blob/master/other/ERRORS_AND_WARNINGS.md#failed-with-exit-code
[pretest-mongo] nps pretest-mongo exited with code 127
--> Sending SIGTERM to other processes..
[pretest-redis] nps pretest-redis exited with code 127
[lint] nps lint exited with code null
[build] nps build exited with code null
The script called "pretest" which runs "node node_modules/concurrently/src/main.js --kill-others-on-fail --prefix-colors "bgBlue.bold,bgMagenta.bold,bgGreen.bold,bgBlack.bold" --prefix "[{name}]" --names "lint,build,pretest-mongo,pretest-redis" 'nps lint' 'nps build' 'nps pretest-mongo' 'nps pretest-redis'" failed with exit code 1 https://github.com/sezna/nps/blob/master/other/ERRORS_AND_WARNINGS.md#failed-with-exit-code
npm ERR! Test failed.  See above for more details.

Anyway, I added an unrelated package (stream-json) then removed it. Before and after each action I ran npm test and imported re2 from Node command line. Both times it worked:

  • npm test ran identically to above and never complained about re2.
  • From the command line I was able to import re2 successfully.

I never got the error you got.

As much as it pains to tell it — it WFM.

@niftylettuce
Copy link
Author

Okay I will close for the time being. Will try to wipe Node / reinstall all. Not sure what's up... Thank you so much for your time.

@niftylettuce
Copy link
Author

This is most definitely a bug. On my production server after a pm2 deploy this happened:

3|api    | Error: Cannot find module './build/Release/re2'
3|api    | Require stack:
3|api    | - /var/www/production/source/node_modules/re2/re2.js
3|api    | - /var/www/production/source/node_modules/forward-email/index.js
3|api    | - /var/www/production/source/app/models/alias.js
3|api    | - /var/www/production/source/app/models/index.js
3|api    | - /var/www/production/source/helpers/passport.js
3|api    | - /var/www/production/source/config/api.js
3|api    | - /var/www/production/source/api.js
3|api    |     at Function.Module._resolveFilename (internal/modules/cjs/loader.js:966:15)
3|api    |     at Module.Hook._require.Module.require (/home/deploy/.nvm/versions/node/v12.18.0/lib/node_modules/pm2/node_modules/require-in-the-middle/index.js:61:29)
3|api    |     at require (internal/modules/cjs/helpers.js:72:18)
3|api    |     at Object.<anonymous> (/var/www/production/source/node_modules/re2/re2.js:3:13)
3|api    |     at Module._compile (internal/modules/cjs/loader.js:1138:30)
3|api    |     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1158:10)
3|api    |     at Module.load (internal/modules/cjs/loader.js:986:32)
3|api    |     at Function.Module._load (internal/modules/cjs/loader.js:879:14)
3|api    |     at Module.require (internal/modules/cjs/loader.js:1026:19)
3|api    |     at Module.Hook._require.Module.require (/home/deploy/.nvm/versions/node/v12.18.0/lib/node_modules/pm2/node_modules/require-in-the-middle/index.js:80:39)
3|api    |     at require (internal/modules/cjs/helpers.js:72:18)
3|api    |     at Object.<anonymous> (/var/www/production/source/node_modules/forward-email/index.js:10:13)
3|api    |     at Module._compile (internal/modules/cjs/loader.js:1138:30)
3|api    |     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1158:10)
3|api    |     at Module.load (internal/modules/cjs/loader.js:986:32)
3|api    |     at Function.Module._load (internal/modules/cjs/loader.js:879:14)
3|api    |     at Module.require (internal/modules/cjs/loader.js:1026:19)
3|api    |     at Module.Hook._require.Module.require (/home/deploy/.nvm/versions/node/v12.18.0/lib/node_modules/pm2/node_modules/require-in-the-middle/index.js:80:39)
3|api    |     at require (internal/modules/cjs/helpers.js:72:18)
3|api    |     at Object.<anonymous> (/var/www/production/source/app/models/alias.js:2:22)
3|api    |     at Module._compile (internal/modules/cjs/loader.js:1138:30)
3|api    |     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1158:10)

@niftylettuce niftylettuce reopened this Jun 15, 2020
@niftylettuce
Copy link
Author

deploy@forwardemail:~$ node --version
v12.18.0
deploy@forwardemail:~$ yarn --version
1.22.4

@niftylettuce
Copy link
Author

Does not with npm, only yarn it seems.

niftylettuce added a commit to forwardemail/forwardemail.net that referenced this issue Jun 15, 2020
@uhop
Copy link
Owner

uhop commented Jun 16, 2020

Could it be a yarn problem? This error message:

Error: Cannot find module './build/Release/re2'

Indicates that a binary module is not in its place. This module either:

  • Get copied from a secure location specific for your architecture, then verified.
  • If the previous step has failed for any reason, the standard build is started.

I presume you didn't see any errors while installing the package, so it was probably fine. That's it if yarn calls all appropriate steps as defined in the npm documentation.

You can check what step breaks re2 with this command:

$ node -e 'console.log(require("re2"))'
[Function: RE2] {
  getUtf8Length: [Function (anonymous)],
  getUtf16Length: [Function (anonymous)],
  unicodeWarningLevel: 'nothing'
}

Just run it after any operation and you can pinpoint what exact command breaks it.

As you remember I was not able to reproduce the error before. Could you create a minimal project that demonstrates the problem reliably? That would speed up the triage and the eventual fixing.

@uhop
Copy link
Owner

uhop commented Jun 16, 2020

@niftylettuce
Copy link
Author

I'm really not sure. That link is kind of vague and doesn't provide tips to fix the issue.

@niftylettuce
Copy link
Author

I have had zero issues with npm so far. Seems yarn related.

@niftylettuce
Copy link
Author

I also reproduced the problem with yarn in another project of mine, https://github.com/spamscanner/spamscanner and am switching that to use npm instead now.

Repository owner deleted a comment from elazutkin-dynata Jun 16, 2020
@uhop
Copy link
Owner

uhop commented Jun 16, 2020

I also reproduced the problem with yarn in another project of mine

That's bizarre. This project of yours uses re2 1.11.0, which is plain vanilla, with no recent updates. I can understand that something is up in 1.15.0 because:

  • It can load a prebuilt binary from a secure location instead of building it.
  • It uses NPM environment variables to do so.

But 1.11.0 has nothing of that.

Meanwhile, I double-checked that:

  • If loading fails, or the loaded file does not work ⇒ we are back to building from sources.
  • I checked with yarn run env that all NPM environment variables are present as expected.
    • Nevertheless, if they are not there ⇒ back to building code.

Must be some exotic configuration bug, like yarn PnP kicks in without requesting or something to that effect. While I have some yarn users, so far I didn't see any complaints. The only yarn-related ticket I had is #62, which was an enhancement ticket implemented some time ago.

Repository owner deleted a comment from elazutkin-dynata Jun 16, 2020
@uhop
Copy link
Owner

uhop commented Jun 16, 2020

There is a way to prevent the loading of the prebuilt file: just define globally an environment variable named DEVELOPMENT_SKIP_GETTING_ASSET to any non-empty string. See if it helps.

@niftylettuce
Copy link
Author

I think we should just advise people to use npm if they are using re2.

@uhop
Copy link
Owner

uhop commented Jun 16, 2020

@niftylettuce
Copy link
Author

niftylettuce commented Jun 16, 2020 via email

@uhop
Copy link
Owner

uhop commented Jul 5, 2020

Added to the readme, which will be published with the next update.

@uhop uhop closed this as completed Jul 5, 2020
uhop added a commit that referenced this issue Jul 5, 2020
@niftylettuce
Copy link
Author

Thank you @uhop 🙏

@johnnymetz
Copy link

Seeing this error with both yarn and npm in my Vue.js app:

This relative module was not found:

* ./build/Release/re2 in ./node_modules/re2/re2.js

@uhop
Copy link
Owner

uhop commented Sep 27, 2020

Let me guess: you are trying to use the binary Node extension in a browser, right?

@johnnymetz
Copy link

I don't believe so. Just trying to use it in my Vue.js v2.6.12 project. I'm importing it into my entrypoint file with:

import RE2 from 're2'

This throws the relative module was not found error I mentioned a few comments above.

@uhop
Copy link
Owner

uhop commented Sep 28, 2020

How can I repro it?

PS: Out of curiosity: how and why do you use Vue.js in a non-browser project?

@johnnymetz
Copy link

Let me guess: you are trying to use the binary Node extension in a browser, right?

I was confused by this comment but I am indeed running this in a browser.

Steps to reproduce

npm i -g @vue/cli
vue config --set packageManager npm
vue create my-vue-app  # select "Default ([Vue 2] babel, eslint)"
cd my-vue-app
npm i --save re2
npm run serve

App runs fine. Then add the following line to src/main.js:

import RE2 from 're2'

App breaks with the following error:

This relative module was not found:

* ./build/Release/re2 in ./node_modules/re2/re2.js

Dependencies

@vue/cli@4.5.6
npm@6.14.7
node v14.6.0
re@1.15.4

@uhop
Copy link
Owner

uhop commented Sep 29, 2020

This is a binary Node extension written in C++. It is meant to be running in Node, not in browser. It means that your use case is explicitly unsupported.

Usually there are not many benefits running something similar to re2 in a browser. Many cases are about secure evaluation of possibly malicious input written by hackers as a part of DDOS attacks on servers that use regular expressions to parse such input, or web crawlers. Having it in a browser usually means that a hacker runs their attack on themselves slowing down their own browser. (If they control somebody else's browser — it is a separate problem, because they are likely to have an access to that computer as well).

I suggest to re-evaluate the need for running re2 in a browser. If there is a legitimate business reason for that, I suggest to still run it on a server as a micro-service or a similar arrangement.

@viceice
Copy link

viceice commented Dec 1, 2020

We see this regularly on github actions, eg: https://github.com/renovatebot/renovate/runs/1470583624?check_suite_focus=true

I've seen this also today on my wsl2 ubuntu focal while pulled latest changes from master (updated yarn.lock). 😕

Maybe install-from-cache command failed silently?

@uhop
Copy link
Owner

uhop commented Dec 1, 2020

install-from-cache is trivial one-file script. This is the main function: https://github.com/uhop/install-artifact-from-github/blob/6bd8ccbe4fd9a1d7451cdfb47694ca848333cfc2/bin/install-from-cache.js#L139

All non-trivial operations are wrapped in try/catch and the only conceivable way to fail silently is to kill its process.

What I don't like the most is that any code path in main() involves printing something on the console. I don't see its output in your link to a build. It is either suppressed by yarn or it was never run.

Did you notice any patterns with the problem? Like "always on Windows"? When it happens once is it robustly reproducible or intermittent? Any insight is appreciated.

I'll look at the renovate repo hoping I would glean something, but it would be nice to improve my chances to repro the problem.

@viceice
Copy link

viceice commented Dec 1, 2020

I know the output is hidden by yarn. ☹️

It is also happening platform independent.

I guess it could be a node issue, as I had some silent exits on my renovate run when using http2 requests with got. Node was simply terminated with exit code zero. Was very hard so find http2 as cause, but found no solution yet. ☹️

Also seen this issue randomly on node v12 and v14.

@uhop
Copy link
Owner

uhop commented Dec 1, 2020

I had some silent exits on my renovate run when using http2 requests

Thank you, I didn't know that. I'll take it into consideration for my other project.

Is it possible to update your Github action to run yarn install --verbose? Obviously the key here is to add --verbose flag to any existing ones. It should add much more information, which will give us the whole picture of what is going on when it happens again.

@uhop
Copy link
Owner

uhop commented Dec 1, 2020

@viceice
Copy link

viceice commented Dec 2, 2020

I think the problem is related to yarnpkg/yarn#5476

@uhop
Copy link
Owner

uhop commented Dec 3, 2020

It looks like yarn does not work robustly with native modules by design. To wit:

Basically they ask to get rid of postinstall, which is the only way to compile code in npm/yarn.

Their proposed solution? "Use WebAssembly", which is laughable. The current state of WASM is all about compiling a full program and starting it: https://nodejs.org/api/wasi.html — there is no access to JavaScript internals, nor any sort of efficient communications between JS and WASM.

Why do they want to get rid of postinstall? Because if you have a static package, which does not create any files in any ways, and come as is, it can be efficiently distributed as unmodified zip files — PnP. It makes sense, and I understand the rationale, yet it doesn't provide an answer for legitimate modules that should create user/platform-specific resources.

It looks like the logical gap between yarn's PnP and how native modules are compiled provides a foundation for numerous bugs.

On top of that yarn runs scripts in parallel (for efficiency), which provides a fertile ground for race condition bugs as evidenced in multiple issues. This explains why the observed problems are intermittent, hard to reproduce, especially with simple tests, and usually manifest itself in large projects with multiple dependencies where a lot of scripts run in parallel competing for resources. It all works fine as long as the scripts are independent, but as soon as there are any external expectations their execution can break.

Looking through >2000 issues I found many related to native packages. A random example:

While re2 has an explicit postinstall script, removing it leaves the implicit postinstall script that runs node-gyp with binding.gyp. The original author of this issue reported the problem with 1.11, which relied only on implicit scripts — see #69 (comment).

One possible solution is to provide an explicit command that runs postinstall for re2, but it should be done at the user's level after running install of the top project, not at re2 level. Yet re2 should provide some means to do that. At the moment I don't see a good solution for that. I think switching large projects to the latest npm is a better solution irrespective of re2.

PS: BTW, deno is in the same boat — no good story for binary extensions.

@viceice
Copy link

viceice commented Dec 4, 2020

Thanks for investigate.

So we can workaround by simply explicit calling the re2 postinstall at our prepare script, where we check for re2?

@uhop
Copy link
Owner

uhop commented Dec 4, 2020

So we can workaround by simply explicit calling the re2 postinstall at our prepare script, where we check for re2?

It is not that easy. re2 doesn't have postinstall, just install. Apparently, it is not enough. The previous versions before 1.14 didn't even have it relying on defaults, which was the problem too. Calling install or even rebuild might help, but I bet both scripts expect a set of environment variables defined by npm/yarn, which is a project-specific.

It doesn't hurt to try obviously. If it doesn't pick up the right project try to set npm_package_json environment variable to point to package.json of re2. Running it in the root directory of re2 might help as well.

Let me know if any of that helps. Obviously, I want this problem to be solved, but right now I don't even have a cohesive set of recommendations other than "just use npm".

@uhop
Copy link
Owner

uhop commented Dec 4, 2020

BTW, I still think yarn --verbose would be useful to confirm the problem or to discover that something else is responsible for not building the binary extension.

@viceice
Copy link

viceice commented Dec 4, 2020

OK, I'll try to add some code to call the rebuild.

BTW, I still think yarn --verbose would be useful to confirm the problem or to discover that something else is responsible for not building the binary extension.

Already added here and all other yarn install calls on renovate

@viceice
Copy link

viceice commented Dec 7, 2020

We catched a log: pipeline.zip

@uhop
Copy link
Owner

uhop commented Dec 8, 2020

It looks like install of re2 is not called. The "good" run has it right after highlight, the "bad" run doesn't have it at all (shown only the relevant fragment, front timestamps are removed, internal timestamps are masked):

--- bad.txt	2020-12-08 10:49:24.000000000 -0600
+++ good.txt	2020-12-08 10:49:14.000000000 -0600
@@ -11,11 +11,13 @@
    https://github.com/highlightjs/highlight.js/blob/master/VERSION_10_UPGRADE.md
   �[41m�[37m
  -*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*�[0m
- �[2K�[1G�[90mverbose�[39m xx.xxxxxxx
  �[2K�[1G�[90mverbose�[39m xx.xxxxxxx husky > Setting up git hooks
  CI detected, skipping Git hooks installation.
  husky > Done
  �[2K�[1G�[90mverbose�[39m xx.xxxxxxx
+ �[2K�[1G�[90mverbose�[39m xx.xxxxxxx Trying https://github.com/uhop/node-re2/releases/download/1.15.9/win32-x64-83.br ...
+ Writing to build/Release/re2.node ...
+ Done.
  �[2K�[1G�[90mverbose�[39m xx.xxxxxxx D:\a\renovate\renovate\node_modules\dtrace-provider>if not defined npm_config_node_gyp (node "C:\hostedtoolcache\windows\node\14.15.1\x64\node_modules\npm\bin\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" rebuild )  else (node "" rebuild )
  gyp info it worked if it ends with ok
  gyp info using node-gyp@5.1.0

The bad run doesn't have any output from installing re2, but curiously it has an empty line. A remnant from a required run of install?

@uhop
Copy link
Owner

uhop commented Dec 8, 2020

Another thing that coincides with the problem is this ticket: #62 — a direct dependency on node-gyp was added on "yarn's advice". It could be a coincidence but that was the change in re2 1.11 — the one, which was first reported. What if in spite of the "advice" yarn has a bug in the code that runs the required version of node-gyp?

Looking at the verbose output I see a lot of activity, which involves setting up .cache and installing node-gyp specifically for re2 with node_modules, yet no calling it.

A sidenote:

That's why I don't like any complex tooling and try to avoid external dependencies, if possible. They introduce too much unknowns, increase the overall complexity, and make a project a hostage of external whims.

@uhop
Copy link
Owner

uhop commented Dec 8, 2020

A possible solution: I can provide a script, which can be called directly to verify that the binary artifact is installed, and run the installation if it is not there. It can be called directly from the top level.

PS: I think you can remove --verbose from the build action, if you don't need it anymore.

@viceice
Copy link

viceice commented Dec 8, 2020

Thanks for further investigation. A callable script would be very helpful.

Have seen this error again today. 😕

https://github.com/renovatebot/renovate/runs/1514798642?check_suite_focus=true

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug A reported bug. help wanted
Projects
None yet
Development

No branches or pull requests

4 participants