Fix npm_execpath for bundled release #2843

kouhin · 2017-03-06T09:26:42Z

Summary

Docker is using a bundled version of yarn, and the yarn.js is moved to /usr/local/bin/yarn.
However yarn always generates npm_execpath by the following code:

env.npm_execpath = path.join(__dirname, '..', '..', 'bin', 'yarn.js');

Then the generated npm_execpath with docker pre-installed yarn will be /bin/yarn.js which does not exist.

Test plan

Test 1: Bundled version:

Run:

docker run --rm -it node:7.7 /bin/bash -c 'echo "{\"scripts\": {\"test\": \"echo \$npm_execpath\"}}" > package.json && yarn test'

Output:

/usr/local/bin/yarn

Test 2: Common installed version:

docker run --rm -it node:7.7 /bin/bash -c 'curl -o- -L https://yarnpkg.com/install.sh | bash && echo "{\"scripts\": {\"test\": \"echo \$npm_execpath\"}}" > package.json && yarn test'

Output:

/root/.config/yarn/global/node_modules/yarn/bin/yarn.js

#fix #2345

bestander · 2017-03-07T15:25:53Z

Awesome, another issue with bundled build fixed!
How about the other places where __dirname__ is used?

kouhin · 2017-03-08T09:41:34Z

@bestander Thank you!
I'm trying to solve the __dirname problem of node-gyp (https://github.com/kouhin/yarn/blob/9349e01a8a2ddd44d363d2d2f92175d706a590a6/src/util/execute-lifecycle-script.js#L120). It's a little complex, but wish I could solve it.

bestander · 2017-03-08T09:45:30Z

The trick with node-gyp is that we don't bundle it with the yarn.js build.
So it must be installed separately.

See discussions and workarounds:
#2266
nodejs/docker-node#346
#2828

The workaround to make Yarn install node-gyp and then yarn.js be able to access it.

This reverts commit 2f49dcf.

kouhin · 2017-03-09T05:50:17Z

@bestander Yes, I see. yarn global add node-gyp is a simple workaround, but what about this:

docker run --rm -it node:7.7 /bin/bash -c "yarn global add yarn && yarn add webworker-threads"

It will cause an error:

error /root/node_modules/webworker-threads: Command failed.
Exit code: 1
Command: sh
Arguments: -c node-gyp rebuild
Directory: /root/node_modules/webworker-threads
Output:
module.js:472
    throw err;
    ^

Error: Cannot find module '/root/.config/yarn/global/node_modules/yarn/node_modules/node-gyp/bin/node-gyp.js'
    at Function.Module._resolveFilename (module.js:470:15)
    at Function.Module._load (module.js:418:25)
    at Module.runMain (module.js:605:10)
    at run (bootstrap_node.js:425:7)
    at startup (bootstrap_node.js:146:9)
    at bootstrap_node.js:540:3

yarn global add yarn will install a common version of yarn with node-gyp,
yarn is installed in /root/.config/yarn/global/node_modules/yarn, however node-gyp is installed in /root/.config/yarn/global/node_modules/node-gyp.

Maybe this code causes the error or I'm wrong?
https://github.com/yarnpkg/yarn/blob/master/src/util/execute-lifecycle-script.js#L120

  // add node-gyp
  pathParts.unshift(path.join(__dirname, '..', '..', 'bin', 'node-gyp-bin'));

kouhin · 2017-03-09T06:00:24Z

I may made a mistake.
This one works:

docker run --rm -it node:7.7 /bin/bash -c "yarn global add yarn && echo \"env\" >> /root/.config/yarn/global/node_modules/yarn/bin/node-gyp-bin/node-gyp && yarn add webworker-threads"

Why?

bestander · 2017-03-09T06:07:21Z

Are you sure you do `yarn global add yarn`?

…

On 9 March 2017 at 06:00, HOU Bin ***@***.***> wrote: I may made a mistake. This one works: docker run --rm -it node:7.7 /bin/bash -c "yarn global add yarn && echo \"env\" >> /root/.config/yarn/global/node_modules/yarn/bin/node-gyp-bin/node-gyp && yarn add webworker-threads" Why? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2843 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACBdWGS4TKmZHGuqyDjIDIjamOCpdPxbks5rj5V_gaJpZM4MT5zY> .

kouhin · 2017-03-09T06:09:56Z

Yes, yarn global add yarn
A symlink will be created at /usr/local/bin and override the bundled one.

/usr/local/bin/yarn -> ../../../root/.config/yarn/global/node_modules/yarn/bin/yarn.js

bestander · 2017-03-10T16:48:58Z

I haven't followed the log through yet but I would advise against yarn global add yarn, it confuses what yarn binary is used.
What if you do yarn global add yarn 2 or 3 times in a row?

kouhin · 2017-03-10T17:04:14Z

@bestander It's easy to confirm what yarn binary is used.

❯ docker run --rm -it node:7.7 /bin/bash
root@2ce437d7a73d:/# which yarn
/usr/local/bin/yarn
root@2ce437d7a73d:/# cd /usr/local/bin
root@2ce437d7a73d:/usr/local/bin# ls -al
total 37932
drwxrwxr-x  2  500   500     4096 Mar  2 19:40 .
drwxrwsr-x 18 root staff     4096 Mar  2 19:40 ..
-rwxrwxr-x  1  500   500 35393839 Mar  2 00:23 node
lrwxrwxrwx  1 root root        19 Mar  2 19:40 nodejs -> /usr/local/bin/node
lrwxrwxrwx  1  500   500       38 Mar  2 00:23 npm -> ../lib/node_modules/npm/bin/npm-cli.js
-rwxr-xr-x  1 root root   3432504 Mar  2 19:40 yarn

The default yarn is a bundled yarn binary, and then

root@2ce437d7a73d:/usr/local/bin# yarn global add yarn
yarn global v0.21.3
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Installed "yarn@0.21.3" with binaries:
      - yarn
      - yarnpkg
Done in 12.15s.
root@2ce437d7a73d:/usr/local/bin# which yarn
/usr/local/bin/yarn
root@2ce437d7a73d:/usr/local/bin# ls -al
total 34584
drwxrwxr-x  2  500   500     4096 Mar 10 14:59 .
drwxrwsr-x 19 root staff     4096 Mar 10 14:59 ..
-rwxrwxr-x  1  500   500 35393839 Mar  2 00:23 node
lrwxrwxrwx  1 root root        19 Mar  2 19:40 nodejs -> /usr/local/bin/node
lrwxrwxrwx  1  500   500       38 Mar  2 00:23 npm -> ../lib/node_modules/npm/bin/npm-cli.js
lrwxrwxrwx  1 root root        63 Mar 10 14:59 yarn -> ../../../root/.config/yarn/global/node_modules/yarn/bin/yarn.js
lrwxrwxrwx  1 root root        63 Mar 10 14:59 yarnpkg -> ../../../root/.config/yarn/global/node_modules/yarn/bin/yarn.js

Common yarn overrides pre-installed version.

We can also confirm the version, and install webworker-threads again.

❯ docker run --rm -it node:7.7.0 /bin/bash -c "echo '==BEFORE==' && yarn -v && echo '==== INSTALL YARN ====' && yarn global add yarn@0.22.0 && echo '==AFTER==' && yarn -v && yarn add webworker-threads"
==BEFORE==
yarn install v0.21.3
info No lockfile found.
[1/4] Resolving packages...
success Nothing to install.
success Saved lockfile.
Done in 0.14s.
==== INSTALL YARN ====
yarn global v0.21.3
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Installed "yarn@0.22.0" with binaries:
      - yarn
      - yarnpkg
Done in 9.92s.
==AFTER==
yarn install v0.22.0
[1/4] Resolving packages...
success Already up-to-date.
Done in 0.10s.
yarn add v0.22.0
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
[1/1] ⠂ webworker-threads
[-/1] ⠂ waiting...
error /node_modules/webworker-threads: Command failed.
Exit code: 1
Command: sh
Arguments: -c node-gyp rebuild
Directory: /node_modules/webworker-threads
Output:
module.js:472
    throw err;
    ^

Error: Cannot find module '/root/.config/yarn/global/node_modules/yarn/node_modules/node-gyp/bin/node-gyp.js'
    at Function.Module._resolveFilename (module.js:470:15)
    at Function.Module._load (module.js:418:25)
    at Module.runMain (module.js:605:10)
    at run (bootstrap_node.js:425:7)
    at startup (bootstrap_node.js:146:9)
    at bootstrap_node.js:540:3
info Visit https://yarnpkg.com/en/docs/cli/add for documentation about this command.

use process.mainModule.filename as npm_execpath

9349e01

kouhin changed the title ~~Use process.mainModule.filename as npm_execpath~~ Fix npm_execpath for bundled release Mar 7, 2017

bestander merged commit 2f49dcf into yarnpkg:master Mar 7, 2017

This was referenced Mar 8, 2017

yarn-e2e/label=docker,os=ubuntu-14.04 #123 failed #2866

Closed

yarn-e2e/label=docker,os=ubuntu-16.04 #123 failed #2867

Closed

yarn-e2e/label=docker,os=ubuntu-12.04 #123 failed #2868

Closed

skevy pushed a commit to expo/yarn that referenced this pull request Mar 8, 2017

Revert "use process.mainModule.filename as npm_execpath (yarnpkg#2843)"

bc2fd0e

This reverts commit 2f49dcf.

kouhin deleted the feature/fix_npm_execpath branch April 18, 2017 05:20

timwis mentioned this pull request Jun 21, 2017

[yarn] Doesn't seem to support this installation path mysticatea/npm-run-all#100

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix npm_execpath for bundled release #2843

Fix npm_execpath for bundled release #2843

kouhin commented Mar 6, 2017

bestander commented Mar 7, 2017

kouhin commented Mar 8, 2017

bestander commented Mar 8, 2017

kouhin commented Mar 9, 2017 •

edited

Loading

kouhin commented Mar 9, 2017 •

edited

Loading

bestander commented Mar 9, 2017 via email

kouhin commented Mar 9, 2017

bestander commented Mar 10, 2017

kouhin commented Mar 10, 2017

Fix npm_execpath for bundled release #2843

Fix npm_execpath for bundled release #2843

Conversation

kouhin commented Mar 6, 2017

Test 1: Bundled version:

Test 2: Common installed version:

bestander commented Mar 7, 2017

kouhin commented Mar 8, 2017

bestander commented Mar 8, 2017

kouhin commented Mar 9, 2017 • edited Loading

kouhin commented Mar 9, 2017 • edited Loading

bestander commented Mar 9, 2017 via email

kouhin commented Mar 9, 2017

bestander commented Mar 10, 2017

kouhin commented Mar 10, 2017

kouhin commented Mar 9, 2017 •

edited

Loading

kouhin commented Mar 9, 2017 •

edited

Loading