Behavior of cimg/node seems to change with respect to global modules

[vazexqi]

Orb Version
2.x.y

Describe the bug

This bug is more about the behavior of cimg/node rather than the orb. But since I observed in in this orb and I think it will affect others, I'm reporting it here.

It seems that cimg/node docker images other than the 12.6 which is the default for this orb doesn't support npm install sfdx-cli@latest --global

If you do use something more recent than 12.6 you will see an error like

added 1134 packages from 731 contributors in 60.568s
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated mkdirp@0.5.1: Legacy versions of mkdirp are no longer supported. Please update to mkdirp 1.x. (Note that the API surface has changed to use Promises in 1.x.)
npm WARN deprecated request@2.88.0: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated webdriverio@4.14.4: outdated version, please use @next
npm WARN deprecated core-js@2.6.11: core-js@<3 is no longer maintained and not recommended for usage due to the number of issues. Please, upgrade your dependencies to the actual version of core-js@3.
npm WARN checkPermissions Missing write access to /usr/local/lib/node_modules
npm ERR! code EACCES
npm ERR! syscall access
npm ERR! path /usr/local/lib/node_modules
npm ERR! errno -13
npm ERR! Error: EACCES: permission denied, access '/usr/local/lib/node_modules'
npm ERR!  [Error: EACCES: permission denied, access '/usr/local/lib/node_modules'] {
npm ERR!   errno: -13,
npm ERR!   code: 'EACCES',
npm ERR!   syscall: 'access',
npm ERR!   path: '/usr/local/lib/node_modules'
npm ERR! }
npm ERR!
npm ERR! The operation was rejected by your operating system.
npm ERR! It is likely you do not have the permissions to access this file as the current user
npm ERR!
npm ERR! If you believe this might be a permissions issue, please double-check the
npm ERR! permissions of the file and its containing directories, or try running
npm ERR! the command again as root/Administrator.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/circleci/.npm/_logs/2020-07-10T21_20_26_899Z-debug.log

To Reproduce

Use this for loop in your terminal with docker installed:

for value in 12.6 12.18 lts
do
  docker run -it cimg/node:$value npm install sfdx-cli@latest --global
done

You will see that it passes the first loop with 12.6 but then fails with 12.18 or lts.

Expected behavior

It should work with all versions of the cimg/node image.

[gmemstr]

Interesting - taking a look at this now. I suspect appending sudo will resolve the issue like some of our other orbs.

[vazexqi]

@gmemstr I'm not sure what you have done will fix all the issues.

You have made it so that the CLI is now installed as root.

So when you run subsequent commands (including the ones like your built-in authenticate) will throw errors of the form:

Error: EACCES: permission denied, open '/home/circleci/.sfdx/sfdx-config.json'
    at Object.readJSONSync (/usr/local/lib/node_modules/sfdx-cli/node_modules/salesforce-alm/dist/lib/core/srcDevUtil.js:180:28)
    at SfdxConfig.readSync (/usr/local/lib/node_modules/sfdx-cli/node_modules/salesforce-alm/dist/lib/config/SfdxConfig.js:135:40)
    at exports.Config.getApiVersion (/usr/local/lib/node_modules/sfdx-cli/node_modules/salesforce-alm/dist/lib/core/configApi.js:82:51)
    at Object.<anonymous> (/usr/local/lib/node_modules/sfdx-cli/node_modules/salesforce-alm/dist/lib/org/orgPrefRegistry.js:15:52)
    at Module._compile (/usr/local/lib/node_modules/sfdx-cli/node_modules/v8-compile-cache/v8-compile-cache.js:192:30)
    at Function._load (/usr/local/lib/node_modules/sfdx-cli/node_modules/@salesforce/lazy-require/lib/LazyLoader.js:89:24)
    at require (/usr/local/lib/node_modules/sfdx-cli/node_modules/v8-compile-cache/v8-compile-cache.js:161:20)
    at Object.<anonymous> (/usr/local/lib/node_modules/sfdx-cli/node_modules/salesforce-alm/dist/lib/org/scratchOrgSettingsGenerator.js:15:25)
    at Module._compile (/usr/local/lib/node_modules/sfdx-cli/node_modules/v8-compile-cache/v8-compile-cache.js:192:30)

I noticed that this orb is still pegging itself to 12.6 at https://github.com/CircleCI-Public/Salesforce-sfdx-cli-orb/blob/master/src/executors/default.yml#L8 so that might be why you are not seeing the issues. But if you use a different version like cimg/node:12.18 you will see the errors because some things are running as root and some things are not.

[gmemstr]

@vazexqi Curious - I added testing for regressions of this behaviour using the most recent Node image, but I was not expecting this.

I wonder if the better solution is to just specify a custom npm directory, but I can't predict how that might interact with other projects. Ultimately might be what we have to do (could also benefit from caching using this method).

I wasn't really expecting this specific error though - is the file in question being created on install, by the root user?

[vazexqi]

I've roped in others from the CLI team so that they can comment on the best route moving forward.

The current version of our documentation for CircleCI is a bit old and doesn't use this orb (https://developer.salesforce.com/docs/atlas.en-us.sfdx_dev.meta/sfdx_dev/sfdx_dev_ci_circle.htm) so not everyone is affected by this. That being said, we did promote the orb at https://developer.salesforce.com/blogs/2019/12/announcing-circleci-orbs-for-salesforce.html so we expect many to be using this (I am using this which is why I'm hitting this issue and have to peg to 12.6).

[amphro]

I would not use sudo to install the CLI. What about just doing a local install with npm install sfdx-cli@latest (without --global) and adding ./node_modules/.bin/sfdx to the PATH?

[gmemstr]

Digging deeper into this over in #19.

[gmemstr]

@vazexqi I did some digging into this issue, especially regarding authentication when installed with sudo as a global npm module - I wasn't able to replicate the result you had, as demonstrated here: https://app.circleci.com/pipelines/github/CircleCI-Public/Salesforce-sfdx-cli-orb/161/workflows/9e3d75f2-0fcf-4106-b238-965d9645eb6f/jobs/540

I think it's worth investigating whether upgrading the default executor Node version is worth it, since 12.6 is fairly old.

[vazexqi]

Ah, I see in https://app.circleci.com/pipelines/github/CircleCI-Public/Salesforce-sfdx-cli-orb/161/workflows/9e3d75f2-0fcf-4106-b238-965d9645eb6f/jobs/540/parallel-runs/0/steps/0-0 that you are using cimg/node:14.7.

I'll give that a try.

But, I think what @amphro says still stands though - even if installing with sudo works, is that really what we want to do? That being said, the risks of a sudo install on CI is less than your own machine (since it gets torn down after each run). Although, it might be harder to repro issues locally and in CI if locally you are not using sudo but in CI you are.

I am still leaning toward not using sudo and follow Manually change npm’s default directory which is linked from Install the CLI with npm.

@gmemstr - What are your thoughts?

[vazexqi]

@vazexqi I did some digging into this issue, especially regarding authentication when installed with sudo as a global npm module - I wasn't able to replicate the result you had, as demonstrated here: https://app.circleci.com/pipelines/github/CircleCI-Public/Salesforce-sfdx-cli-orb/161/workflows/9e3d75f2-0fcf-4106-b238-965d9645eb6f/jobs/540

Just to set context. The issue I am running into is mentioned earlier

Error: EACCES: permission denied, open '/home/circleci/.sfdx/sfdx-config.json'
....

So that issue that I am running into seems to a different issue (not related to sudo). It's because I am running into, what I believe is, https://support.circleci.com/hc/en-us/articles/360004250693-Restoring-cache-fails-with-Permission-Denied- with my persisted workspaces across workflows. I will need to debug that on my own.

This doesn't change my recommendation to not use sudo but I am mentioning this here so that anyone searching for this issue won't be confused.

[KyleTryon]

Thank you all, I have further investigated the issue and am working with the maintainer of the image on this issue here: CircleCI-Public/cimg-node#214

This configuration issue seems to affect more than just global packages but we see a resolution path and will update all images moving forward.