feat(@codeshift/cli): fetching packages from private registry

[alirezamirian]

codeshift cli is capable of fetching packages over npm public registry, to run codemods from. It would be great if it would take npm registry configuration (local .npmrc file, or user-level), into account while resolving packages, OR, it would at least allow for passing --registry option.

[danieldelcore]

Ah – that's very interesting, this is definitely something we need to support!

Under the hood we use live-plugin-manager to talk to NPM. It supports both a npm URL and npm config, so all we would need to do is pass the information over to it and it should be able to pull from private registries as well!

const manager = new PluginManager({
  npmRegistryUrl: "http://your-private-registry",
  npmRegistryConfig: {
    auth: {
      token: "your-token"
    }
  }
});

I've marked this as a priority feature 🔥

[tomheller]

What would be the preferred way of extracting the registry information? If it would just be about the registry, it would be probably easiest to add an additional parameter to the cli command to allow specification of the registry.

When extracting registry and potentially authentication from .npmrc files, I would assume that this can get a bit tricky. Given the tree like setup (project, global, and user config) of the npmrc files, multiple files would need to be considered.

The latter option seems to be a lot nicer to use, if one already has a npmrc setup for multiple / private registries for projects.

[danieldelcore]

Hey folks, just sharing this context from the PR above ☝️

unsure of the best way to approach private registries and if that should change our overall strategy here 🤔 I'd love to get your advice if it's something you're familiar with.

AFAIK, (and as stated here) there are multiple ways to auth against a private registry.

Via an NPM token
Via username, password and email

Perhaps instead of a --registry flag, we could create an --npmrc flag which would accept a path to your custom .npmrc. Which would then include one or more credentials for various registries.

For example, here's an excerpt from one of mine:

@foobar:registry=https://packages.foo.com/custom-npm-registry/npm/
@baz:registry=https://packages.baz.com/custom-npm-registry/npm/

//packages.foo.com/custom-npm-registry/npm/:_password=fff==
//packages.foo.com/custom-npm-registry/npm/:username=dd
//packages.foo.com/custom-npm-registry/npm/:email=dd@foo.com
//packages.foo.com/custom-npm-registry/npm/:always-auth=true

//packages.baz.com/custom-npm-registry/npm/:_password=fff==
//packages.baz.com/custom-npm-registry/npm/:username=dd
//packages.baz.com/custom-npm-registry/npm/:email=dd@foo.com
//packages.baz.com/custom-npm-registry/npm/:always-auth=true
That file could then be parsed and sent to the PluginManager via the [npmRegistryConfig](https://www.npmjs.com/package/live-plugin-manager) property, but unsure if that can handle multiple authentication methods at once.

Perhaps a combination of the two approaches could work here:

• User provides the registry URL via --registry
• User provides the path to --npmrc
• The CLI will parse the npmrc and extract the credentials that match the registry (not sure if people would be comfortable with this)
• The credentials will be passed onto PluginManager
• Fin.

Hmm which all seems a bit messy 🤔 ...

Alternatively, we could just expose --registry and --registryToken and that would be all that we care about. 🤔 If you're running the CLI in a CI context then the token would just be an ENV variable but could be awkward for local runners...

[danieldelcore]

Alternatively, we could just expose --registry and --registryToken and that would be all that we care about. 🤔 If you're running the CLI in a CI context then the token would just be an ENV variable but could be awkward for local runners...

I feel like the simplest path forward might be the best one 🤔 but keen to hear your thoughts. Are any of you using the CLI in CI? Do you folks have the ability to generate your own npm tokens or is that locked-down at your company?

[tomheller]

I feel that pointing to an npmrc file via a CLI flag, then following the approach outlined in #148 (comment) might be the smoothest approach for a user of the CLI.

Although this might require some refactoring within the fetcher logic of the CLI. As far as I can tell, the PluginManager will only accept one registryURL in it's configuration, which would mean that for each package that should be fetched, it might need an individual instance. I concur that this could get quite messy.

Alternatively, we could just expose --registry and --registryToken and that would be all that we care about. thinking If you're running the CLI in a CI context then the token would just be an ENV variable but could be awkward for local runners...

Somewhat this does feel like the most easy and also understandable approach for the users.

We currently use a self hosted registry that has no read access limit other than actual physical network access. It also fall-through proxies to the public npm registry if packages are not found in the self hosted one. You do not need an authentication for reading the packages to get the codeshift.config files.

But that will very well be different for users that have private npm registries with other access restrictions in place.

We are not using the CLI in an CI environment. It usually falls onto someone within the team to run upgrades on some packages and they would run the CLI on a local setup and PR the changeset manually.

Let me know how to proceed with the PR, as I'm still happy to contribute here. Maybe you would also have some pointers for me on what you preferred way of unit-testing this feature would be, if we go forward with the simple exposed registry and token approach.

[danieldelcore]

Awesome! Yeah totally agree, let's proceed with the --registryToken approach. I think the other is too complicated, I'm probably overthinking it anyway!

Thanks again!

[alirezamirian]

Isn't it a better option to use npm itself to get the configured registry (by running npm config get @${scope}:registry)? It will take npmrc files and everything into account, and it's easy to implement/use.
Of course an explicit input for npm registry configuration could also make sense if it's deemed necessary. But just using the current npm registry configuration by default makes sense to me, and I think is enough in most of the cases.

[tomheller]

@alirezamirian In theory, yes that would be easier. But it would still leave us parsing the output of the child process. Additionally the npm config get command will not give access to the authentication token or username/password as it is listed as (protected).

See my example

npm config list @foo:registry  
; "user" config from /home/me/.npmrc

//packages.foo.com/custom-npm-registry/npm/:_password = (protected) 
//packages.foo.com/custom-npm-registry/npm/:username = "my.user.name" 
//localhost:4873/:_authToken = (protected) 

I would still think that the explicit flags on the CLI make sense, because it does not hide any arbitrary parsing logic and makes it very clear, which connection will be used in the CLI run.
Of course the --registry and --registryToken flags need to be optional and the CLI needs to use the default npm registry with read access by default.

[alirezamirian]

But it would still leave us parsing the output of the child process

The output is as straightforward as it could be. It's just the url. So at most a .trim() call is needed only.

Additionally the npm config get command will not give access to the authentication token or username/password as it is listed as (protected).

Fair enough. An option would make sense for such use cases, but I still think, getting the registry url from npm by default makes sense, at the same time.

[tomheller]

getting the registry url from npm by default makes sense, at the same time.

I do agree, but given that the CLI accepts multiple packages to update, this would mean that we would need to split the live-plugin-manager instances for the fetcher, as this one can only deal with one registry configuration at a time. I'll explore on how much change would be required to do this in the overall CLI setup and structure.
And if we cannot get the authentication from the npm config get call, this would mean that the --registryToken flag, if necessary would only match one of the registries. This somehow feels awkward imho.