Can't specify which registry pnpm should use

[dancallaghan]

Is there an existing issue for this?

• I have searched the existing issues

Package ecosystem

pnpm

Package manager version

8.9.2

Language version

18.17.1

Manifest location and content before the Dependabot update

No response

dependabot.yml content

Here is a minimal version of our dependabot.yml. I can provide more detail if required

version: 2
registries:
  npm-github:
    type: npm-registry
    url: https://npm.pkg.github.com
    username: foo@bar.com
    password: ${{secrets.GH_TOKEN}}
updates:
  - package-ecosystem: npm
    directory: /
    open-pull-requests-limit: 10
    registries:
      - npm-github

Updated dependency

No response

What you expected to see, versus what you actually saw

We are using https://npm.pkg.github.com as a private registry for our scoped private packages (e.g. @org/logger). We are running into issues because dependabot is checking https://npm.pkg.github.com registry first for all public packages. In most cases this is 404ing and falling back to the global registry https://registry.npmjs.org.

There are situations where the public package exists in the GitHub registry, however it has stale information about the package. This is causing issues like this:

The latest release of @types/node is "17.0.18".

(Spoiler, this is not the latest release of @types/node 😅)

Unlike npm and yarn it looks like there is no way to specify a locked_registry with pnpm so we can tell dependabot to use a specific registry when resolving a dependency. With a pnpm lockfile, you can include a tarball URL, however from looking at the code this is not respected, unlike properties like resolved.

Not sure if there is another way to achieve this? For example, is there a way to tell dependabot only to use a scoped private registry for scoped dependencies?

Native package manager behavior

No response

Images of the diff or a link to the PR, issue, or logs

proxy | 2023/10/19 21:54:50 [032] GET https://npm.pkg.github.com:443/@sentry%2Fnode
proxy | 2023/10/19 21:54:50 [032] * authenticating npm registry request (host: npm.pkg.github.com, basic auth)
proxy | 2023/10/19 21:54:50 [032] 404 https://npm.pkg.github.com:443/@sentry%2Fnode
proxy | 2023/10/19 21:54:51 [034] GET https://registry.npmjs.org:443/@sentry%2Fnode
proxy | 2023/10/19 21:54:51 [034] 200 https://registry.npmjs.org:443/@sentry%2Fnode
proxy | 2023/10/19 21:54:52 [036] GET https://registry.npmjs.org:443/@sentry%2Fnode/7.74.1
proxy | 2023/10/19 21:54:52 [036] 200 https://registry.npmjs.org:443/@sentry%2Fnode/7.74.1

…

proxy | 2023/10/19 21:56:24 [176] GET https://npm.pkg.github.com:443/@types%2Fnode
proxy | 2023/10/19 21:56:24 [176] * authenticating npm registry request (host: npm.pkg.github.com, basic auth)
proxy | 2023/10/19 21:56:24 [176] 200 https://npm.pkg.github.com:443/@types%2Fnode

One of the errors

Handled error whilst updating @google-cloud/storage: dependency_file_not_resolvable
{:message=>"Error whilst updating @google-cloud/storage in /pnpm-lock.yaml:
 WARN  Unsupported engine: wanted: {\"node\":\"18.17.1\"} (current: {\"node\":\"v18.18.2\",\"pnpm\":\"8.9.2\"})
 WARN  The field \"resolutions\" was found in /home/dependabot/dependabot-updater/repo/subdir/package.json. This will not take effect. You should configure \"resolutions\" at the root of the workspace instead.
Progress: resolved 1, reused 0, downloaded 0, added 0
Progress: resolved 39, reused 38, downloaded 0, added 0
Progress: resolved 63, reused 62, downloaded 0, added 0
Progress: resolved 89, reused 69, downloaded 0, added 0
Progress: resolved 127, reused 77, downloaded 0, added 0
 ERR_PNPM_NO_MATCHING_VERSION  No matching version found for @types/node@18.17.17

This error happened while installing the dependencies of @google-cloud/storage@7.3.1
 at retry-request@7.0.1
 at @types/request@2.48.8

The latest release of @types/node is \"17.0.18\".

Other releases are:
  * newest8: 8.10.59
  * newest9: 9.6.55
  * ts2.0: 12.12.6
  * ts2.1: 12.12.6
  * ts2.2: 12.12.6
  * ts2.3: 12.12.6
  * ts2.4: 12.12.6
  * ts2.5: 12.12.6
  * ts2.6: 12.12.6
  * ts2.7: 12.12.6
  * ts2.8: 13.13.4
  * ts2.9: 14.0.1
  * ts3.0: 14.6.0
  * ts3.1: 14.10.1
  * ts3.2: 14.14.9
  * ts3.3: 14.14.20
  * ts3.4: 14.14.31
  * ts3.5: 15.6.1
  * ts3.6: 16.6.2
  * ts3.7: 16.11.7
  * ts3.8: 17.0.18
  * ts3.9: 17.0.18
  * ts4.0: 17.0.18
  * ts4.1: 17.0.18
  * ts4.2: 17.0.18
  * ts4.3: 17.0.18
  * ts4.4: 17.0.18
  * ts4.5: 17.0.18
  * ts4.6: 17.0.18
  * ts4.7: 17.0.18

If you need the full list of all 599 published versions run \"$ pnpm view @types/node versions\"."}

Running the view command locally shows that the version exists:

$ pnpm view @types/node versions | grep 18.17.17
  '18.17.17',     '18.17.18',     '18.17.19',     '18.18.0',      '18.18.1',

Smallest manifest that reproduces the issue

No response

[dancallaghan]

Extra Detail

I have done a fair bit of debugging locally and thought it might be helpful to link to some parts of the code that are relevant.

From my understanding, for a dependency, it looks for a locked registry first. This looks through the requirements for a source, which in our case is always nil. In the FileParser, it looks for the source, however this doesn't take into account the tarball property. This is understandable because the pnpm-lock.yaml does not have this by default however it can be included optionally.

After it can't find a locked registry, it goes through the known registries and uses those. In our case it tries the private registry first and then the central registry after.

Strange Behaviour with @types/*

Someone on our team figured out why some of the packages are being found in the GitHub registry but appear out-of-date.

It looks like there is a mirror that used to publish the DefinitelyTyped packages to GitHub, however it has since been abandoned. So the public packages are sitting there but are stuck in time.

https://github.com/types/_definitelytypedmirror
https://github.com/types/_definitelytypedmirror/pkgs/npm/node

[rosenquist]

We have been having the same issue in one of our repos for a while with the same setup (pnpm + private github package registry) but haven't had time to look into it further until today. Looking at our history it seems like we started getting the errors around Sep 26. Before that dependabot was working fine with both pnpm and private registry.

[deivid-rodriguez]

This is caused by #8094.

A fix would indeed be to add support for parsing locked pnpm-lock.yaml registries as suggested.

My question is: if you commit a npmrc with the proper scope configuration, does it fix the issue?

[rosenquist]

This is caused by #8094.

A fix would indeed be to add support for parsing locked pnpm-lock.yaml registries as suggested.

My question is: if you commit a npmrc with the proper scope configuration, does it fix the issue?

We were able to temporarily remove the private registry from our config since we haven't added any private deps to our project yet. When we need it we'll make sure to configure an npmrc accordingly. Thanks for the help @deivid-rodriguez

[cysp]

Thank you for the response and suggestions, @deivid-rodriguez

I’m a colleague of @dancallaghan who is currently on leave.

We have a .npmrc specifying the registry for our private scope committed to the repository, and have attempted to set a default registry in it too, but unfortunately it hasn’t resolved our problem:

registry=https://registry.npmjs.org
@scentregroup:registry=https://npm.pkg.github.com
//npm.pkg.github.com/:_authToken=${GH_TOKEN}
save-prefix=''

Our private packages do get found correctly from ghpr, but the problem remains that ghpr is given priority and is allowed to provide packages that should be provided by the public registry but happen to exist in ghpr with colliding names.

It occurs to me that we haven’t tried specifying the scopes for public packages (like @types/*) but that feels like it would be a long game of whack-a-mole.

Is this the form of .npmrc you expected in your question? Please let me know if there’s any more information I can provide to help. 🙂

[deivid-rodriguez]

Yes, that seems unexpected to me. With that .npmrc my understanding is that Dependabot should pick gems scoped to @scentregroup from the private registry, and anything else from the public one. I'll see if I can figure out what's going on 👍.

[dancallaghan]

Hi @deivid-rodriguez, I'm back from vacation today.

Just wanted to let you know that I have done some debugging locally, dry running dependabot-core @ b39f6b0 against our repository.

I have set LOCAL_CONFIG_VARIABLES to try and replicate the dependabot.yml configuration that we have:

export LOCAL_CONFIG_VARIABLES="[{\"type\":\"npm_registry\",\"registry\":\"npm.pkg.github.com\",\"username\":\"username@example.com\",\"password\":\"$TOKEN\"}]"

And then dry run against our repo:

$ bin/dry-run.rb npm_and_yarn ScentreGroup/our-repo

Thought it might be of interest the generated .npmrc file:

$ cat tmp/ScentreGroup/our-repo/.npmrc 

registry=https://registry.npmjs.org
@scentregroup:registry=https://npm.pkg.github.com

save-prefix=''

[deivid-rodriguez]

Sorry for the delay @dancallaghan, currently looking into both issues reported here, namely:

• Dependabot can't infer the proper registry for each dependency by locking at registry urls in pnpm-lock.yaml.
• Dependabot incorrectly tries to infer the proper registry for each dependency by looking at presence in the registry, even when registries & scopes are explicitly configured in .npmrc.

[deivid-rodriguez]

Working on fixes at #8330. Feel free to give that a try, it should be ready other than adding some tests!

[dancallaghan]

Hi @deivid-rodriguez, thank you so much for looking into this 🙏

I've just tested your branch locally on our use-case and it looks like it's part of the way there to solving the issue.

The remaining issue I can see is that scoped dependencies are not using the configured global registry when their scope is not defined in the .npmrc file.

e.g. when it looks up @commitlint/cli it correctly identifies it as a scoped dependency, however the @commitlint scope is not defined in our .npmrc. It then falls back on the unwanted code path of looking up first_registry_with_dependency_details instead of looking up configured_global_registry.

I tweaked the code of explicit_registry_from_rc a tad and it seems to be using the correct registry:

def explicit_registry_from_rc(dependency_name)
  if dependency_name.start_with?("@") && dependency_name.include?("/")
    scope = dependency_name.split("/").first
    scoped_registry(scope) || configured_global_registry
  else
    configured_global_registry
  end
end

[deivid-rodriguez]

Thanks, that makes sense!

Can you create a PR to my branch with that tweak to see if it passes CI?