Auth error when scoped private package has unscoped name in package.json

[rjp44]

Bug description

Dependencies on private packages fail for (at least) github package manager with an authorization error during fetch phase.

I have a project that has a dependency on a private package hosted on the Github package manager. npm install is able to access this package using my config, yarn install cannot.

This is sort of a dup of previous issues like #4451 (there are several non-overlapping issues about private package registry auth that I found trying to debug it).

This is a specific reproducible case and I think I have a vague idea of the code issue, so opening a new issue. Please merge it with an older open issue if it overlaps.

Config

user .npmrc:

//npm.pkg.github.com/:_authToken=MY_GITHUB_TOKEN

project dir .npmrc:

@MY_ORG:registry=https://npm.pkg.github.com/
registry=https://registry.npmjs.org/

What I've Tried

Every permutation of .npmrc/.yarnrc suggested by the various issues/how-to's here and on stackoverflow with several different versions of yarn has been tried (registry with/without trailing slash, private registry org name etc etc). The most working I can get it, fails like this on all yarn versions including current yarn master (MY_GITHUB_TOKEN, MY_ORG, and MY_PACKAGE are anonymisations of valid Github User Token, Private Package Org, Package name, to protect the innocent)..

Command

$ ../yarn/bin/yarn install
yarn install v1.23.0-0
info No lockfile found.
[1/4] 🔍  Resolving packages...
warning tap > coveralls > request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
[2/4] 🚚  Fetching packages...
error An unexpected error occurred: "https://npm.pkg.github.com/download/@MY_ORG/MY_PACKAGE/1.0.4/1cc5433537c398ef3c14e6df736436e48d7a22c501d7f3208d9ab16347976dc5: Request failed \"401 Unauthorized\"".
info If you think this is a bug, please open a bug report with the information provided in "/Users/rob/Aplisay/code/smb-database/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

Further immediately after the above fails, I can fetch the package manually using the token:

curl --oauth2-bearer MY_GITHUB_TOKEN https://npm.pkg.github.com/download/@MY_ORG/MY_PACKAGE/1.0.4/1cc5433537c398ef3c14e6df736436e48d7a22c501d7f3208d9ab16347976dc5

works and fetches the package (well gives me a 302 redirect to an aws bucket which then works).

Yarn seems to have the right config to fetch the package:

../yarn/bin/yarn config list
yarn config v1.23.0-0
info yarn config
{
  'version-tag-prefix': 'v',
  'version-git-tag': true,
  'version-commit-hooks': true,
  'version-git-sign': false,
  'version-git-message': 'v%s',
  'init-version': '1.0.0',
  'init-license': 'MIT',
  'save-prefix': '^',
  'bin-links': true,
  'ignore-scripts': false,
  'ignore-optional': false,
  registry: 'https://registry.yarnpkg.com',
  'strict-ssl': true,
  'user-agent': 'yarn/1.23.0-0 npm/? node/v12.17.0 darwin x64',
  lastUpdateCheck: 1595744734733
}
info npm config
{
  '//npm.pkg.github.com/:_authToken': 'MY_GITHUB_TOKEN',
  '@MY_ORG:registry': 'https://npm.pkg.github.com/',
  registry: 'https://registry.npmjs.org/'
}

It is only the fetching phase that fails to send the token, it apparently successfully auths to npm.pkg.github.com during package resolution. --verbose tells me:

$ ../yarn/bin/yarn install
yarn install v1.23.0-0
...
[1/4] 🔍  Resolving packages...
verbose 0.470612147 Performing "GET" request to "https://npm.pkg.github.com/@MY_ORG%2fMY_PACKAGE".
verbose 0.963959286 Request "https://npm.pkg.github.com/@MY_ORG%2fMY_PACKAGE" finished with status code 200.
...
[2/4] 🚚  Fetching packages...
verbose 12.033322146 Performing "GET" request to "https://npm.pkg.github.com/download/@MY_ORG/MY_PACKAGE/1.0.4/1cc5433537c398ef3c14e6df736436e48d7a22c501d7f3208d9ab16347976dc5".
verbose 12.174885237 Error: https://npm.pkg.github.com/download/@MY_ORG/MY_PACKAGE/1.0.4/1cc5433537c398ef3c14e6df736436e48d7a22c501d7f3208d9ab16347976dc5: Request failed "401 Unauthorized"

Environment

• Node Version: 12.17.0
• Yarn v1 Version: 1.23.0-0
• OS and version: MacOS

Several versions of yarn tried, up to 1.22.3, error is slightly different (checksum error masks 401), but 1.22.4 and current master head both give same 401 result.

[rjp44]

Have debugged this and I think I can see why NpmRegistry.request is failing to send the token.
The fetcher (e.g. tarball-fetcher) calls NpmRegistry.request with parameters

pathname: "https://npm.pkg.github.com/download/@MY_ORG/MY_PACKAGE/..."
packageName: "MY_PACKAGE"

The scope has been stripped off packageName for the fetch request.

This means that NpmRegistry.request doesn't think it is a scoped request that needs authentication, and even if it did (by setting 'alwaysAuth') it can't find an auth token using this.getAuth("MY_PACKAGE") as that resolves to the unauthenticated public registry.

I have nowhere near enough of an idea about yarn internals to know if the issue is that the fetcher is passing the un-scoped package name, or the request method needs to be smarter about finding an auth token for this path/name combination.

I have a fix that works for me by having NpmRegistry.request decide if the package is scoped and, if so which token to use based on pathname as well as packageName, but have no idea if this the actual fix or a band-aid for a problem elsewhere.

It does however seem to fix this issue and doesn't cause any test regressions, should I submit a pull request?

[rjp44]

This wasn't really a yarn issue.

Somehow a package in github package manager with the scoped @MY_ORG/MY_PACAKGE url, but with (unscoped):

{ 
  "name": "MY_PACKAGE", ...
}

in package.json had been created.

Npm was happy to publish this, yarn failed with this issue when I tried to use it. Maybe an error check somewhere during package resolution would be a good idea?

[zodman]

i had the same issue .but the definition of the package is:

    "serverless-feature-flag": "npm:@myscope/serverless-feature-flag@^1.7.7-38",

it install good with:

"@myscope/serverless-feature-flag": "^1.7.7-38",

[zodman]

cache problem i solved

[mrjasongorman]

I had the same problem, so thankful for this thread 🤝

Turns out a package is published for example with this package name @MyScope/package-name and your project .yarnrc has things setup for @myscope/package-name Yarn is very case sensitive, and won't authenticate. Republishing with the package name as @myscope/package-name fixed the problem 🙌

[rachelslurs]

Another ah ha moment for me using private scoped github package registry packages was found here: yarnpkg/berry#316 (comment)

Using yarn 3, you don't have to use .npmrc. Instead, you can use npmAuthIdent: "base64(username:password)" under your npmScopes section of your .yarnrc.yml:

npmScopes:
  yourorg:
    npmAlwaysAuth: true
    npmRegistryServer: "https://npm.pkg.github.com"
    npmAuditRegistry: "https://npm.pkg.github.com"
    npmPublishRegistry: "https://npm.pkg.github.com"
    npmAuthIdent: "base64(whateverusernamedontmatter:${NPM_TOKEN})"
    npmAuthToken: ${NPM_TOKEN}

I know npmAuthToken is preferable, but I haven't found a way to make it work with github package registry.

Also another thing to note is yarn doesn't handle adding things to the process.env for you with regards to environment variables, so in order to make ${NPM_TOKEN} evaluate properly in the .yarnrc.yml, you need to either set it when you install: NPM_TOKEN=abcd1234 yarn install or you can set it as an ENV in your Dockerfile.