Not working with github private repo

[seoker]

pnpm version:

v0.67.0

Code to reproduce the issue:

pnpm i with packages.json like:

"dependencies": {
    "@SOME_ORG/SOME_PRIVATE_REPO": "git+https://{token}:x-oauth-basic@github.com/SOME_ORG/SOME_PRIVATE_REPO.git"
}

Expected behavior:

The private repo gets well installed.

Actual behavior:

WARN Error while trying to resolve https://{token}:x-oauth-basic@github.com/SOME_ORG/SOME_PRIVATE_REPO.git via GitHub API
  ERROR fetch failed with status code 404

Additional information:

• node -v prints: v7.6.0
• Windows, OS X, or Linux?: OS X

[vjpr]

Reproduced.

pnpm i git+https://MYPRIVATETOKENREMOVED:x-oauth-basic@github.com/private/repo.git --reporter=ndjson --no-lock
{"time":1494450808877,"hostname":"Vaughan-MacBook-Pro.local","pid":29163,"level":"warn","name":"pnpm","message":"using --no-lock I sure hope you know what you are doing"}
{"time":1494450809053,"hostname":"Vaughan-MacBook-Pro.local","pid":29163,"level":"debug","name":"pnpm:progress","status":"installing","pkg":{"rawSpec":"git+https://MYPRIVATETOKENREMOVED:x-oauth-basic@github.com/private/repo.git"}}
{"time":1494450809058,"hostname":"Vaughan-MacBook-Pro.local","pid":29163,"level":"debug","name":"pnpm:registry","message":"http request uri https://api.github.com/repos/private/repo/commits/HEAD"}
{"time":1494450809059,"hostname":"Vaughan-MacBook-Pro.local","pid":29163,"level":"debug","name":"pnpm:registry","message":"http request no auth needed"}
{"time":1494450809062,"hostname":"Vaughan-MacBook-Pro.local","pid":29163,"level":"info","name":"pnpm:registry","message":"attempt registry request try #1 at 11:13:29 PM"}
{"time":1494450809063,"hostname":"Vaughan-MacBook-Pro.local","pid":29163,"level":"debug","name":"pnpm:registry","message":"http request id 391683d41a458c0c"}
{"time":1494450809064,"hostname":"Vaughan-MacBook-Pro.local","pid":29163,"level":"debug","name":"pnpm:registry","message":"http request GET https://api.github.com/repos/private/repo/commits/HEAD"}
{"time":1494450809676,"hostname":"Vaughan-MacBook-Pro.local","pid":29163,"level":"debug","name":"pnpm:registry","message":"http 404 https://api.github.com/repos/private/repo/commits/HEAD"}
{"time":1494450809688,"hostname":"Vaughan-MacBook-Pro.local","pid":29163,"level":"debug","name":"pnpm:registry","message":"http headers { server: 'GitHub.com',\n  date: 'Wed, 10 May 2017 21:13:30 GMT',\n  'content-type': 'application/json; charset=utf-8',\n  'transfer-encoding': 'chunked',\n  status: '404 Not Found',\n  'x-ratelimit-limit': '60',\n  'x-ratelimit-remaining': '52',\n  'x-ratelimit-reset': '1494453734',\n  'x-github-media-type': 'github.v3; format=json',\n  'access-control-expose-headers': 'ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval',\n  'access-control-allow-origin': '*',\n  'content-security-policy': 'default-src \\'none\\'',\n  'strict-transport-security': 'max-age=31536000; includeSubdomains; preload',\n  'x-content-type-options': 'nosniff',\n  'x-frame-options': 'deny',\n  'x-xss-protection': '1; mode=block',\n  'content-encoding': 'gzip',\n  'x-github-request-id': 'REQUESTID' }"}
{"time":1494450809689,"hostname":"Vaughan-MacBook-Pro.local","pid":29163,"level":"warn","name":"pnpm:git-logger","message":"Error while trying to resolve https://MYPRIVATETOKENREMOVED:x-oauth-basic@github.com/private/repo.git via GitHub API","err":{"pkgid":"repos","statusCode":404,"code":"E404"}}
{"time":1494450810572,"hostname":"Vaughan-MacBook-Pro.local","pid":29163,"level":"debug","name":"pnpm:progress","status":"resolved","pkgId":"github.com/private/repo/HEAD","pkg":{"rawSpec":"git+https://MYPRIVATETOKENREMOVED:x-oauth-basic@github.com/private/repo.git"}}
{"time":1494450810580,"hostname":"Vaughan-MacBook-Pro.local","pid":29163,"level":"info","name":"pnpm:registry","message":"attempt registry request try #1 at 11:13:30 PM"}
{"time":1494450810580,"hostname":"Vaughan-MacBook-Pro.local","pid":29163,"level":"debug","name":"pnpm:registry","message":"http fetch GET https://codeload.github.com/private/repo/tar.gz/HEAD"}
{"time":1494450811179,"hostname":"Vaughan-MacBook-Pro.local","pid":29163,"level":"debug","name":"pnpm:registry","message":"http fetch 404 https://codeload.github.com/private/repo/tar.gz/HEAD"}
{"time":1494450811182,"hostname":"Vaughan-MacBook-Pro.local","pid":29163,"level":"error","name":"pnpm","message":{},"err":{"name":"Error","message":"fetch failed with status code 404","stack":"Error: fetch failed with status code 404\n    at Request.<anonymous> (/Users/Vaughan/nvm/versions/node/v6.5.0/lib/node_modules/pnpm/lib/node_modules/npm-registry-client/lib/fetch.js:58:14)\n    at emitOne (events.js:96:13)\n    at Request.emit (events.js:188:7)\n    at Request.onRequestResponse (/Users/Vaughan/nvm/versions/node/v6.5.0/lib/node_modules/pnpm/lib/node_modules/request/request.js:1074:10)\n    at emitOne (events.js:96:13)\n    at ClientRequest.emit (events.js:188:7)\n    at HTTPParser.parserOnIncomingClient (_http_client.js:472:21)\n    at HTTPParser.parserOnHeadersComplete (_http_common.js:105:23)\n    at TLSSocket.socketOnData (_http_client.js:361:20)\n    at emitOne (events.js:96:13)"}}

Token is simply not being used.

[vcfvct]

Looks like i am having
ERROR fetch failed with status code 406 with version 1.10.0

[zkochan]

In case someone would like to work on this. It is now a lot easier because the git resolver code was moved to @pnpm/git-resolver and the git fetcher to @pnpm/git-fetcher. I just did not move all the git-related tests there yet

[3cp]

I had similar issue on bitbucket private repo.

Failed on

"bcx-ui": "bitbucket:buttonwoodcx/bcx-ui.git#v1.9.5",

Message:

ERROR  Command failed: git ls-remote --refs 
fatal: 'null' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

But works on full url

"bcx-ui": "https://bitbucket.org/buttonwoodcx/bcx-ui.git#v1.9.5",

or

"bcx-ui": "git+ssh://git@bitbucket.org/buttonwoodcx/bcx-ui.git#v1.9.5",

[3cp]

Hi all, I am working on add support of bitbucket/gitlab (public and private) repos in @pnpm/git-resolver.

I have question on the usability of using OAuth token in dependencies of package.json. The OAuth token has limited lifespan, it will become invalid very soon. It seems to me it should not be put in package.json.

For that reason, I am reluctant to add support of token based full url. But I am not sure about your use case. Let me know your thoughts.

For private repos, as long as you have ssh key probably setup on your github/bitbucket/gitlab account, future git-resolver is going to support private repos in either

1. shortcut form username/repo, bitbucket:username/repo, gitlab:username/repo#ref
2. or full git url without using OAuth token

[zkochan]

Changes from pnpm/git-resolver#12 published in pnpm v2.10.0.

[zkochan]

I have no private repositories, so I cannot verify that it works.

If someone verified it with the latest pnpm, lets close this issue

[mvayngrib]

@zkochan didn't work for me! I've given you access to this private repo: https://github.com/mvayngrib/pnpmtest to help you test :) Thanks! I'm looking forward to this feature

[zkochan]

@mvayngrib thanks

if it costs you money to keep the private repo, create it later.

I don't know when will I have time to look into it because I work on some big changes currently

[3cp]

@mvayngrib how do you reference the private repo in package.json?

[mvayngrib]

@zkochan i'm already paying for a few so I can keep this around for a bit (it doesn't cost me extra)
@huochunpeng i didn't, i was trying to install it with one of these:

pnpm i https://github.com/mvayngrib/pnpmtest
pnpm i mvayngrib/pnpmtest

[3cp]

It requires you have ssh public key set probably on github.

If you can run git clone git@github... successfully, the pnpm i should work.

If it doesn’t work, please share your console error of that pnpm i command.

[mvayngrib]

@huochunpeng I can clone it like this:

btw, i've given you access to the repo too :)

$ git clone https://github.com/mvayngrib/pnpmtest
Cloning into 'pnpmtest'...
remote: Enumerating objects: 4, done.
remote: Counting objects: 100% (4/4), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 4 (delta 0), reused 4 (delta 0), pack-reused 0
Unpacking objects: 100% (4/4), done.

[3cp]

@mvayngrib I got no problem

⋊> ~/playground pnpm i mvayngrib/pnpmtest
Packages: +1
+
Resolving: total 1, reused 0, downloaded 1, done

dependencies:
+ pnpmtest 1.0.0

The way you do git clone is using https authentication, not ssh + public key.
There are two ways to access github repo

1. git clone https://... you will be asked for login username/passwd for first time, but git can use saved login info if you setup git properly.
2. git clone git@github.com:mvayngrib/pnpmtest.git this is using ssh protocol where public key can be used. This is the recommended way to access github. https://help.github.com/articles/connecting-to-github-with-ssh/

The pnpm git-resolver supports both saved https login and ssh public key. I am guessing something messed up with your saved https login.

If every time you were asked to key in username/passwd when you do git clone https://, you DO NOT have a saved login.

Anyway, recommend you to follow the article to setup ssh public key for your github account. Using ssh is much smoother user experience than using https.

[mvayngrib]

@huochunpeng interesting! I don't put in my username/password, as you can see from what I pasted. Let me see if i can give you more info, maybe it'll help to figure out what's wrong:

$ pnpm i mvayngrib/pnpmtest --reporter ndjson
{"time":1539434597968,"hostname":"mark.local","pid":26503,"level":"debug","name":"pnpm:package-json","initial":{"name":"pnpmtest-proj","version":"1.0.0","description":"","main":"index.js","scripts":{"test":"echo \"Error: no test specified\" && exit 1"},"keywords":[],"author":"","license":"ISC","dependencies":{"pnpmtest":"github:mvayngrib/pnpmtest"},"readme":"ERROR: No README data found!","_id":"pnpmtest-proj@1.0.0"},"prefix":"/Users/tenaciousmv/Code/pnpmtest-proj"}
{"time":1539434597974,"hostname":"mark.local","pid":26503,"level":"debug","name":"pnpm:stage","message":"resolution_started"}
{"time":1539434597976,"hostname":"mark.local","pid":26503,"level":"debug","name":"pnpm:progress","pkg":{"rawSpec":"mvayngrib/pnpmtest"},"status":"installing"}
{"time":1539434598833,"hostname":"mark.local","pid":26503,"level":"debug","name":"pnpm:progress","status":"resolved","pkgId":"github.com/mvayngrib/pnpmtest/401326db4beb9f33eeea9c6dd445abf19e52de9b","pkg":{"rawSpec":"mvayngrib/pnpmtest"}}
{"time":1539434669308,"hostname":"mark.local","pid":26503,"level":"warn","name":"pnpm:store","message":"Fetching github.com/mvayngrib/pnpmtest/401326db4beb9f33eeea9c6dd445abf19e52de9b failed!"}
{"time":1539434669310,"hostname":"mark.local","pid":26503,"level":"error","name":"pnpm","message":{"code":"E404","uri":"https://codeload.github.com/mvayngrib/pnpmtest/tar.gz/401326db4beb9f33eeea9c6dd445abf19e52de9b","response":{"body":{"_readableState":{"objectMode":false,"highWaterMark":16384,"buffer":{"head":{"data":{"type":"Buffer","data":[52,48,52,58,32,78,111,116,32,70,111,117,110,100,10]},"next":null},"tail":{"data":{"type":"Buffer","data":[52,48,52,58,32,78,111,116,32,70,111,117,110,100,10]},"next":null},"length":1},"length":15,"pipes":null,"pipesCount":0,"flowing":null,"ended":true,"endEmitted":false,"reading":false,"sync":false,"needReadable":false,"emittedReadable":true,"readableListening":false,"resumeScheduled":false,"destroyed":false,"defaultEncoding":"utf8","awaitDrain":0,"readingMore":false,"decoder":null,"encoding":null},"readable":true,"domain":null,"_events":{},"_eventsCount":2,"_writableState":{"objectMode":false,"highWaterMark":16384,"finalCalled":false,"needDrain":false,"ending":true,"ended":true,"finished":true,"destroyed":false,"decodeStrings":true,"defaultEncoding":"utf8","length":0,"writing":false,"corked":0,"sync":false,"bufferProcessing":false,"writecb":null,"writelen":0,"bufferedRequest":null,"lastBufferedRequest":null,"pendingcb":0,"prefinished":true,"errorEmitted":false,"bufferedRequestCount":0,"corkedRequestsFree":{"next":null,"entry":null}},"writable":false,"allowHalfOpen":true,"_transformState":{"needTransform":false,"transforming":false,"writecb":null,"writechunk":null,"writeencoding":"buffer"}},"size":0,"timeout":0,"url":"https://codeload.github.com/mvayngrib/pnpmtest/tar.gz/401326db4beb9f33eeea9c6dd445abf19e52de9b","status":404,"statusText":"Not Found","headers":{}},"attempts":3,"resource":"https://codeload.github.com/mvayngrib/pnpmtest/tar.gz/401326db4beb9f33eeea9c6dd445abf19e52de9b"},"err":{"name":"Error","message":"404 Not Found: https://codeload.github.com/mvayngrib/pnpmtest/tar.gz/401326db4beb9f33eeea9c6dd445abf19e52de9b","code":"E404","stack":"Error: 404 Not Found: https://codeload.github.com/mvayngrib/pnpmtest/tar.gz/401326db4beb9f33eeea9c6dd445abf19e52de9b\n    at /usr/local/lib/node_modules/pnpm/lib/node_modules/@pnpm/tarball-fetcher/lib/createDownloader.js:55:41\n    at Generator.next (<anonymous>)\n    at fulfilled (/usr/local/lib/node_modules/pnpm/lib/node_modules/@pnpm/tarball-fetcher/lib/createDownloader.js:4:58)\n    at <anonymous>\n    at process._tickCallback (internal/process/next_tick.js:188:7)"}}
Error: 404 Not Found: https://codeload.github.com/mvayngrib/pnpmtest/tar.gz/401326db4beb9f33eeea9c6dd445abf19e52de9b
    at /usr/local/lib/node_modules/pnpm/lib/node_modules/@pnpm/tarball-fetcher/lib/createDownloader.js:55:41
    at Generator.next (<anonymous>)
    at fulfilled (/usr/local/lib/node_modules/pnpm/lib/node_modules/@pnpm/tarball-fetcher/lib/createDownloader.js:4:58)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)
Error: 404 Not Found: https://codeload.github.com/mvayngrib/pnpmtest/tar.gz/401326db4beb9f33eeea9c6dd445abf19e52de9b
    at /usr/local/lib/node_modules/pnpm/lib/node_modules/@pnpm/tarball-fetcher/lib/createDownloader.js:55:41
    at Generator.next (<anonymous>)
    at fulfilled (/usr/local/lib/node_modules/pnpm/lib/node_modules/@pnpm/tarball-fetcher/lib/createDownloader.js:4:58)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)

my ~/.gitconfig has:

[url "https://"]
	insteadOf = "git://"
[credential]
	helper = osxkeychain
[hub]
	protocol = https

[3cp]

@mvayngrib I saw you have special gitconfig that forces changing git:// url to https:// ?
Why were you against using ssh protocol?

From your log, it looks like pnpm successfully resolved into a git commit hash github.com/mvayngrib/pnpmtest/401326db4beb9f33eeea9c6dd445abf19e52de9b.
It definitely did some reading from your private repo.

[3cp]

@mvayngrib I am guessing you prefer https because of some firewall?

[mvayngrib]

@huochunpeng yea, I don't remember...I think I had some trouble with npm or yarn at some point, with git or private git dependencies, and I took someone's recommendation to add this :)

[mvayngrib]

the good news is that if I remove that block in gitconfig, pnpm i works. I need to see what other things stop working :)

[StJohn3D]

I am able to install from a private repo locally on windows but my CI server (docker/linux) is failing (npm works in both environments).

pnpm -v 3.8.1

Here's the package.json dependency declaration
"<pkgName>": "git+https://github.com/<orgName>/<repoName>.git#semver:7.3.0",

Here's the error I'm currently seeing

Resolving: total 1502, reused 0, downloaded 8 
·WARN· Fetching github.com/<orgName>/<repoName>/<commitHash> failed! 
ERROR· util.getSystemErrorName is not a function 
at getCode        ../../../../../../../../usr/local/lib/node_modules/pnpm/lib/node_modules/execa/lib/error.js:11  return [util.getSystemEr… 
at makeError      ../../../../../../../../usr/local/lib/node_modules/pnpm/lib/node_modules/execa/lib/error.js:50  const [exitCodeName, exi… 
at handlePromise  ../../../../../../../../usr/local/lib/node_modules/pnpm/lib/node_modules/execa/index.js:112     const returnedError = ma… 
at _tickCallback  internal/process/next_tick.js:188 
TypeError: util.getSystemErrorName is not a function 
    at getCode (/usr/local/lib/node_modules/pnpm/lib/node_modules/execa/lib/error.js:11:16) 
    at makeError (/usr/local/lib/node_modules/pnpm/lib/node_modules/execa/lib/error.js:50:35) 
    at handlePromise (/usr/local/lib/node_modules/pnpm/lib/node_modules/execa/index.js:112:26) 
    at <anonymous> 
    at process._tickCallback (internal/process/next_tick.js:188:7) 

I don't have a private repo or an easy way to create a reproducible test environment but I'm hoping the logs above help.

For now I'll have to use npm for CI builds but I'm happy that I can use pnpm locally to save some disc-space ♥

[3cp]

@StJohn3D I think the existing implementation doesn't understand #semver:7.3.0, please try #v7.3.0 (if that's the tag of your version). If that worked, you can shorten your dep to "<pkgName>": "<orgName>/<repoName>#v7.3.0".

[StJohn3D]

@3cp Thanks for the idea! I tried it again without the semver but unfortunately it's the same thing. Pnpm works locally (windows) but not in the aws build (Linux and nodejs:8.11.0) - even though npm still works in the aws code build.

[3cp]

but not in the aws build

Because it's a private package, you need to configure your AWS vm to generate a ssh key pair and add that public key to your GitHub account.

Sorry, I missed that detail in your first question.

[StJohn3D]

@3cp I already have. Like I said, it works with npm in the aws code build, just not pnpm.

[StJohn3D]

@3cp actually I was wrong, my environment is only configured for https.
The commands I'm using to do this are...

      - echo "machine github.com login $GithubAccessToken" >> ~/.netrc
      - git config user.email "redacted@redacted.com"
      - git config user.name "redacted"

Is there any chance of pnpm working over https like npm does for private repos?

[3cp]

Slightly different.

npm will prompt you first time when you access https private repo, then it will use saved login (by some process in the windows/mac/linux, should be keychain in mac) for future access. I don't remember does npm skip prompt in CI environment.

pnpm behaves more like yarn, it doesn't prompt for https login at all, it only uses what OS provides. So you need to the OS to save/cache the login first before you attempt https private repo.

That's as far as I can remember.

[tstewart-klaudhaus]

I also find it not possible to install a Github private package with PNPM via HTTPS. The following in my package JSON:

dependencies: {
"@<org>/<repo>": "git+https://<token>:x-oauth-basic@github.com/<org>/<repo>.git"
}

Works fine with npm install, but pnpm install fails with:

 ERROR  Command failed with exit code 128: git ls-remote --refs git+ssh://git@github.com/<org>/<repo>.git master
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

It seems that PNPM switches from using HTTPS (which works with the embedded access token) to SSH (which does not work as I do not install local SSH key), whilst NPM is happy to install via HTTPS using token.

In my case I prefer to use a token (access limited to package reader) that I can store in projects that need access to private packages, and that works well in all build environments etc., rather than setting up all environments with SSH key.

[3cp]

Permission denied (publickey).

Currently the implementation only supports public key through ssh, not https.

Try "@<org>/<repo>": "<org>/<repo>" which will use ssh, make sure your github account has the public key of your machine. https://help.github.com/en/github/authenticating-to-github/adding-a-new-ssh-key-to-your-github-account

[tstewart-klaudhaus]

Thanks very much 3cp, I can confirm that installation works once I have created an SSH key pair on my machine and added the public key to my account on GitHub.

I suggest it is worth tracking HTTPS support for private Github packages in the PNPM feature backlog, for two reasons:

• equivalent functionality exists in NPM
• in order to manage access from all environments, e.g. external developers and cloud-based build environments

Example 1: I use a cloud-based container (or configurable build service like Netlify, Vercel, etc.) to build and deploy my app, and whilst I can generate a key pair for that container and add it to my GitHub account, the container now has equivalent access as I do, to all my GitHub repos private or public. That's fine if only I can access the build environment, but what if I need to share access with others?

Example 2: I am working with a customer and recommend them to install a dependency from my private package for their development. They would need to (a) have a github account (b) generate a key pair (c) install than in GitHub (d) be granted access to my repo (e) repeat the above for any of their colleagues, build environments etc.

If I generate a package-reader auth token in my github org I can provide a simple URL for a customer or build environment that provides (revokable) access to private packages, without the need for any other steps.

I recommend PNPM whenever discussing package management, but this is one area where I will have to be careful in setting expectations. Do I recommend PNPM and go through the extra overhead of SSH key management across dev/build environments when working private repos, or advise to use NPM and stick with HTTPS auth tokens, which is the default recommended approach by GitHub and Netlify?

Anyway I have an approach for now, but I suggest keeping this in the roadmap for the future, and I'd be happy to hear of any other aspects I may not have considered here.

[3cp]

I understand and agree on the assessment. At least pnpm needs to cover what npm can do.

[zkochan]

🚢 5.4.0

[robhicks]

What is the disposition of this issue?

I absolutely love pnpm and use it all the time for personal projects. But I work for an organization that will not permit the use of ssh to access it's private repositories on GitHub.

Here's an example of pnpm failing during installation of a package in a private repo:

 ERROR  Command failed with exit code 128: /usr/bin/git ls-remote git+ssh://git@github.com/[private-org]/[private-repo].git HEAD
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

The package is included into the devDependencies section of package.json as follows:

"[private-repo]": "git+https://github.com/[private-org]/[private-repo].git",

To make sure that git always uses https instead of ssh, I have the following in .gitconfig

[credential]
	helper = /usr/share/doc/git/contrib/credential/libsecret/git-credential-libsecret
[url "https://"]
	insteadOf = git://
	insteadOf = ssh://

[RihanArfan]

I've found a solution on a comment on an NPM issue which works with PNPM 7.19.0 too. npm/cli#2610 (comment)

git+https:// >> git@ << github.com/user/repo

In my case I'm using a Codespace which is permitted to access another repo, which works through using Git over HTTPS. https://docs.github.com/en/codespaces/managing-your-codespaces/managing-repository-access-for-your-codespaces