Unable to install plugins from private repo

[esatterwhite]

I've setup a private git repository on github that houses a krew index. I have a plugin that I can install manually locally, pointed krew at the registry successfully. However, It cannot seem to download the tarball from the registry. It is mostly erroring on comparing the checksums, but the checksum it generates is different on every request, so I don't think it is actually getting the tarball.

I0115 16:35:35.030871  220612 scanner.go:78] Reading plugin "razee" from /home/esatterwhite/.krew/index/logdna/plugins
I0115 16:35:35.031374  220612 util.go:56] parsed receipt for ctx: version=v0.9.1
I0115 16:35:35.035813  220612 util.go:56] parsed receipt for images: version=v0.3.2
I0115 16:35:35.036225  220612 util.go:56] parsed receipt for krew: version=v0.4.0
I0115 16:35:35.036459  220612 util.go:56] parsed receipt for ns: version=v0.9.1 
I0115 16:35:35.036775  220612 util.go:56] parsed receipt for oulogin: version=v0.0.3
I0115 16:35:35.037064  220612 util.go:56] parsed receipt for service-tree: version=v0.2.1
I0115 16:35:35.037388  220612 util.go:56] parsed receipt for status: version=v0.4.1
I0115 16:35:35.037667  220612 util.go:56] parsed receipt for tail: version=v0.15.0
I0115 16:35:35.037981  220612 util.go:56] parsed receipt for tree: version=v0.4.0
I0115 16:35:35.038411  220612 util.go:56] parsed receipt for view-utilization: version=v0.3.3
I0115 16:35:35.042009  220612 scanner.go:78] Reading plugin "razee" from /home/esatterwhite/.krew/index/logdna/plugins
I0115 16:35:35.042604  220612 install.go:148] Will install plugin: logdna/razee 
Installing plugin: razee                  
I0115 16:35:35.042624  220612 install.go:58] Looking for installed versions
I0115 16:35:35.042644  220612 platform.go:43] Matching platform for labels(arch=amd64,os=linux)
I0115 16:35:35.042667  220612 platform.go:51] Found matching platform with index (0)
I0115 16:35:35.042677  220612 install.go:77] Install plugin razee at version=v0.0.0
I0115 16:35:35.042683  220612 install.go:94] Creating download staging directory
I0115 16:35:35.042707  220612 install.go:99] Successfully created download staging directory "/tmp/krew-downloads497292283"
I0115 16:35:35.042724  220612 fetch.go:39] Fetching "https://github.com/answerbook/tooling-kubectl/releases/download/razee@0.0.0/razee-0.0.0.tgz"
I0115 16:35:35.301622  220612 downloader.go:42] Reading archive file into memory
I0115 16:35:35.403461  220612 downloader.go:47] Read 176808 bytes from archive into memory
I0115 16:35:35.403482  220612 verifier.go:51] Compare sha256 (5aad9180d4cac1acbe53fa1e609b141c521f70ac1b606f18254984a1a0cc59df) signed version
I0115 16:35:35.403510  220612 install.go:101] Deleting the download staging directory /tmp/krew-downloads497292283
W0115 16:35:35.403536  220612 install.go:164] failed to install plugin "razee": install failed: failed to unpack into staging dir: failed to unpack the plugin archive: checksum does not match, want: 5aad9180d4cac1acbe53fa1e609b141c521f70ac1b606f18254984a1a0cc59df, got 5f9d7bcff7b4ef23d3e25f947e1b81ef61eac343b8f8cb1ce
40b2230ab479bf0                                                   
F0115 16:35:35.403586  220612 root.go:77] checksum does not match, want: 5aad9180d4cac1acbe53fa1e609b141c521f70ac1b606f18254984a1a0cc59df, got 5f9d7bcff7b4ef23d3e25f947e1b81ef61eac343b8f8cb1ce40b2230ab479bf0
sigs.k8s.io/krew/internal/download.sha256Verifier.Verify               
        /home/runner/work/krew/krew/internal/download/verifier.go:55

and the next time

F0115 16:36:41.901396  221294 root.go:77] checksum does not match, want: 5aad9180d4cac1acbe53fa1e609b141c521f70ac1b606f18254984a1a0cc59df, got 5a7d6df2fe789c3d1e9287eacfaa52e87db70df38a034180cdc44986cf3d485b

Is this possible to do? Krew would have to make an authenticated request as far as I can tell.

[chriskim06]

No not right now. Krew doesn't support downloading plugins from private repos currently. I think this has come up before but it would require some design before we do any kind of implementation for it.

[corneliusweig]

@esatterwhite Are you using ssh key-based authentication mechanism? If so, I'm surprised this doesn't work.

[ahmetb]

Admittedly, this is something we missed during the design phase of custom indexes.

We handle retrieving the plugin index from a private repository (as ssh keys or remote helpers can help authenticate users). However, we only fetch the plugin bundle over https and without a way of authenticating.

Krew can still fetch the plugin bundle if it was at an unauthenticated (but firewalled endpoint) such as https://git.foo.corp.internal/[...], but it would not work against the traditional GitHub Release endpoints (because this endpoint doesn't authenticate using personal tokens or basic auth with username:password).

/kind bug
/kind design
/priority important-longterm

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[tsunamishaun]

@ahmetb after moving things around in our repo we eventually came to that conclusion 😞 . I'm actually glad someone else came across this, are there any plans to make private indexes/custom indexes a first class citizen? I was looking into krew-release-bot for custom indexes and it appears it is an open issue as well.

[ahmetb]

@tsunamishaun this is probably a v1.0 blocker for us. so we are eager to hear use cases as well as proposed designs if there are any. people have all sorts of methods to authenticate to private urls. what's yours?

[tsunamishaun]

@ahmetb We were planning to use github packages on private repos. I think most package repos (artifactory/github) would require user:token for pulling binaries with slightly varying requirements. Images storing binaries has had some appeal with those around me which would mean docker credentials. I should have looked but I don't think krew supports container type? Hopefully that was helpful, thanks for the quick response!

[ahmetb]

Yeah maybe we can consider supporting OCI layers in addition to https:// URLs and use the docker credentials detection logic docker already has. That would require us to do some work defining the format. I am doubting many other "private access" use cases actually will use a OCI Registry though.

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[millermatt]

/reopen

[k8s-ci-robot]

@millermatt: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[millermatt]

@ahmetb can this get reopened?

[ahmetb]

/reopen

[k8s-ci-robot]

@ahmetb: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tomelliot16]

Any movement on this? Has anyone started a PR yet?

[corneliusweig]

/cc @chriskim06

[chriskim06]

Any movement on this? Has anyone started a PR yet?

no there isn't a PR for this yet. @tomelliot16 what is your specific use case here? private github repos was a use case mentioned earlier in this thread and I'm guessing that would be a large portion of use cases. we need to figure out requirements and come up with a design for this feature before working on a PR

[MadhavJivrajani]

/remove-kind design
/kind feature
kind/design is migrated to kind/feature, see kubernetes/community#6144 for more details

[esatterwhite]

When might the feature design happen? we're closing in on a year.
Are you looking for a PR for that? or is this something the SIG wants to do?

[ahmetb]

@esatterwhite given the user base of the project, this request is only relevant to a very small portion of the users so we didn't prioritize solving this. Open to proposals.

[endzyme]

#816 is a possible solution that could support the features requested here. It's currently pretty rough in it's implementation but I'm looking for feedback on if this path would be viable for other commenters or if a more tailored solution would be better.

[esatterwhite]

@esatterwhite Are you using ssh key-based authentication mechanism? If so, I'm surprised this doesn't work.

The current fetcher tries to download the releases tarball over https. Unless there is a way to change that with config, I don't think that is possible. Not with out hard coding an access token in the url

[esatterwhite]

@esatterwhite given the user base of the project, this request is only relevant to a very small portion of the users so we didn't prioritize solving this. Open to proposals.

In all likely hood you need more than just a simple HTTP fetcher.
a git / github fetcher that uses tags, for example. That would allow ssh key authentication and you don't particularly have to know that it is private or not.

Luarocks (package manage for lua) actually does this fairly well where its fetch handler has many backends and uses the protocol in the manfist to pick the right one

https://github.com/luarocks/luarocks/tree/master/src/luarocks/fetch

[endzyme]

@esatterwhite given the user base of the project, this request is only relevant to a very small portion of the users so we didn't prioritize solving this. Open to proposals.

In all likely hood you need more than just a simple HTTP fetcher. a git / github fetcher that uses tags, for example. That would allow ssh key authentication and you don't particularly have to know that it is private or not.

Luarocks (package manage for lua) actually does this fairly well where it fetch has many backends and uses the protocol in the manfist to pick the right one

https://github.com/luarocks/luarocks/tree/master/src/luarocks/fetch

Cool! That's kind of the direction this is taking in #816 which would enable a set of fetchers. I'd love some feedback on if this would be a solution that could enable your use case.

[shaunofneuron]

@ahmetb recalling our early conversations around use-cases and looking at the proposal in #816 makes me think this would be an extensible way for users to explore and work within the constraints provided by their organizations. If anything this could inform additional more formal protocol support without changing the Plugin schema in the near term.

[daniel-garcia]

This issue also demonstrates a bug in krew not interpreting http 4xx status codes as errors for artifact downloads. #819

[vaibhav2107]

/remove-lifecycle rotten

[esatterwhite]

What else needs to happen for this to be implemented?