Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/vgo: avoid github rate limits? #23955

Closed
flibustenet opened this issue Feb 20, 2018 · 23 comments

Comments

@flibustenet
Copy link

commented Feb 20, 2018

What version of Go are you using (go version)?

go version go1.10 linux/amd64 vgo:2018-02-20.1

What did you do?

vgo build with more than 6 dependencies

What did you see instead?

vgo: 403 response from api.github.com

GitHub applies fairly small rate limits to unauthenticated users, and
you appear to be hitting them....

solution

create a token and put it in .netrc

it can be annoying for new user.

@gopherbot gopherbot added this to the Unreleased milestone Feb 20, 2018

@rsc

This comment has been minimized.

Copy link
Contributor

commented Feb 20, 2018

Now that the tool is public, I will try to find out from GitHub if there's anything we can do about this. I don't really understand how things like OAuth2 apply to command-line binaries that ship to users and therefore can't hold their own secrets. I agree it's annoying.

@rsc rsc changed the title x/vgo github rate limits to unauthenticated users x/vgo: avoid github rate limits? github rate limits to unauthenticated users Feb 20, 2018

@rsc rsc changed the title x/vgo: avoid github rate limits? github rate limits to unauthenticated users x/vgo: avoid github rate limits? Feb 20, 2018

@rsc rsc modified the milestones: Unreleased, vgo Feb 20, 2018

@andybons

This comment has been minimized.

Copy link
Member

commented Feb 20, 2018

Setting an env var with a personal access token will likely be their response: https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/

@flibustenet

This comment has been minimized.

Copy link
Author

commented Feb 20, 2018

I believe it should not be mandatory to have a github account at all to use vgo.

@tianon

This comment has been minimized.

Copy link
Contributor

commented Feb 21, 2018

For just getting a tag list, implementing git ls-remote --tags ... should be sufficient (and not subject to the GitHub API rate limits). That won't work for arbitrary commits, but those don't necessarily need to be metadata'd every time -- only on fetch when a version isn't already supplied AFAICT (since the .zip archive download works with a raw commit hash directly).

@ericlagergren

This comment has been minimized.

Copy link
Contributor

commented Feb 21, 2018

How does homebrew do it? Their entire model is pulling from GitHub, right?

@josharian

This comment has been minimized.

Copy link
Contributor

commented Feb 21, 2018

brew gives you instructions for getting and using a github app token when you hit rate limits.

@freman

This comment has been minimized.

Copy link

commented Feb 21, 2018

I've been using githubs graphql api to avoid ratelimits while checking repo freshness for my caching git proxy...

It's not universal across all hosts, but it works on github :D

@franciscocpg

This comment has been minimized.

Copy link

commented Feb 21, 2018

I believe it should not be mandatory to have a github account at all to use vgo.

Totally agree. Unless you have private repositories in github, there should be no reason to have a github account to use vgo.

I've been using githubs graphql api to avoid ratelimits

The limits are different. But they are still there.

@freman

This comment has been minimized.

Copy link

commented Feb 21, 2018

The limits are different. But they are still there.

True enough, but using graphql I can query every single repo we cache in one request for one API call.

Just putting it out there :)

@drewblas

This comment has been minimized.

Copy link

commented Feb 21, 2018

Many other tools seem to be able to use github-hosted files and repos for dependency fetching without hitting the rate limits. I've personally never experienced a GH rate limit error, so it's surely possible. But now I've hit one with my very first vgo command. So either vgo is doing something very inefficient or its design is such that it requires far too many calls. Hopefully it's fixable!

@terminatingcode

This comment has been minimized.

Copy link

commented Feb 23, 2018

Just FYI the 403 resulted in a panic during a downgrade call. I could not reproduce it a second time:

vgo get rsc.io/sampler@v1.2.0
vgo: finding rsc.io/sampler v1.2.0
vgo: finding rsc.io/quote v1.5.1
vgo: finding rsc.io/quote v1.5.0
vgo: finding rsc.io/quote v1.4.0
vgo: 403 response from api.github.com

GitHub applies fairly small rate limits to unauthenticated users, and
you appear to be hitting them. To authenticate, please visit
https://github.com/settings/tokens and click "Generate New Token" to
create a Personal Access Token. The token only needs "public_repo"
scope, but you can add "repo" if you want to access private
repositories too.

Add the token to your $HOME/.netrc:

    machine api.github.com login YOU password TOKEN

Sorry for the interruption.
vgo: rsc.io/quote v1.4.0: unexpected status (https://api.github.com/repos/rsc/quote/git/refs/tags/v1.4.0): 403 Forbidden
panic: unexpected status (https://api.github.com/repos/rsc/quote/git/refs/tags/v1.4.0): 403 Forbidden

goroutine 1 [running]:
golang.org/x/vgo/vendor/cmd/go/internal/mvs.Downgrade.func2(0xc4201e4213, 0xc, 0xc4201d6c50, 0x6)
	/Users/sarahconnor/go/src/golang.org/x/vgo/vendor/cmd/go/internal/mvs/mvs.go:256 +0x662
golang.org/x/vgo/vendor/cmd/go/internal/mvs.Downgrade(0xc4201f6151, 0x20, 0x0, 0x0, 0x1597960, 0xc420367860, 0xc420367840, 0x1, 0x1, 0xc4200b6000, ...)
	/Users/sarahconnor/go/src/golang.org/x/vgo/vendor/cmd/go/internal/mvs/mvs.go:281 +0x66b
golang.org/x/vgo/vendor/cmd/go/internal/vgo.runGet(0x17b3a60, 0xc42001e0e0, 0x1, 0x1)
	/Users/sarahconnor/go/src/golang.org/x/vgo/vendor/cmd/go/internal/vgo/get.go:121 +0xd0e
golang.org/x/vgo/vendor/cmd/go.Main()
	/Users/sarahconnor/go/src/golang.org/x/vgo/vendor/cmd/go/main.go:155 +0x82e
main.main()
	/Users/sarahconnor/go/src/golang.org/x/vgo/main.go:37 +0x25
@robertlestak

This comment has been minimized.

Copy link

commented Feb 23, 2018

There definitely needs to be a better long term solution that doesn't include requiring users to create a GitHub token just to build their code.

As @drewblas mentioned, this is the first time I've ever hit a GitHub access limit, and it was on my second vgo build. Nonetheless, I created and set an access token to continue testing with vgo.

However when I went to build the application in a Docker container, I ran into the same issue within the container.

So at this point, not only would using vgo require users to create and configure a GH access token for their local machines, but they would also have to modify all of their existing Dockerfiles to copy their local GH token into the container, build the code with vgo using the copied access token, then copy the output binary to a secondary container via a multi-stage build so as to not have their access token in the final container - that throws a real wrench in the workflow.

While doable, it's certainly not scalable.

On the positive side of the access token - vgo builds projects with private dependencies without issue, so that's nice. But I don't think that benefit outweighs the cost of requiring every person who ever wants to build a Go project to create a GitHub account.

@myitcv

This comment has been minimized.

Copy link
Member

commented Feb 25, 2018

Not sure whether it's worth filing this here or in an separate issue; but one change introduced here by vgo is that the git config insteadof:

url.git@github.com:.insteadof=https://github.com/

is now effectively ignored (because explicit HTTP calls are used as opposed to delegating to the git command?)

This was previously my preferred approach to avoiding hitting rate limits. Not saying this is a preference per se, just adding a data point to this conversation.

@bpicode bpicode referenced this issue Feb 25, 2018
douglarek added a commit to douglarek/gist that referenced this issue Feb 28, 2018
Fix vgo travis-ci build
It is associated with golang/go#23955.
@doxxx

This comment has been minimized.

Copy link

commented Feb 28, 2018

I'm not having any luck with the netrc workaround. I'm on Windows, so I've defined a HOME env var pointing to my user directory, and in that directory I have a _netrc file (so-named "because Windows" but I also tried .netrc) and it contains a single line: machine api.github.com login doxxx password <personal access token>

Any idea what I'm doing wrong?

@docwhat

This comment has been minimized.

Copy link

commented Mar 2, 2018

The ~/.netrc entry should be:

# You can put these on one line if you want, I just prefer them like this.
machine api.github.com
    login sekret-oauth-token
    password x-oauth-basic

My token (which I use for other things and already had in my ~/.netrc) has these permissions:

  • read:gpg_key
  • read:org
  • read:public_key
  • repo
  • user:email

I suspect that no permissions are needed, since it is only reading information.

This is the page to manage your OAuth tokens.

@henjue

This comment has been minimized.

Copy link

commented Mar 3, 2018

GitHub applies fairly small rate limits to unauthenticated users, and
you appear to be hitting them. To authenticate, please visit
https://github.com/settings/tokens and click "Generate New Token" to
create a Personal Access Token. The token only needs "public_repo"
scope, but you can add "repo" if you want to access private
repositories too.

Add the token to your $HOME/.netrc:

    machine api.github.com login YOU password TOKEN

Sorry for the interruption.
vgo: import "github.com/henjue/note_api" ->
        import "github.com/henjue/note_api/api" ->
        import "github.com/henjue/note_api/sessions" ->
        import "github.com/kataras/iris/sessions/sessiondb/redis/service": unexpected status (https://api.github.com/repos/kataras/iris): 403 Forbidden

github.com/kataras/iris/sessions/sessiondb/redis/service is Multilevel directory

@doxxx

This comment has been minimized.

Copy link

commented Mar 10, 2018

@docwhat I tried that as well -- didn't work. Does the netrc method work on Windows?

@docwhat

This comment has been minimized.

Copy link

commented Mar 11, 2018

@doxxx I’m not sure. I think the file name is different (_netrc) and you’d have to set $HOME.

I would have to search the source to be sure.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Mar 30, 2018

The netrc is $HOME/.netrc on Windows too. That's a bug. If someone can tell me what the canonical netrc location is on Windows, we can fix that. It really should be working.

@doxxx

This comment has been minimized.

Copy link

commented Mar 30, 2018

Anybody else having trouble with this on Windows, I got it working. See #24606 (comment).

@NobbZ

This comment has been minimized.

Copy link

commented Apr 12, 2018

This hinders us to use vgo in our company, as it requires to put the access token in a semi public place in the CI service we use.

It would be really nice if vgo would be able to work without doing this API calls, but just by clonig the repository and then using git locally.

I mean, it has to work for self hosted git repositories as well, doesn't it?

@gopherbot

This comment has been minimized.

Copy link

commented Apr 18, 2018

Change https://golang.org/cl/107657 mentions this issue: cmd/go/internal/modfetch: update for new gitrepo backend

@gopherbot

This comment has been minimized.

Copy link

commented Apr 18, 2018

Change https://golang.org/cl/107656 mentions this issue: cmd/vgo/internal/modfetch/gitrepo: add general git repo access

gopherbot pushed a commit to golang/vgo that referenced this issue Apr 25, 2018
cmd/vgo/internal/modfetch/gitrepo: add general git repo access
A goal of introducing modules was to move away from invoking
version control tools directly, but it has become clear that we're not
ready to do that today. GitHub in particular imposes draconian limits
on HTTPS API access that they don't impose on ordinary Git access.
And we can avoid for now breaking company setups using private
Git servers.

Because GitHub Enterprise now serves ?go-get=1 queries in a way
that is compatible with old go get, dropping back to version control
tools makes that compatible with vgo too.

The next CL hooks this code into the rest of vgo.

For golang/go#24915.
For golang/go#23955.
For golang/go#24076.

Change-Id: I76bea30081047ab68286a5d095a0d55872c5a1a3
Reviewed-on: https://go-review.googlesource.com/107656
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>

@golang golang locked and limited conversation to collaborators Apr 25, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
You can’t perform that action at this time.