Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gerrit: quota issues accessing golang.org/x/sys #31826

Closed
myitcv opened this issue May 3, 2019 · 8 comments

Comments

Projects
None yet
7 participants
@myitcv
Copy link
Member

commented May 3, 2019

Raising as a result of https://groups.google.com/d/msg/golang-dev/LggZwiBTIvc/7kN0w_krAQAJ

I'm seeing the following in my Travis logs; not running a large number of builds here, at most 4 in parallel:

go: golang.org/x/sys@v0.0.0-20190429190828-d89cdac9e872: git fetch -f
origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in
/home/travis/gopath/pkg/mod/cache/vcs/76a8992ccba6d77c6bcf031ff2b6d821cf232e4ad8d1f2362404fbd0a798d846:
exit status 128:
remote: RESOURCE_EXHAUSTED Resource has been exhausted (e.g. check quota).
[type.googleapis.com/google.rpc.QuotaFailure]
violations {
 subject: "IP 104.154.113.151"
 description: "Short term bandwidth rate limit exceeded"
}
fatal: protocol error: bad pack header

cc @dborowitz @hanwen @tparker (per @bradfitz)

@dmitshur

This comment has been minimized.

Copy link
Member

commented May 3, 2019

@myitcv Can you please provide more information about the frequency of that error? Have you seen it occur just once so far, or is it happening more often than that today?

@myitcv

This comment has been minimized.

Copy link
Member Author

commented May 3, 2019

Started seeing it ~1 hour ago. I'm attempting ~4 parallel builds every ~10 mins trying to fix a build failure. I've seen the failure ~5 times in total.

If it's relevant, there also appear to be some connectivity issues from Travis to GitHub; I'm seeing timeouts accessing https://raw.githubusercontent.com/ URLs (suspect however this is very much unrelated, but I report it here for completeness)

@tparker

This comment has been minimized.

Copy link

commented May 3, 2019

This was a quota denial. It looks like you generated some bursty activity. Quota buckets refill every 5 seconds, so an exponential backoff retry strategy should keep builds making steady progress.

This looks like unauthenticated access, which has a smaller quota than authenticated access. Can you say more about the environment in which these builds are running?

@bradfitz

This comment has been minimized.

Copy link
Member

commented May 3, 2019

Can you say more about the environment in which these builds are running?

At the top he mentioned Travis. Travis is https://travis-ci.org/. It's a popular TAP/Sponge-ish thing for non-Googlers.

@tparker

This comment has been minimized.

Copy link

commented May 3, 2019

We had special configuration for Travis IP addresses to give more quota, but our list was out-of-date. We just updated it so your Travis machines should have more quota going forward.

@jrn

This comment has been minimized.

Copy link

commented May 4, 2019

https://docs.travis-ci.com/user/encrypting-files/ allows passing in a credential without exposing it to the public. But then your build has access to that credential, which likely defeats the purpose.

Sorry for the trouble. The quota increase @tparker mentioned grants each Travis instance the same quota as ordinary authenticated users get; if that's not sufficient, we can increase it more.

@odeke-em odeke-em changed the title gerrit: quota issues accessing golang.org/x/sys x/build/gerrit: quota issues accessing golang.org/x/sys May 4, 2019

@gopherbot gopherbot added this to the Unreleased milestone May 4, 2019

@gopherbot gopherbot added the Builders label May 4, 2019

@hanwen hanwen removed their assignment May 4, 2019

@dmitshur

This comment has been minimized.

Copy link
Member

commented May 4, 2019

Based on #31826 (comment), this should be resolved now. I'll close this, but if the issue persists, please feel free to re-open and provide more information so we can investigate further.

Thank you @myitcv for reporting this and @tparker and others for resolving this.

@dmitshur dmitshur closed this May 4, 2019

@dmitshur dmitshur changed the title x/build/gerrit: quota issues accessing golang.org/x/sys gerrit: quota issues accessing golang.org/x/sys May 4, 2019

@dmitshur dmitshur removed the Builders label May 4, 2019

@myitcv

This comment has been minimized.

Copy link
Member Author

commented May 4, 2019

Thanks all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.