go get on https://go.googlesource.com gets rate-limited on Travis CI

[cotsog]

Hi,

I work for Travis CI and we have a customer that got rate-limited while doing a go get on https://go.googlesource.com and its build failed because of that.

Is there any way we could prevent this situation from happening?

Here is the logs he got:

go get -t ./...
# cd .; git clone https://go.googlesource.com/net /home/travis/gopath/src/golang.org/x/net
Cloning into '/home/travis/gopath/src/golang.org/x/net'...
remote: Short term bandwidth rate limit exceeded for IP 54.172.141.90.
fatal: protocol error: bad pack header
package github.com/Sirupsen/logrus
    imports github.com/davecgh/go-spew/spew
    imports github.com/fatih/color
    imports github.com/mattn/go-isatty
    imports github.com/shiena/ansicolor
    imports github.com/gorilla/websocket
    imports github.com/imdario/mergo
    imports github.com/mitchellh/mapstructure
    imports github.com/prometheus/client_golang/prometheus
    imports github.com/beorn7/perks/quantile
    imports github.com/golang/protobuf/proto
    imports github.com/prometheus/client_model/go
    imports github.com/prometheus/common/expfmt
    imports bitbucket.org/ww/goautoneg
    imports github.com/matttproud/golang_protobuf_extensions/pbutil
    imports github.com/prometheus/procfs
    imports github.com/ugorji/go/codec
    imports golang.org/x/net/context: exit status 128
# cd .; git clone https://go.googlesource.com/crypto /home/travis/gopath/src/golang.org/x/crypto
Cloning into '/home/travis/gopath/src/golang.org/x/crypto'...
remote: Short term bandwidth rate limit exceeded for IP 54.172.141.90.
fatal: protocol error: bad pack header
package github.com/Sirupsen/logrus
    imports github.com/davecgh/go-spew/spew
    imports github.com/fatih/color
    imports github.com/mattn/go-isatty
    imports github.com/shiena/ansicolor
    imports github.com/gorilla/websocket
    imports github.com/imdario/mergo
    imports github.com/mitchellh/mapstructure
    imports github.com/prometheus/client_golang/prometheus
    imports github.com/beorn7/perks/quantile
    imports github.com/golang/protobuf/proto
    imports github.com/prometheus/client_model/go
    imports github.com/prometheus/common/expfmt
    imports bitbucket.org/ww/goautoneg
    imports github.com/matttproud/golang_protobuf_extensions/pbutil
    imports github.com/prometheus/procfs
    imports github.com/ugorji/go/codec
    imports github.com/stretchr/testify/assert
    imports github.com/codegangsta/cli
    imports github.com/boltdb/bolt
    imports github.com/aws/aws-sdk-go/aws
    imports github.com/vaughan0/go-ini
    imports github.com/aws/aws-sdk-go/service/sqs
    imports github.com/garyburd/redigo/redis
    imports github.com/smartystreets/goconvey/convey
    imports github.com/jtolds/gls
    imports github.com/smartystreets/assertions
    imports github.com/abbot/go-http-auth
    imports golang.org/x/crypto/bcrypt: exit status 128
make: *** [deps-get] Error 1
K
The command "make deps-get" failed and exited with 2 during .

Thanks in advance!

[bradfitz]

If you authenticate as a user, you get a much higher quota. Use a ~/.gitcookies files. Click "Generate Password" at the top of https://go.googlesource.com/

We don't run the Gerrit server, though, so I'm unable to special case anything.

I'm closing this because it's not really a Go bug but feel free to mail me bradfitz at golang org if you have questions.

[cotsog]

@bradfitz: This is super helpful. Thank you!

[solarce]

We (Travis CI build infrastructure team) are also looking into getting in touch about some network level based authentication that could help reduce the work needed for individual users.

[mholt]

In the meantime, this can be worked around on a per-repo basis... (I did it on Travis but other CI systems may be affected and have similar solutions.)

• You have to go to go.googlesource.com and click "Generate Password"
• On that page, you'll be given a bash script to makes a ~/.gitcookies file. Copy that into your repo but do not commit it.
• Using the Travis client, encrypt the script file (instructions: https://docs.travis-ci.com/user/encrypting-files/)
• The Travis client will give you a command to decrypt the file. Add that, plus a line to run the script, to your .travis.yml file (for example)
• Commit the encrypted script and updated manifest to your repo.

This makes your git clones authenticated so your rate limits are higher.

Edit: Apparently on Travis, this doesn't work for pull requests :( So you have to surround the scripts with if [ "$TRAVIS_PULL_REQUEST" = "false" ]; then and ; fi and just hope you don't hit rate limits.

[dinoboff]

I don't think it will work on pull request, or a pull request could expose the password.