ISO download step fails due to GitHub API rate limit #481

Open
srid opened this Issue Aug 12, 2014 · 16 comments

Comments

Projects
None yet
9 participants
@srid

srid commented Aug 12, 2014

using the latest release of boot2docker:

$ boot2docker init -v
2014/08/12 12:46:30 executing: VBoxManage showvminfo boot2docker-vm --machinereadable
2014/08/12 12:46:30 Downloading boot2docker ISO image...
2014/08/12 12:46:31 Failed to get latest release: json: cannot unmarshal object into Go value of type []struct { TagName string "json:\"tag_name\"" }

Workaround

run

$ boot2docker --iso-url=https://github.com/boot2docker/boot2docker/releases/download/v1.3.2/boot2docker.iso download
$ boot2docker init
@SvenDowideit

This comment has been minimized.

Show comment
Hide comment
@SvenDowideit

SvenDowideit Aug 13, 2014

Contributor

@huselage - is this that OSX beta thing you were talking about?

Contributor

SvenDowideit commented Aug 13, 2014

@huselage - is this that OSX beta thing you were talking about?

@srid

This comment has been minimized.

Show comment
Hide comment
@srid

srid Aug 13, 2014

Yup it's the developer preview build.

I was wondering if the logging can be made verbose so as to directly see the source of the problem.

srid commented Aug 13, 2014

Yup it's the developer preview build.

I was wondering if the logging can be made verbose so as to directly see the source of the problem.

@srid

This comment has been minimized.

Show comment
Hide comment
@srid

srid Aug 13, 2014

nothing to do with OS X, beta or not.

accessing the failing URL https://api.github.com/repos/boot2docker/boot2docker/releases in my browser, i see:

{
  "message": "API rate limit exceeded for [IP ADDR SNIPPED]. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)",
  "documentation_url": "https://developer.github.com/v3/#rate-limiting"
}

srid commented Aug 13, 2014

nothing to do with OS X, beta or not.

accessing the failing URL https://api.github.com/repos/boot2docker/boot2docker/releases in my browser, i see:

{
  "message": "API rate limit exceeded for [IP ADDR SNIPPED]. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)",
  "documentation_url": "https://developer.github.com/v3/#rate-limiting"
}

@srid srid changed the title from ISO download step fails on OS X 10.10 to ISO download step fails due to GitHub API rate limit Aug 13, 2014

@SvenDowideit

This comment has been minimized.

Show comment
Hide comment
@SvenDowideit

SvenDowideit Aug 14, 2014

Contributor

oh wow :/ sounds like its time to move away from github :(

Contributor

SvenDowideit commented Aug 14, 2014

oh wow :/ sounds like its time to move away from github :(

@tianon

This comment has been minimized.

Show comment
Hide comment
@tianon

tianon Aug 14, 2014

Contributor

Couldn't we do this same thing with the equivalent of "git ls-remote"
directly instead?

Contributor

tianon commented Aug 14, 2014

Couldn't we do this same thing with the equivalent of "git ls-remote"
directly instead?

@SvenDowideit

This comment has been minimized.

Show comment
Hide comment
@SvenDowideit

SvenDowideit Aug 14, 2014

Contributor

that would work - I was thinking more along the lines of pushing the b2d release artifacts onto get.docker.io.

Contributor

SvenDowideit commented Aug 14, 2014

that would work - I was thinking more along the lines of pushing the b2d release artifacts onto get.docker.io.

@SvenDowideit

This comment has been minimized.

Show comment
Hide comment
@SvenDowideit

SvenDowideit Aug 15, 2014

Contributor

Mind you - I wonder why @srid 's IP is hitting the GitHub API more than 60 times an hour.

Sounds like the first fix is to improve our error message.

Contributor

SvenDowideit commented Aug 15, 2014

Mind you - I wonder why @srid 's IP is hitting the GitHub API more than 60 times an hour.

Sounds like the first fix is to improve our error message.

@rtoma

This comment has been minimized.

Show comment
Hide comment
@rtoma

rtoma Sep 11, 2014

Same issue here.

Downloading boot2docker ISO image...
error in run: Failed to get latest release: Error getting releases: API rate limit exceeded for X.X.X.X.
(But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation
 for more details.)
 see
Client version: v1.2.0
Git commit: a551732

My workaround:

voila.

rtoma commented Sep 11, 2014

Same issue here.

Downloading boot2docker ISO image...
error in run: Failed to get latest release: Error getting releases: API rate limit exceeded for X.X.X.X.
(But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation
 for more details.)
 see
Client version: v1.2.0
Git commit: a551732

My workaround:

voila.

@tianon

This comment has been minimized.

Show comment
Hide comment
@tianon

tianon Sep 13, 2014

Contributor

So what I'm essentially talking about here is using something like: git ls-remote --tags https://github.com/boot2docker/boot2docker.git | cut -d$'\t' -f2 | cut -d'v' -f2 | sort -V | tail -1 (only implemented nicely and cleanly in Go, of course).

Contributor

tianon commented Sep 13, 2014

So what I'm essentially talking about here is using something like: git ls-remote --tags https://github.com/boot2docker/boot2docker.git | cut -d$'\t' -f2 | cut -d'v' -f2 | sort -V | tail -1 (only implemented nicely and cleanly in Go, of course).

@tianon

This comment has been minimized.

Show comment
Hide comment
@tianon

tianon Sep 14, 2014

Contributor

An alternative that might be simpler than implementing the Git protocol itself would be to HEAD request https://github.com/boot2docker/boot2docker/releases/latest, then steal the redirect header it generates and grab everything after .../tag/ (or just the last path bit, which should be pretty easy to do with some Go standard library function).

Contributor

tianon commented Sep 14, 2014

An alternative that might be simpler than implementing the Git protocol itself would be to HEAD request https://github.com/boot2docker/boot2docker/releases/latest, then steal the redirect header it generates and grab everything after .../tag/ (or just the last path bit, which should be pretty easy to do with some Go standard library function).

@tianon

This comment has been minimized.

Show comment
Hide comment
@tianon

tianon Sep 14, 2014

Contributor

If we wanted a really error-prone solution, we could also just grab https://raw.githubusercontent.com/boot2docker/boot2docker/master/VERSION, but that's a pretty hacky solution IMO, especially since VERSION being bumped in master doesn't necessarily mean a new release is actually deployed.

Contributor

tianon commented Sep 14, 2014

If we wanted a really error-prone solution, we could also just grab https://raw.githubusercontent.com/boot2docker/boot2docker/master/VERSION, but that's a pretty hacky solution IMO, especially since VERSION being bumped in master doesn't necessarily mean a new release is actually deployed.

@jbayer

This comment has been minimized.

Show comment
Hide comment
@jbayer

jbayer Jan 2, 2015

confirmed work-around for this same issue in boot2docker 1.4.1 is:

$ boot2docker --iso-url=https://github.com/boot2docker/boot2docker/releases/download/v1.4.1/boot2docker.iso download
Downloading boot2docker ISO image...
Success: downloaded https://github.com/boot2docker/boot2docker/releases/download/v1.4.1/boot2docker.iso
    to /Users/jamesbayer/.boot2docker/boot2docker.iso
$ boot2docker init
$ boot2docker start

jbayer commented Jan 2, 2015

confirmed work-around for this same issue in boot2docker 1.4.1 is:

$ boot2docker --iso-url=https://github.com/boot2docker/boot2docker/releases/download/v1.4.1/boot2docker.iso download
Downloading boot2docker ISO image...
Success: downloaded https://github.com/boot2docker/boot2docker/releases/download/v1.4.1/boot2docker.iso
    to /Users/jamesbayer/.boot2docker/boot2docker.iso
$ boot2docker init
$ boot2docker start
@silarsis

This comment has been minimized.

Show comment
Hide comment
@silarsis

silarsis Feb 2, 2015

@SvenDowideit we just hit the same problem - because we're running in a large organisation behind NAT, the github API almost always gives errors due to arbitrary other people behind the NAT hitting it.

Either not using the API, or falling back to some other method cleanly in case of this error, would be really useful. Alternately, print out some instructions if this happens so people at least know what to do.

Or yeah, host the iso somewhere that's not github ;)

silarsis commented Feb 2, 2015

@SvenDowideit we just hit the same problem - because we're running in a large organisation behind NAT, the github API almost always gives errors due to arbitrary other people behind the NAT hitting it.

Either not using the API, or falling back to some other method cleanly in case of this error, would be really useful. Alternately, print out some instructions if this happens so people at least know what to do.

Or yeah, host the iso somewhere that's not github ;)

@docwhat

This comment has been minimized.

Show comment
Hide comment
@docwhat

docwhat Apr 27, 2015

You could also just put a .json file on the ghpages that contains all the boot2docker ISO releases. If you auto-generate it when you do a release, it should be easy to keep up-to-date.

docwhat commented Apr 27, 2015

You could also just put a .json file on the ghpages that contains all the boot2docker ISO releases. If you auto-generate it when you do a release, it should be easy to keep up-to-date.

@cuttlefish

This comment has been minimized.

Show comment
Hide comment
@cuttlefish

cuttlefish Apr 27, 2015

homebrew supports setting your own API token via environment variable. That would work without changing the discovery mechanism.

homebrew supports setting your own API token via environment variable. That would work without changing the discovery mechanism.

@rtoma

This comment has been minimized.

Show comment
Hide comment
@rtoma

rtoma Aug 21, 2015

+1 for the API token mechanism like Homebrew has implemented.

rtoma commented Aug 21, 2015

+1 for the API token mechanism like Homebrew has implemented.

@wglambert wglambert added the Issue label Jul 9, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment