New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Super crawler #101

Closed
Clidus opened this Issue Apr 29, 2016 · 1 comment

Comments

Projects
None yet
1 participant
@Clidus
Copy link
Owner

Clidus commented Apr 29, 2016

Upgrade the crawler to use the "games" API resource, which can return 100 games at a time.

The API reports that there are 49,361 games currently on Giant Bomb, so we would be able to crawl the entire database in 494 requests. At a casual 1 request every 5 minutes this would take two days, which is a fantastic improvement over the current speed of database growth.

@Clidus Clidus added this to the Version 0.4.5 milestone Apr 29, 2016

@Clidus Clidus self-assigned this Apr 29, 2016

@Clidus Clidus added the enhancement label Apr 29, 2016

Clidus added a commit that referenced this issue May 2, 2016

Clidus added a commit that referenced this issue May 2, 2016

Clidus added a commit that referenced this issue May 2, 2016

Clidus added a commit that referenced this issue May 3, 2016

@Clidus

This comment has been minimized.

@Clidus Clidus closed this May 3, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment