Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Super crawler #101

Closed
Clidus opened this issue Apr 29, 2016 · 1 comment
Closed

Super crawler #101

Clidus opened this issue Apr 29, 2016 · 1 comment
Assignees
@Clidus
Labels
enhancement
Milestone
Version 0.4.5

Comments

@Clidus
Copy link
Owner

Clidus commented Apr 29, 2016

Upgrade the crawler to use the "games" API resource, which can return 100 games at a time.

The API reports that there are 49,361 games currently on Giant Bomb, so we would be able to crawl the entire database in 494 requests. At a casual 1 request every 5 minutes this would take two days, which is a fantastic improvement over the current speed of database growth.
@Clidus Clidus added this to the Version 0.4.5 milestone Apr 29, 2016
@Clidus Clidus self-assigned this Apr 29, 2016
Clidus added a commit that referenced this issue May 2, 2016
@Clidus
Create settings table to store crawler offset value #101
32913f6
Clidus added a commit that referenced this issue May 2, 2016
@Clidus
Capatalise database fieldname #101
dd2aa6e
Clidus added a commit that referenced this issue May 2, 2016
@Clidus
Use games resource to get 100 games at a time to be processed #101
64d2885
Clidus added a commit that referenced this issue May 2, 2016
@Clidus
Allow games API to be logged #101
46bd0f0
Clidus added a commit that referenced this issue May 2, 2016
@Clidus
Remove debug value #101
f887917
Clidus added a commit that referenced this issue May 3, 2016
@Clidus
API log result should allow nulls. #101
5998e97
@Clidus
Copy link
Owner Author

Clidus commented May 3, 2016

Documentation: https://github.com/Clidus/gwl/wiki/Giant-Bomb-database-cache-and-updater

@Clidus Clidus closed this as completed May 3, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Clidus Clidus

Labels
enhancement
Projects
None yet
Milestone
Version 0.4.5
Development

No branches or pull requests
1 participant
@Clidus