Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Super crawler #101
Upgrade the crawler to use the "games" API resource, which can return 100 games at a time.
The API reports that there are 49,361 games currently on Giant Bomb, so we would be able to crawl the entire database in 494 requests. At a casual 1 request every 5 minutes this would take two days, which is a fantastic improvement over the current speed of database growth.