New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove dependancy on Giant Bomb API #57

Closed
Clidus opened this Issue Apr 4, 2015 · 8 comments

Comments

Projects
None yet
1 participant
@Clidus
Copy link
Owner

Clidus commented Apr 4, 2015

Waiting for API responses make the site slow and unresponsive.

Switch to scraping the API and only using GB search on the site.

@Clidus

This comment has been minimized.

Copy link
Owner

Clidus commented Jul 31, 2015

http://www.giantbomb.com/forums/api-developers-3017/crazy-api-usage-and-ip-bans-1775692/#2

There's been a bunch of scrapers ripping Giant Bomb and hosting that ripped content at their domains and serving their own ads and sometimes malware and messing with our SEO. As of now all scrapers that are not coming from known addresses of legitimate search engines (I have a detailed list, I know who is valid and who is not) will be quickly IP banned.

If this is from aggressive API usage, as long as you have logged into the site from the same address you use the API from you were not banned and you received a message from me asking for more information about what you're doing. If you're doing something cool then you'll be perfectly fine with crazy usage and I may even offer help.

If there is no record of a login from an address that's hitting us with A LOT of requests then that address is now IP banned. If you're legitimate and got caught up in that, I apologize. Clearly you're here on another IP address, so send me a message explaining what you're doing and tell me the IP address you're coming from.

We may have to require that you login from the same IP address you're using the API from.

There's a few exceptions that I know about and didn't get caught up, Kodi (XMBC), Roku etc. for example. To have triggered the ban you would have had to have done 2 - 3x the requests that Google does. Yeah, that crazy.

@Clidus

This comment has been minimized.

Copy link
Owner

Clidus commented Jul 31, 2015

Scrape it... nicely?

@Clidus Clidus self-assigned this Oct 24, 2015

@Clidus Clidus changed the title Scrape Giant Bomb API Remove dependancy on Giant Bomb API Oct 24, 2015

@Clidus Clidus added this to the Version 0.4.1 milestone Oct 24, 2015

@Clidus

This comment has been minimized.

Copy link
Owner

Clidus commented Oct 24, 2015

Game pages should load directly from GWL database, and call GB API only if the game hasn't be cached yet.

@Clidus

This comment has been minimized.

Copy link
Owner

Clidus commented Oct 24, 2015

Chron job needs to crawl GB API for new games to grow the database.

@Clidus

This comment has been minimized.

Copy link
Owner

Clidus commented Oct 25, 2015

Need to record all platforms a game is on, not just the ones added by a user.

Clidus added a commit that referenced this issue Oct 25, 2015

Clidus added a commit that referenced this issue Oct 25, 2015

Clidus added a commit that referenced this issue Oct 25, 2015

@Clidus

This comment has been minimized.

Copy link
Owner

Clidus commented Oct 25, 2015

Update 0.4.1 is live. Crawler is now recording platforms a game is on when it updates a game's cache. Once this has crawled all the games currently in the db, the game page can be updated to read entirely from the database (when the game exists in db).

@Clidus Clidus referenced this issue Oct 25, 2015

Merged

Version 0.4.1 #84

Clidus added a commit that referenced this issue Oct 31, 2015

This was referenced Oct 31, 2015

@Clidus

This comment has been minimized.

Copy link
Owner

Clidus commented Feb 18, 2016

getGameByGBID needs to check the database for a game. If it cant find it, it should make a request to Giant Bomb, save the response, then load the game page from the database.

Clidus added a commit that referenced this issue Feb 18, 2016

Load game page from database #57
Many features currently broken

Clidus added a commit that referenced this issue Feb 18, 2016

Clidus added a commit that referenced this issue Feb 18, 2016

Clidus added a commit that referenced this issue Apr 2, 2016

Clidus added a commit that referenced this issue Apr 3, 2016

Clidus added a commit that referenced this issue Apr 3, 2016

Clidus added a commit that referenced this issue Apr 3, 2016

Clidus added a commit that referenced this issue Apr 3, 2016

Clidus added a commit that referenced this issue Apr 3, 2016

Clidus added a commit that referenced this issue Apr 3, 2016

Clidus added a commit that referenced this issue Apr 3, 2016

Clidus added a commit that referenced this issue Apr 3, 2016

Clidus added a commit that referenced this issue Apr 3, 2016

Clidus added a commit that referenced this issue Apr 3, 2016

Clidus added a commit that referenced this issue Apr 4, 2016

Clidus added a commit that referenced this issue Apr 4, 2016

Fix adding game collection #57
(picking default platform is currently broken)

Clidus added a commit that referenced this issue Apr 4, 2016

Clidus added a commit that referenced this issue Apr 16, 2016

Clidus added a commit that referenced this issue Apr 16, 2016

Clidus added a commit that referenced this issue Apr 16, 2016

Clidus added a commit that referenced this issue Apr 16, 2016

Clidus added a commit that referenced this issue Apr 17, 2016

Clidus added a commit that referenced this issue Apr 17, 2016

Clidus added a commit that referenced this issue Apr 17, 2016

Clidus added a commit that referenced this issue Apr 17, 2016

Clidus added a commit that referenced this issue Apr 17, 2016

Clidus added a commit that referenced this issue Apr 17, 2016

@Clidus

This comment has been minimized.

Copy link
Owner

Clidus commented Apr 17, 2016

This is done and merged into dev branch. Game pages now default to loading from the database, and fall back to the GB API if the game can't be found. I did significant restructuring at the same time to split out a lot of the functionality in the Game model into a Collections model. It makes a lot more sense now.

@Clidus Clidus closed this Apr 17, 2016

@Clidus Clidus referenced this issue Apr 28, 2016

Merged

Version 0.4.4 #100

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment