Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Internal server error due to scraping speed #194

Closed
NeonLightning opened this issue Apr 1, 2021 · 3 comments
Closed

[BUG] Internal server error due to scraping speed #194

NeonLightning opened this issue Apr 1, 2021 · 3 comments
Labels

Comments

@NeonLightning
Copy link

NeonLightning commented Apr 1, 2021

when doing a scrape. after about 150-200 it gets an internal server error. if i wait for a couple minutes and try again it works for another bunch
but if i start again right away it'll error right away.

@lainverse
Copy link

I have the same issue, but now it stops after ~25 scrapes and then goes VERY slow.
Scraping speed is fine for small library updates, but first run on a large library is awful.

@Tythesly
Copy link

Tythesly commented Jun 15, 2021

@NeonLightning @lainverse I have found that just having the program Fiddler (either the Everywhere or Classic version) running will prevent it from producing errors. It scraped over 3k 5.1k games without a single error when I did it.

@mvegter mvegter pinned this issue Jun 22, 2021
@mvegter mvegter changed the title [BUG] scrapes too quickly and doesn't handle errors well [BUG] Internal server error due to scraping speed Jun 22, 2021
@mvegter
Copy link
Member

mvegter commented Jun 22, 2021

This is part of the Depressurizer (Premium) API, for the normal Depressurizer using the Steam Store page scraper you will have to manage the batch size yourself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

No branches or pull requests

4 participants