Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Web scrape not fetching #258

Open
Pixieismyname opened this issue Nov 10, 2023 · 2 comments
Open

[BUG] Web scrape not fetching #258

Pixieismyname opened this issue Nov 10, 2023 · 2 comments

Comments

@Pixieismyname
Copy link

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

  1. Go to 'Database Editor'
  2. Click on 'Scrape unscraped'
  3. Only scrapes info on a small number of titles
  4. Repeat step 2, Scrapes some more titles that were not scraped, repeat with diminishing returns

Expected behavior
Scrape to find all available info

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: Win 11
  • Version v5.3.0
@Pixieismyname
Copy link
Author

06/12/2023 09:36:38 Info | Database: Saved database to 'database.json', in 42ms.
06/12/2023 09:40:04 Warn | Scraping 204080: Location header points to the Steam Store homepage, aborting scraping.
06/12/2023 09:40:04 Info | CancelableDlg:Scraping data... | Thread completed, still running 20.
06/12/2023 09:40:04 Warn | Scraping 33910: Location header points to the Steam Store homepage, aborting scraping.
06/12/2023 09:40:04 Info | CancelableDlg:Scraping data... | Thread completed, still running 19.
06/12/2023 09:40:04 Warn | Scraping 202090: Location header points to the Steam Store homepage, aborting scraping.
06/12/2023 09:40:04 Info | CancelableDlg:Scraping data... | Thread completed, still running 18.
06/12/2023 09:40:04 Warn | Scraping 251670: Location header points to the Steam Store homepage, aborting scraping.
06/12/2023 09:40:04 Info | CancelableDlg:Scraping data... | Thread completed, still running 17.
06/12/2023 09:40:04 Warn | Scraping 28050: Location header points to the Steam Store homepage, aborting scraping.
06/12/2023 09:40:04 Info | CancelableDlg:Scraping data... | Thread completed, still running 16.
06/12/2023 09:40:04 Warn | Scraping 480: Location header points to the Steam Store homepage, aborting scraping.
06/12/2023 09:40:04 Info | CancelableDlg:Scraping data... | Thread completed, still running 15.
06/12/2023 09:40:05 Warn | Scraping 200110: Location header points to the Steam Store homepage, aborting scraping.
06/12/2023 09:40:05 Info | CancelableDlg:Scraping data... | Thread completed, still running 14.
06/12/2023 09:40:05 Warn | Scraping 42160: Location header points to the Steam Store homepage, aborting scraping.
06/12/2023 09:40:05 Info | CancelableDlg:Scraping data... | Thread completed, still running 13.
06/12/2023 09:40:05 Warn | Scraping 407510: Location header points to the Steam Store homepage, aborting scraping.
06/12/2023 09:40:05 Info | CancelableDlg:Scraping data... | Thread completed, still running 12.
06/12/2023 09:40:05 Warn | Scraping 369200: Location header points to the Steam Store homepage, aborting scraping.
06/12/2023 09:40:05 Info | CancelableDlg:Scraping data... | Thread completed, still running 11.
06/12/2023 09:40:05 Warn | Scraping 245130: Location header points to the Steam Store homepage, aborting scraping.
06/12/2023 09:40:05 Info | CancelableDlg:Scraping data... | Thread completed, still running 10.
06/12/2023 09:40:05 Warn | Scraping 239220: Location header points to the Steam Store homepage, aborting scraping.
06/12/2023 09:40:05 Info | CancelableDlg:Scraping data... | Thread completed, still running 9.
06/12/2023 09:40:05 Warn | Scraping 227700: Location header points to the Steam Store homepage, aborting scraping.
06/12/2023 09:40:05 Info | CancelableDlg:Scraping data... | Thread completed, still running 8.
06/12/2023 09:40:05 Warn | Scraping 226980: Location header points to the Steam Store homepage, aborting scraping.
06/12/2023 09:40:05 Info | CancelableDlg:Scraping data... | Thread completed, still running 7.
06/12/2023 09:40:05 Warn | Scraping 224420: Location header points to the Steam Store homepage, aborting scraping.
06/12/2023 09:40:05 Info | CancelableDlg:Scraping data... | Thread completed, still running 6.
06/12/2023 09:40:06 Warn | Scraping 726360: Hit an age check without redirect, aborting scraping.

@Pixieismyname
Copy link
Author

Using the premium API, it works alot better

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

1 participant