New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Smarter scraper that commits "diffs" to the database #268

Open

ericyoondotcom opened this issue Jul 17, 2024 · 0 comments

Assignees

Labels

will be fixed during rewrite

Contributor

ericyoondotcom commented Jul 17, 2024

Right now, every time the scraper is run, the entire database is wiped, and recreated from scratch.

In the unlikely chance that the scraper misses a user for some reason, that user is completely deleted from the database.

Scraper should be more intelligent about this.

If a user is added, add the user.
If a user is deleted—and not simply missing because of an HTTP or parsing error—remove them.
Same with individual user data points.

ericyoondotcom self-assigned this

ericyoondotcom added the will be fixed during rewrite label

ericyoondotcom mentioned this issue

Create backup database to be used if scraper fails #91

Closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment