Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove new users in the webscrape #10

Closed
Nighkali opened this issue Feb 7, 2021 · 1 comment
Closed

Remove new users in the webscrape #10

Nighkali opened this issue Feb 7, 2021 · 1 comment

Comments

@Nighkali
Copy link
Contributor

Nighkali commented Feb 7, 2021

Considering adding a post.author during the webscraper and dumping those names into a filter. Praw can snag the userinfo and test to see if they have been active for over a certain time, might dump out the bots. We can keep the user info in a cache. Since we are talking about thousands of comments to search through, we can pass these off to a separate thread to build a blacklist of commentors to compare against the main thread. This would allow them first pass through but after they are checked and rejected, future checks would hit on the blacklist. So 1 bot that spams a random pump and dump would have low volume and not show up hopefully. Let me know if its worth the effort.

@iam-abbas
Copy link
Owner

Thanks for the suggestion! I am going to add this in beta 1.3 😃

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants