Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing user agent #16

Open
janldk opened this issue May 30, 2023 · 0 comments
Open

Missing user agent #16

janldk opened this issue May 30, 2023 · 0 comments

Comments

@janldk
Copy link

janldk commented May 30, 2023

Thanks for putting this together.

One thing, tho, you might want to add a specific or a random one from a pool of user agent string(s). Not doing so causes every request by your checkers to send python-requests/2.xx.x to the servers you are querying.

While this might work it's poor opsec and will eventually increase the risk of being banned / endpoints being hardened / cool methods being removed as it is pretty noisy.

Of course users can monkey patch this globally by - e.g. inserting something like requests.utils.default_user_agent = lambda: 'Mozilla Firefox....' - but imho the tool should provide proper opsec or clearly state in the README that it broadcast itself as a crawler.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant