Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aws doesnt get all the tweets #155

Open
fb0x opened this issue Jan 22, 2024 · 7 comments
Open

aws doesnt get all the tweets #155

fb0x opened this issue Jan 22, 2024 · 7 comments

Comments

@fb0x
Copy link

fb0x commented Jan 22, 2024

hi everyone, im scraping tweets for 3 users and was running fine locally and deployed on aws. But now I only get the tweets from 2 users not for the third one. I ran a simple script to see if i can get his tweets and it worked fine locally but on aws it doesn't work properly. any idea why? only for this user and his tweets area accessible if i run the code locally

@mahrtayyab
Copy link
Owner

  1. Twitter blocks about all AWS IP addresses
  2. It is weird , that it is working for some , and not for other , can you please by any chance send the username of the concerned user.

@fb0x
Copy link
Author

fb0x commented Jan 23, 2024

  1. Twitter blocks about all AWS IP addresses
  2. It is weird , that it is working for some , and not for other , can you please by any chance send the username of the concerned user.

that's interesting, so where do your recommend me to deploy? other 2 users work fine but this one doens't indeed interesting.

Is that the case with other servers as well? Like google etc?

username is : Crypto_Shaggy

also what do you recommend me to do bypass this twitter block?

@mahrtayyab
Copy link
Owner

Use good proxy

@fb0x
Copy link
Author

fb0x commented Jan 23, 2024

Use good proxy

did it work for you on aws?

@EsqerYasen
Copy link

EsqerYasen commented Feb 13, 2024

  1. Twitter blocks about all AWS IP addresses
  2. It is weird , that it is working for some , and not for other , can you please by any chance send the username of the concerned user.

is there any way that can set on proxy for tweety ? I couldn't find in document page.
like set with ip:port something like this .

@mahrtayyab
Copy link
Owner

Twitter class accepts an optional argument proxy, it could be dict or tweety.types.Proxy, documentation is outdated right now, I will update it soon.

from tweety import Twitter
from tweety.types import Proxy, PROXY_TYPE_HTTP

proxy = Proxy(host="127.0.0.1", port=8080, proxy_type=PROXY_TYPE_HTTP, username="username", password="password")

client = Twitter("session", proxy=proxy)

@EsqerYasen
Copy link

Twitter class accepts an optional argument proxy, it could be dict or tweety.types.Proxy, documentation is outdated right now, I will update it soon.

from tweety import Twitter
from tweety.types import Proxy, PROXY_TYPE_HTTP

proxy = Proxy(host="127.0.0.1", port=8080, proxy_type=PROXY_TYPE_HTTP, username="username", password="password")

client = Twitter("session", proxy=proxy)

thank you so much! this helps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants