- Scrape the Twitter followers of some accounts.
- Get the corresponding wallet address for all users with a
.eth
address in their name or bio.
- First, scrape the Twitter followers of the accounts using the Twitter API.
scrape.py
does this. - Then for each user, check if they have a .eth address in their name or bio.
parse.py
does this. - If they do, get the corresponding wallet address using the ENS API.
wallet.py
does this.
- The Twitter API only allows 15 API requests per 15 minutes window. This is managed by putting code to sleep at 429 error.
- (My setup) The ENS API is working on a remote provider with a cap of 1 Million API credits.
- Clone the repo.
- Create a virtual environment.
python -m venv venv
- Install the requirements.
pip install -r requirements.txt
- Provide the Twitter API credentials and a remote eth node.
# To set your environment variables in your terminal, run the following lines: export BEARER_TOKEN='<your_bearer_token>' export PROVIDER_URL='<your_http_provider_url>'
- If you want to scrape the followers of a different account, change the
usernames
variable inmain.py
. - Run
main.py
. Using the command:python main.py
- The followers' data of users will be stored in
out/<username>.json
. - The wallet addresses will be stored in
out/wallet.csv
.
- Occurrences of
.eth
in the entire user model are either in name, description, or as URL in the description or profile URL and location. Per the instruction, the code only scrapes for name and bio but could easily be scaled to all aforementioned parameters. - eth addresses are only having alphanumeric characters and period(.) and hyphen(-) ending with .eth. If there are more allowed characters, regex can be updated.
- For the given usernames, the output is stored in
out
folder. Available here. - All wallet addresses are also stored in google sheets.