X (twitter) scraper

intro

This project is based on https://github.com/Mostafa-Ehab/Twitter-Scrapper

A selenium based X (twitter) scraper

...

install & env

git clone git@github.com:LthreeC/X-scrapper.git
cd X-scrapper
pip install -r requirements.txt

ensure PYTHONPATH is set to the root of the project

export PYTHONPATH=<your project root>:$PYTHONPATH

run

input/conf.json: configuration file, make sure access token is set

how to get access token

scrap_tweets: input a list of id, get tweets

python script/scrap_tweets.py --threads 1 --users -1 --tweet_num 100 --time_range 240 --homepage_list home

output format:

{
  "<token>": {
    "<url>": {
      "url": "<url>",
      "ID": "",
      "Nickname": "",
      "Date": "",
      "isReposted": ,
      "isMedia": ,
      "Language": "",
      "Replies": ,
      "Reposts": ,
      "Likes": ,
      "Bookmarks": ,
      "Views": ,
      "Text": ""
    }
  }
}

scrap_interactions: input a list of id, get interactions

python script/scrap_interactions.py --threads 1 --users 1 --url_list "https://x.com/MathVerseNFT/status/1899369827866685661" "https://x.com/MathVerseNFT/status/1874155560058196310"

output format:

{
  "<token>": {
    "<url>": {
      "quotes": [],
      "retweets": [
        {
          "ID": "",
          "Nickname": ""
        },
        {
          "ID": "",
          "Nickname": ""
        }
      ]
    }
  }
}

scrap_homepage: input a list of id, get homepage info

python script/scrap_homepage.py --threads 1 --users 1 --url_list "elonmusk" "NASA" "realDonaldTrump" "realDonaldTrump"

output format:

{
    "<token>": {
        "<id>": {
            "ID": "",
            "Nickname": "",
            "Description": "",
            "dateCreated": "",
            "homeLocation": "",
            "followers": ,
            "following": ,
            "posts": 
        },
    }
}

webui

python webui.py

or

gradio webui.py