Web Scraping from Crypto-exchange Websites with Python & Selenium

This python script uses selenium browser to collect Price, Bid-Ask spread and Trade history data from online trade platforms of several crypto-exchanges. Data is being collected continually and stored in mySQL database.

Demonstration of the Running Script

Installation

You can clone below repository:
git clone https://github.com/serhatci/web-scraping-from-cryptoexchanges.git

install the requirements:
pip install -r requirements.txt

Be sure following chrome driver file is in the script folder:
chromedriver.exe

You have to provide your own SQL database connection information in the run.py file:

    #### BELOW INFORMATION SHOULD BE SUBMITTED ###########
    pair = 'your_downloaded_trade_pair' (For example: btc_usd)
    db_name = 'your_database_name' (For example: Bitcoin)
    host = 'your_host'
    user = 'your_username'
    password = 'your_passwords'
    ######################################################

and run the application:
python script/run.py

Customization

Script uses a JSON file to learn target data for web scraping. Xpath of price, spread and trade history data are declared for each exchange in the script/exchanges.json file as below. You can update the JSON file by adding new exchanges or overwriting the existing ones. Script will handle opening necessary number of the browser windows&tabs and collecting&saving the data.

"FTX": {  
        "name" : "FTX",  
        "web": "https://ftx.com/trade/USDT/USD",  
        "homePageClick": {  
            "clickXpath":"//*[@id='root']/div/div[2]/div/header/div[1]/div[2]/button"},
        "price": {
            "clickXpath": "",
            "xpath": "//*[@id='root']/div/div[2]/div/main/div[1]/div[2]/div/div[3]"},
        "spread": {
            "clickXpath": "",
            "calculate": "Yes",
            "xpathBuy": "/html/body/div/div/div[2]/div/main/div[3]/div[4]/div/div/div[2]/div[2]/table/tbody/tr[1]/td[2]",
            "xpathSell": "//*[@id='root']/div/div[2]/div/main/div[3]/div[4]/div/div/div[2]/div[3]/div/table/tbody/tr[1]/td[1]"},
        "trade": {
            "clickXpath": "",
            "xpath": "/html/body/div/div/div[2]/div/main/div[3]/div[5]/div/table/tbody",
            "order": ["Price","Amount","Time"]}}