Crawler to map out all cryptocurrency nodes based on the Bitcoin protocol
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
geoip
static
templates
LICENSE
README.md
app.py
autodoc.py
config.py
crawler.py
crawler_config.yml
flask.ini
flask_config.py
models.py
protocol.py
requirements.txt
wsgi.py

README.md

Open Nodes

Open Nodes is a crawler that attempts to map out all nodes of crypto currencies based on the bitcoin protocol.

A flask web server is included to display the data.

Usage

# Install python virtual environment
apt install python3-pip python3-venv
python3 -m venv venv
source venv/bin/activate

# Install packages (Python 3)
# psycopg2-binary is required for postgres support
# uwsgi is required for nginx/apache deployment
pip install -r requirements.txt

# setup geoip database
cd geoip && ./update.sh && cd ..

# run crawler
python crawler.py --seed --crawl --dump

# run development flask server
python app.py

The --seed parameter is only needed for the first run or when adding a new network. It will hit all the DNS seeds specified in the config file, as well as all individual seeder nodes (if applicable)

The --crawl parameter iterates through all known nodes and stores them in the specified database

The --dump parameter writes all data to disk in json, csv, and txt format for ingestion by the webserver

IPv6 Nodes will only be reachable if you have IPv6 Routing available. To set up IPv6 routing on an AWS deployment see here

Onion Nodes will only be reachable if you have a Tor server running (apt install tor)

Deployment

The crawler is best run via cron jobs, --dump instances should be scheduled separately from --crawl jobs.

flock should be used to prevent multiple instances from running concurrently

For production use, the default database (sqlite) should not be used, as the file lock timeout will prevent simultaneous crawling, dumping, and reporting/api calls.

NGINX deployment

wsgi.py and flask.ini are included for nginx deployment. You will need to pip install uwsgi and then set up a system service (see https://www.digitalocean.com/community/tutorials/how-to-serve-flask-applications-with-uwsgi-and-nginx-on-ubuntu-16-04)

nano /etc/systemd/system/opennodes.service

and add

[Unit]
Description=OpenNodes uWSGI instance
After=network.target

[Service]
User= {{ USERNAME }}
Group=www-data
WorkingDirectory=/home/{{ PROJECT ROOT }}
Environment="PATH=/home/{{ PATH TO PYTHON/VENV BIN DIR }}"
ExecStart=/home/{{ PATH TO PYTHON/VENV BIN DIR }}/uwsgi --ini flask.ini

[Install]
WantedBy=multi-user.target

Now you need to add nginx configuration to handle proxy requests

nano /etc/nginx/sites-available/opennodes

and add

server {
    listen 80;
    server_name {{ SERVER DOMAIN OR IP }};
    
    location / {
        include uwsgi_params;
        uwsgi_pass unix:///home/{{ PROJECT ROOT }}/opennodes.sock;
    }
}

Where the domain/ip is likely 127.0.0.1

Then add a symbolic link, remove the default configuration, and restart nginx

ln -s /etc/nginx/sites-available/opennodes /etc/nginx/sites-enabled
rm /etc/nginx/sites-enabled/default
systemctl restart nginx