Skip to content
Crawler to map out all Groestlcoin nodes for mainnet and testnet
Python HTML CSS Shell
Branch: master
Clone or download
Pull request Compare This branch is 9 commits ahead, 4 commits behind blakebjorn:master.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
geoip add geoip Jan 11, 2019
static Minor fixes Jul 2, 2019
templates Minor fixes Jul 2, 2019
LICENSE init Dec 6, 2018
README.md Remove standalone option for certbot Dec 13, 2019
app.py Enable https Jul 4, 2019
autodoc.py init Dec 6, 2018
config.py init Dec 6, 2018
crawler.py More fixes Jul 2, 2019
crawler_config.yml Minor fixes Jul 4, 2019
flask.ini More updates Jul 2, 2019
flask_config.py init Dec 6, 2018
groestlnodes Add nginx and systemd files with instructions Jul 7, 2019
groestlnodes.service Add nginx and systemd files with instructions Jul 7, 2019
models.py Groestlize part 1 Jul 2, 2019
protocol.py Groestlize part 1 Jul 2, 2019
requirements.txt More updates Jul 2, 2019
wsgi.py init Dec 6, 2018

README.md

Groestl Nodes

Groestl Nodes is a crawler that attempts to map out all Groestlcoin nodes for mainnet and testnet

A flask web server is included to display the data.

Usage


# Install packages
sudo apt-get install python3-pip python3-dev nginx

# Clone repository
git clone https://github.com/Groestlcoin/groestl-nodes /groestl-nodes
cd /groestl-nodes

# psycopg2-binary is required for postgres support
# uwsgi is required for nginx/apache deployment
pip install -r requirements.txt

# Setup geoip database
cd geoip && ./update.sh && cd ..

# Run crawler in loop
python3.7 crawler.py --seed --crawl --dump
screen
python3.7 crawler.py --seed --crawl --dump --loop
ctrl+A and ctrl+D

# Enable https
sudo apt-get install software-properties-common
sudo add-apt-repository ppa:certbot/certbot
sudo apt-get update
sudo apt-get install certbot
certbot certonly -d nodes.groestlcoin.org

# Change the project directory ownership to www-data
chown -R www-data.www-data /groestl-nodes

# Copy uWSGI startup file
cp groestlnodes.service /etc/systemd/system/groestlnodes.service

# Start uWSGI
sudo systemctl start groestlnodes

# Check the status
sudo systemctl status groestlnodes

# You should be able to see the socket with
ls /groestl-nodes/groestlnodes.sock

# Enable it on startup
sudo systemctl enable groestlnodes

# Copy Nginx file
cp groestlnodes /etc/nginx/sites-available/

# Add symbolic link for site-enabled
ln -s /etc/nginx/sites-available/groestlnodes /etc/nginx/sites-enabled

# Remove the default configuration of Nginx
rm /etc/nginx/sites-enabled/default

# Restart nginx
systemctl restart nginx

The --seed parameter is only needed for the first run. It will hit all the DNS seeds specified in the config file, as well as all individual seeder nodes (if applicable)

The --crawl parameter iterates through all known nodes and stores them in the specified database

The --dump parameter writes all data to disk in json, csv, and txt format for ingestion by the webserver

IPv6 Nodes will only be reachable if you have IPv6 Routing available. To set up IPv6 routing on an AWS deployment see here

Onion Nodes will only be reachable if you have a Tor server running (apt install tor)

Deployment

The crawler is best run via cron jobs, --dump instances should be scheduled separately from --crawl jobs.

flock should be used to prevent multiple instances from running concurrently

You can’t perform that action at this time.