New version of BGP Ranking, complete rewrite in python3.6+ and an ARDB backend
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
api
bgpranking
bin
cache
client
storage
temp
website
.gitignore
LICENSE
Pipfile
Pipfile.lock
README.md
requirements.txt
setup.py

README.md

BGP-Ranking

New version of BGP Ranking, complete rewrite in python3.6+ and an ARDB backend

Installation

IMPORTANT: Use pipenv

NOTE: Yes, it requires python3.6+. No, it will never support anything older.

Install redis

git clone https://github.com/antirez/redis.git
cd redis
git checkout 5.0
make
make test
cd ..

Install ardb

git clone https://github.com/yinqiwen/ardb.git
cd ardb
DISABLE_WARNING_AS_ERROR=1 make  # ardb (more precisely rocksdb) doesn't compile on ubuntu 18.04 unless you disable warning as error
cd ..

Install & run BGP Ranking

git clone https://github.com/D4-project/BGP-Ranking.git
cd BGP-Ranking
pipenv install
echo BGPRANKING_HOME="'`pwd`'" > .env
pipenv shell
# Starts all the backend
start.py
# Start the web interface
start_website.py

Shutdown BGP Ranking

stop.py

Directory structure

Config files: bgpranking / config / *.json

Per-module parsers: bgpraking / parsers

Libraries : brpranking / libs

Raw dataset directory structure

Files to import

Note: The default location of <storage_directory> is the root directory of the repo.

<storage_directory> / <vendor> / <listname>

Last modified date (if possible) and lock file

<storage_directory> / <vendor> / <listname> / meta

Imported files less than 2 months old

<storage_directory> / <vendor> / <listname> / archive

Imported files more than 2 months old

<storage_directory> / <vendor> / <listname> / archive / deep

Databases

Intake (redis, port 6579)

Usage: All the modules push their entries in this database.

Creates the following hashes:

UUID = {'ip': <ip>, 'source': <source>, 'datetime': <datetime>}

Creates a set intake for further processing containing all the UUIDs.

Pre-Insert (redis, port 6580)

Usage: Make sure th IPs are global, validate input from the intake module.

Pop UUIDs from intake, get the hashes with that key

Creates the following hashes:

UUID = {'ip': <ip>, 'source': <source>, 'datetime': <datetime>, 'date': <date>}

Creates a set to_insert for further processing containing all the UUIDs.

Creates a set for_ris_lookup to lookup on the RIS database. Contains all the IPs.

Routing Information Service cache (redis, port 6581)

Usage: Lookup IPs against the RIPE's RIS database

Pop IPs from for_ris_lookup.

Creates the following hashes:

IP = {'asn': <asn>, 'prefix': <prefix>, 'description': <description>}

Ranking Information cache (redis, port 6582)

Usage: Store the current list of known ASNs at RIPE, and the prefixes originating from them.

Creates the following sets:

asns = set([<asn>, ...])
<asn>|v4 = set([<ipv4_prefix>, ...])
<asn>|v6 = set([<ipv6_prefix>, ...])

And the following keys:

<asn>|v4|ipcount = <Total amount of IP v4 addresses originating this AS>
<asn>|v6|ipcount = <Total amount of IP v6 addresses originating this AS>

Long term storage (ardb, port 16579)

Usage: Stores the IPs with the required meta informations required for ranking.

Pop UUIDs from to_insert, get the hashes with that key

Use the IP from that hash to get the RIS informations.

Creates the following sets:

# All the sources, by day
<YYYY-MM-DD>|sources = set([<source>, ...])
# All the ASNs by source, by day
<YYYY-MM-DD>|<source> -> set([<asn>, ...])
# All the prefixes, by ASN, by source, by day
<YYYY-MM-DD>|<source>|<asn> -> set([<prefix>, ...])
# All the tuples (ip, datetime), by prefixes, by ASN, by source, by day
<YYYY-MM-DD>|<source>|<asn>|<prefix> -> set([<ip>|<datetime>, ...])