NeuViz is a data processing and visualization architecture for network measurement experiments. NeuViz has been tailored to work on the data produced by Neubot (Net Neutrality Bot), an Internet bot that performs periodic, active network performance tests.
How it works
The Python scripts used for the Elaboration stage are stored in the
backend folder. The scripts for the Web API (written in Node.js) are stored in the
frontend folder. The files for of the Web user interface (HTML + D3.js) are available in the
In the backend folder you can find the Python scripts of the Elaboration stage. We started from the Neubot data CSV files and we imported using the map.py script into a MongoDB database. After the import step we executed the reduce.py script to elaborate the aggregate the data by country, city, and provider for each month. We elaborated the median of these values and we exported it into a JSON structure, in order to be read from the Visualization client.
NeuViz exposes a Web API that allows the web interface (and possibly other users) to retrieve the JSON files prepared by the Elaboration stage. To expose the web API, run:
$ node index.js
Web user interface
- Python 2.7.3 (with pymongo, numpy)
- Node.js v0.10.25
How to execute the Elaboration stage
In order to elaborate the Neubot data from the CSV files we need to execute a couple of Python scripts to insert some geographical information about the country and the city that we are going to use. This step is needed because the Neubot data don't contain the latitude and longitude of the information reported.
You need to execute the following scripts:
$ python insert_city_mongodb.py worldcitiespop.txt $ python insert_country_mongodb.py GeoIPCountryWhois.csv
After that you can execute the map and reduce scripts to elaborate the Neubot data:
- Execute the map.py script to insert the Neubot data in MongoDB:
$ python map.py neubot.csv
where neubot.csv is the Neubot CSV file related to a specific month.
- Execute the reduce.py script to aggregate the data and elaborate the median values.
$ python reduce.py [month] [year] [fileout]
Where month is the month number and year is the year expressed using four digits. The fileout is the file name for the JSON result.
We already elaborated all the Neubot data from January 2012 to May 2013 and we stored the result in the folder
How to visualize the result of the project
We implemented an interactive world map to visualize the Network Neutrality results of the collected Neubot data.
In order to visualize the world map you need to execute the public folder inside a web server. We suggest to use the built in web server of python without any configuration. You can go inside the public folder and execute the following command:
$ python -m SimpleHTTPServer
Now you can point your browser to the URL http://localhost:8000/Network_Neutrality.html and enjoy the interactive world map of the Neubot data (the port 8000 is the default one provided by python).
A special thanks to Christian Racca of the TOP-IX Consortium, and all the staff and teachers of the BigDive 2013 course for their support during the development of the first prototype of NeuViz (aka "GramsciDevoted" project).