This tool allows analyzing a large number of web servers that can be further categorized. It detects if the web page is hosted on a shared environment, support of security protocols as a DNSSEC, HTTPS. It also provides an information about IPv6 support. During processing, the actual location of the detected web servers is also detected.
The obtained data can be printed or exported to a CSV file. Geographical data can be shown on the interactive map.
The application requires an Python 3.5 or greater support. It is recommended to using with virtual environment (venv).
Running the app using virtual environment
$ python3 -m venv env $ source env/bin/activate (env) $ == work in virtual environment == (env) $ deactivate
Installing the requirements dependencies
Application requires couple of Python packages which are listed in
requirements.txt. Using pip command you can
easily install them:
pip install -r requirements.txt
Creating an empty database
SQLite database is used for storing evaluated pages. It supports division to certain categories. Empty database schema can be created:
For evaluating input domains, you can use
process_csv.py scripts. For example:
./process_page.py cat1 vutbr.cz.
Obtained data stored in SQLite database can be viewed or exported using commands:
./print_category.pyprints results within the category.
./print_stats.pyprints overall results of all categories.
./print_pages.pyallows to print the webs which are hosted in network of some provider. You can create query by network address or provider domain name (--webhoster).
./export_csv.pyit create an CSV output file named according the selected category.
Database of known web hosters
Some internal tests works with database of know hosters. To supply this database you can use command:
./database.py addhoster amazon.com
Server location map visualisation
The position of all detected webservers can be shown in map using command:
Print server information
modules/webserver_stats.py command you can print the server information which are used in evaluating scripts. It includes informations such a server software version and allowance of monitored protocols.
This software was supported supported by FEKT-S-17-4184 research project of Brno University of Technology.