This is a crawler for the Open Data-Plattform of the Swiss Confederation. opendata.swiss collects open datasets from the national, cantonal and communal administration which is published for free use. The crawler attempts to download each dataset and analyze it.
The result is an interactive viusalization that allows the exploriation of the data on the platform by different criteria.
First of all you should set all the options in
config.py to your preference.
The dependecies are:
- Python 2.7 and everything in
And that is basically it. You can now run:
in your console.
One complete run takes about 16 hours in order to download and analyze over 3000 datasets.
The result of the analysis is stored at the
output_file path that is defined in the
config.py file. Examples of previous runs can be found in the
index.html file displays a visualization of a data file. By default it uses one of the existing files in
data from a previous analysis. If you want to modify it to display the data from your own run, it's not very hard, although the code is a bit messy. You need to set the variable
index.html to the path of you output file.
The visits data is not open. Therefore this tab doesn't work in the visualization. Sorry about that.