In the beginning, TDP cranked out some
quick and dirty statistics
to illustrate the diversity issues in the Tor overlay network, among
other things. They were generated from the old-style Tor data via
blutmagie, using some simple sh scripts.
Now we've moved on to the next generation, still quick but not so dirty. We're using the new JSON data which gives us the ability to not only look at this point in time, but at the history of certain metrics. It also provides a more convenient interface for querying and accessing the raw data.
For now we're trying to keep it simple. There is a Perl script to produce ranked output from raw data, rankit.pl. This script is fed by output produced by jq over the raw JSON input data we get from OnionOO. Currently we only have point-in-time reports like the old Quick-and-Dirty ones; the simple-reports.sh script is the driver for generating these.
Next up will be reports that show how things are changing over various units of time. We also will be switching to HTML output from plain text. We have not yet integrated these new reports into our web site... all a work in progress.
Feel free to open Issues with us or ask questions on IRC.
simple-reports.sh script relies on jq.
Under OpenBSD it is available in ports and can be installed like this:
$ doas pkg_add jq
rankit.pl script relies on
Modern::Perl. Under OpeBSD
it can be installed like so:
$ doas pkg_add p5-Modern-Perl
Scripts in this repository:
- get-all.sh: download a dump of all current stats into a date-stamped directory, produce compressed tarball of the result for easy copying.
- simple-reports.sh: driver script for the simple,
non-historical reports. Invokes jq on
details.jsonand pump the results through various invocations of
- rankit.pl: Perl script to produce ranked output of various kinds from raw input data.
Our process is as follows:
get-all.shto produce a directory named
$ date +%Y%m%d 20170629 # we only need the details data for now $ ./get-all.sh details # now ./20170629/details.json exists
simple-reports.shto produce .txt reports; by default they will go in the current directory, but you can use the
--outdiroption to change this:
$ ./simple-reports.sh --indir=20170629 --outdir=20170629 :: generating 20170629/bw-by-os.txt :: generating 20170629/os-count.txt :: generating 20170629/cweight-by-os.txt :: generating 20170629/bw-by-vers.txt :: generating 20170629/vers-count.txt :: generating 20170629/cweight-by-vers.txt :: generating 20170629/bw-by-cc.txt :: generating 20170629/cweight-by-cc.txt :: generating 20170629/bw-by-as.txt :: generating 20170629/bw-by-asn.txt :: generating 20170629/cweight-by-as.txt :: generating 20170629/cweight-by-asn.txt
Let us know if you have any problems.