Onionscan backed by elasticsearch.
Python
Latest commit 68f1c01 Aug 30, 2016 Gerry Fixed parsing of certs
Permalink
Failed to load latest commit information.
.gitignore Initial commit Aug 25, 2016
README.md Updated readme Aug 25, 2016
elastic_onion.py Initial code commit Aug 25, 2016
logstash.conf Fixed parsing of certs Aug 29, 2016

README.md

Elastic Onion

Onionscan backed by elasticsearch.

Fair warning, I've been on a get all the X into ES kick lately, and this is just the latest value of X. It was also hacked together in a late caffeine fueled night. Don't expect it to be too stable, but it runs pretty well for me and has provided some fun/interesting data to build dashboards with or build out other scans from ES.

I blame @jms_dot_py and this post of his for giving me the idea.

It should be noted that there are a few extraneous moving parts to this. There are reasons, but for a PoC could be removed. E.g., redis in favor of just throwing json at logstash, beanstalk instead of a file/mem based queue, etc.

Install requirements

These notes should be enough to show you what you need. Installing beanstalk, redis, elasticsearch, logstash, kibana, their prerequisites and some python libraries.

$ sudo apt-get install tor git bison libexif-dev screen python-pip golang
$ sudo apt-get install beanstalkd
$ pip install stem pyyaml beanstalkc
$ go get github.com/s-rah/onionscan
$ mkdir ~/gocode
$ echo 'GOLANG=~/gocode' >> .bash_profile
$ tor --hash-password mysecrettorpassword
$ sudo bash -c 'cat << EOF >> /etc/tor/torrc
> ControlPort 9051
> ControlListenAddress 127.0.0.1
> HashedControlPassword 16:5A4ED0F4254848636082D9CBB95379E8BBAD2245D24A091B230B661D7A
> EOF'
$ sudo service tor restart
$ sudo apt-get install redis-server
$ pip install redis
$ sudo add-apt-repository ppa:webupd8team/java -y
$ sudo apt-get update && sudo apt-get install oracle-java8-installer -y
$ wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
$ echo "deb https://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list
$ sudo apt-get update && sudo apt-get install elasticsearch -y
$ sudo /bin/systemctl daemon-reload
$ sudo /bin/systemctl enable elasticsearch.service
$ pip install elasticsearch
$ echo "deb http://packages.elastic.co/kibana/4.4/debian stable main" | sudo tee -a /etc/apt/sources.list.d/kibana-4.4.x.list
$ sudo apt-get update && sudo apt-get -y install kibana
$ sudo service kibana start
$ echo 'deb http://packages.elastic.co/logstash/2.2/debian stable main' | sudo tee /etc/apt/sources.list.d/logstash-2.2.x.list
$ sudo apt-get update && sudo apt-get install logstash -y
$ sudo cp logstash.conf /etc/logstash/conf.d/elasticonion.conf'
$ sudo service logstash restart