Permalink
Switch branches/tags
Nothing to show
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
35 lines (26 sloc) 1.04 KB

#Artemis - a scalable, decentralized and fault-tolerant web crawler.

##Features

  • Supports http(s)/ftp(s)/tor hidden service/magnet URIs
  • Supports ftp/http basic/http digest authentifications and authentication forms
  • Supports specific handlers ( based on URIs rules)
  • Tailored crawl based on URI/content-type/time
  • Smart crawl to reduce bandwidth consumption

##Install

###Needed dependencies

  • Python3.4 or highter
  • Tor relay on each slave
  • transmission-daemon on each slave

###Config

  • Master config : See config_path/master.ini
  • Monitor config : See config_path/monitor.ini
  • Slave config : See config_path/slave.ini
  • Gateway : See config_path/gateway.ini

NB : default_config_path = /etc/artemis

######Warning : Do not used the SSL certificats provided( for debug only )

##Quickstart

 python3.4 slave.py  #To start a slave node
 python3.4 master.py  #To start a master node
 python3.4 monitor.py  #To start a monitor node
 python3.4 gateway.py --host a_master_ip --port a_netarea_of_the_selected_master  #To init crawl