Skip to content
master
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

README.md

Build Status Code Health Requirements Status

https://ahmia.fi/

Ahmia is the search engine for .onion domains on the Tor anonymity network. It is led by Juha Nurmi and is based in Finland. This repository contains crawlers used by Ahmia search engine

Prerequisites

Ahmia-index should be installed and running

Installation guide

Install dependencies:

Ubuntu 16.04

apt-get install build-essential python-pip python-virtualenv
apt-get install libxml2-dev libxslt1-dev python-dev libffi-dev libssl-dev
apt-get install tor polipo

Fedora 23

dnf install @development-tools redhat-rpm-config python-pip python-virtualenv
dnf install libxml-devel libxslt-devel python-devel libffi-devel openssl-devel
dnf install tor polipo

Install requirements in a virtual environment

virtualenv /path/to/venv
source /path/to/venv/bin/activate
(venv)pip3 install -r requirements/prod.txt

Prefer own python HTTP proxy

Look fleet installation here.

Usage

In order to execute the crawler to run permanently:

source /path/to/venv/bin/activate
./run.sh &> crawler.log

Specific run examples

Primary
scrapy crawl ahmia-tor -s DEPTH_LIMIT=3 -s ROBOTSTXT_OBEY=0 -s FULL_PAGERANK_COMPUTE=True
or
scrapy crawl ahmia-tor -s DEPTH_LIMIT=5 -s LOG_LEVEL=INFO
or
scrapy crawl ahmia-i2p -s DEPTH_LIMIT=100 -s LOG_LEVEL=DEBUG
or
scrapy crawl ahmia-i2p -s DEPTH_LIMIT=1 -s ROBOTSTXT_OBEY=0
or
scrapy crawl ahmia-tor -o items.json -t json
or
scrapy crawl ahmia-tor -s DEPTH_LIMIT=1 -s ALLOWED_DOMAINS=/home/juha/allowed_domains.txt -s TARGET_SITES=/home/juha/seed_list.txt -s ELASTICSEARCH_TYPE=targetitemtype

About

Collection of crawlers used by the ahmia search engine

Resources

Releases

No releases published
You can’t perform that action at this time.