Skip to content

ahmia/ahmia-crawler

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Build Status Code Health Requirements Status

https://ahmia.fi/

Ahmia is the search engine for .onion domains on the Tor anonymity network. It is led by Juha Nurmi and is based in Finland. This repository contains crawlers used by Ahmia search engine

Prerequisites

Ahmia-index should be installed and running

Installation guide

Install dependencies:

Ubuntu 16.04

apt-get install build-essential python-pip python-virtualenv
apt-get install libxml2-dev libxslt1-dev python-dev libffi-dev libssl-dev
apt-get install tor polipo

Fedora 23

dnf install @development-tools redhat-rpm-config python-pip python-virtualenv
dnf install libxml-devel libxslt-devel python-devel libffi-devel openssl-devel
dnf install tor polipo

Install requirements in a virtual environment

python3 -m virtualenv venv3
source venv3/bin/activate
python3 -m pip install -r requirements/prod.txt

Prefer own python HTTP proxy

Look fleet installation here.

Usage

In order to execute the crawler to run permanently:

source venv/bin/activate
./run.sh &> crawler.log

Specific run examples

Primary
scrapy crawl ahmia-tor -s DEPTH_LIMIT=3 -s ROBOTSTXT_OBEY=0 -s FULL_PAGERANK_COMPUTE=True
or
scrapy crawl ahmia-tor -s DEPTH_LIMIT=5 -s LOG_LEVEL=INFO
or
scrapy crawl ahmia-i2p -s DEPTH_LIMIT=100 -s LOG_LEVEL=DEBUG
or
scrapy crawl ahmia-i2p -s DEPTH_LIMIT=1 -s ROBOTSTXT_OBEY=0
or
scrapy crawl ahmia-tor -o items.json -t json
or
scrapy crawl ahmia-tor -s DEPTH_LIMIT=1 -s ALLOWED_DOMAINS=/home/juha/allowed_domains.txt -s TARGET_SITES=/home/juha/seed_list.txt -s ELASTICSEARCH_TYPE=targetitemtype

About

Collection of crawlers used by the ahmia search engine

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published