Websites crawler with built-in exploration and control web interface
JavaScript Java Python HTML CSS PHP Other
Latest commit 41df1b1 Jun 20, 2017 @jacomyma jacomyma references
Failed to load latest commit information.
_deprecated various cleaning and unused code Jul 5, 2013
config let's save a bit of wasted space by scrapyd Mar 9, 2016
doc adjust doc to explain this is a manual method without any scrapyd ser… Jan 18, 2017
hyphe_backend ensure compat py2.6 Aug 26, 2016
hyphe_frontend fix impossible to use empty query after 1st search in listWEs Aug 28, 2016
memory_structure Don't add prefix on add/merge when no reason to Aug 20, 2016
.dockerignore Frontend, backend and Thrift v0.8 containers Jun 10, 2015
.gitignore add api method to archive a backup of WEs/crawls/links and run it aut… Jun 17, 2015
COPYING Add licenses Sep 18, 2013
LICENSE.CECILL-C Add licenses Sep 18, 2013
LICENSE.LGPL Add licenses Sep 18, 2013 references Jun 20, 2017
Vagrantfile --demo Vagrantfile to test Docker in an Ubuntu environment Jun 10, 2015
docker-compose.yml Frontend, backend and Thrift v0.8 containers Jun 10, 2015
requirements-global-scrapyd.txt cleanup python requirements Apr 15, 2016

Hyphe: web corpus builder & links crawler

Welcome to Hyphe: developped by SciencesPo's médialab for the DIME-SHS Web project (Equipex).

Hyphe aims at providing a tool to crawl data from the web to generate networks between what we call WebEntities, which can be single pages as well as a website or a combination of such.


You can try a restricted version of Hyphe at the following url:

Papers and references

Papers about Hyphe

Papers using Hyphe

Easy start

DISCLAIMER: Hyphe has changed a lot between version 0.1 and 0.2. Migrating from an older version by pulling the code from git was guaranteed as best as possible, although it is highly recommended to reinstall from scratch. Older corpora can be reran by exporting the list of WebEntities from the old version and recrawl from that list of urls in the new version.

Install a release

For an easy install, the best solution is to download directly the release version, which was built to run against various GNU/Linux distributions (Ubuntu, Debian, CentOS...).

MacOS users and other distribution can now also run Hyphe locally on their machine using Docker thanks to @oncletom's work. See the dedicated section below.

Just uncompress the release archive, go into the directory and run the installation script.

Do not use sudo: the script will do so on its own and will ask for your password only once. This works so in order to install all missing dependencies at once, including mainly Java (OpenJDK-6-JRE), Python (python-dev, pip, virtualEnv, virtualEnvWrapper...), Apache2, MongoDB & ScrapyD.

If you are not comfortable with this, you can read the script and run the steps line by line or follow the Advanced install instructions for more control on what is actually installed.

# WARNING: DO NOT prefix any of these commands with sudo!
tar xzvf hyphe-release-*.tar.gz
cd Hyphe

To install from git sources of if you want to contribute to Hyphe's development, please follow the Advanced install documentation.

Configure Hyphe

Before starting Hyphe, you should probably adjust the settings first. Everything you need to change is in the global configuration file config/config.json.

Please read the Configuration documentation for details.

Run Hyphe

Hyphe relies on a web interface communicating with a server daemon which must be running at all times. To start, stop or restart the daemon, run (again, no sudo):

bin/hyphe <start|restart|stop> [--nologs]

By default the starter will display Hyphe's log in the console using tail. You can Ctrl-C whenever you want without shutting it off. Use the --nologs option to disable this.

You can always check the logs for both the core backend and each corpus' MemoryStructure in the log directory:

tail -f log/hyphe-*.log

As soon as the daemon is started, you can start playing with the web interface on your local machine at the following url: http://localhost/hyphe.

Serve on the web

Using the website on localhost, you can already use Hyphe. Although, if you want to let others use it as well (typically if you installed on a distant server), you need to make a few adjustments to the Apache configuration.

Please read the dedicated WebService documentation to do so.

Docker setup

Docker enables isolated install and execution of software stacks, which can be an easy way to install Hyphe locally on an individual computer, including on unsupported distributions like MacOS. Follow Docker install instructions to install Docker on your machine.

Install Docker Compose to set up and orchestrate Hyphe services in a single line.

docker-compose up

When using boot2docker for instance on MacOS, you might need beforehand to run the following:

boot2docker up
# and copy paste the 3 lines starting with export to set the environment variables

It will take a couple of minutes to spin everything up for the first time. Once the services are ready, you can access the frontend interface by connecting on its IP address:

open http://$(docker inspect -f '{{.NetworkSettings.IPAddress}}' hyphe_frontend_1):8000

Or, if you use boot2docker:

open http://$(boot2docker ip):8000

Notice: this is not a production setup. Get some inspiration from the docker-compose.yml to understand how to distribute the application on one or many machines.

Advanced developers features & contributing

Please read the dedicated Developers documentation and the API description.

What's next?

See our roadmap!


Mathieu Jacomy & Benjamin Ooghe-Tabanou @ SciencesPo médialab

Discover more of our projects at médialab tools

This work is supported by DIME-WEB part of DIME-SHS research equipment financed by the EQUIPEX program (ANR-10-EQPX-19-01).

Hyphe is a free software released under LGPL & CECILL-C licenses.