A farm operated by bots to grow and harvest new zim files.
Switch branches/tags
Nothing to show
Clone or download
Latest commit 462edf0 Jul 5, 2018
Permalink
Failed to load latest commit information.
dispatcher Scheduler (#56) Jul 5, 2018
warehouse Dispatcher improvement (#52) Jun 1, 2018
worker Scheduler (#56) Jul 5, 2018
.gitignore Scheduler (#56) Jul 5, 2018
CONTRIBUTING.md Add openZIM project CONTRIBUTING file Nov 25, 2017
LICENSE Initial commit Mar 15, 2017
README.md ftp server Jan 3, 2018

README.md

ZIM Farm

The ZIM farm (zimfarm) is a half-decentralised software solution to build ZIM files efficiently. This means scrapping Web contents, packaging them into a ZIM file and uploading the result to an online ZIM files repository.

Principle

The whole zimfarm system works as a task scheduler and is made of two components:

  • The dispatcher, the central node, which takes and dispatches zim file generation tasks. It is managed by the openZIM project admin and hosted somewhere on the Internet.

  • The workers, task executers, which are hosted by openZIM volunteers in different places around the world through Internet.

To get a new ZIM file the typical workflow is:

  1. User submits a new task (to get a specific ZIM file made).
  2. Dispatcher enqueues the task.
  3. A worker pulls the task.
  4. Worker generates the ZIM file and upload it.

Run the dispatcher

  1. Make sure docker and docker-compose are installed on your system
  2. Clone this repo
  3. Make sure you are in master branch
  4. Open a terminal session change directory to root of dispatcher folder
  5. Run docker-compose up --build
  6. Wait for docker to do its job
  7. Go to http://localhost:8080 with your Web browser

Run worker

  1. Clone this repo
  2. Make sure you are in master branch
  3. Open a terminal session change directory to root of worker folder
  4. Run docker build -t zimfarm_worker . && docker run -v /var/run/docker.sock:/var/run/docker.sock zimfarm_worker

Architecture

The whole system is built by reusing the best of free software libraries components. The whole ZIM farm solutioon itself is made available using Docker images.

Actually, we use even more Docker images, as the ZIM file are produced in dedicated custom Docker images the worker has to "invoke" depending of the properties of the task he has to accomplish.

Softwares

Dispatcher:

  • proxy: nginx
    • frontend: angular 2, Node.js
    • backend: flask, celery, sqlite, python
  • messaging queue: RabbitMQ

Worker:

  • python
  • docker

Example of API request

Login and get a token:

$ curl -i -X POST -H "username: foo" -H "password: bar" "https://farm.openzim.org/api/auth/login"

Request a creation of ZIM file::

curl -i -X POST -H "Content-Type: application/json; charset=utf-8" -H "token: eyJ0eXAiOiJK..." --data "[{ \"mwUrl\": \"https://bm.wikipedia.org/\", \"adminEmail\": \"foo@bar.org\", \"verbose\": true }]" "https://farm.openzim.org/api/task/mwoffliner"