Configuration of Zimfarm local instance

[seniyakk]

Hello.
I am trying to configure the local instance of Zimfarm. I have investigated the documentation, but it looks like I still need help.

Here is my docker-compose.yml

 services:
  zimit-ui:
    image: ghcr.io/openzim/zimit-ui:latest
    container_name: zimit-ui
    restart: unless-stopped
    labels:
      traefik.enable: true
      traefik.docker.network: traefik

      traefik.http.routers.zimit-http.rule: Host(`zimit.example.com`)
      traefik.http.routers.zimit-http.entrypoints: web
      traefik.http.routers.zimit-http.middlewares: https-redirect

      traefik.http.routers.zimit-https.rule: Host(`zimit.example.com`)
      traefik.http.routers.zimit-https.entrypoints: websecure
      traefik.http.routers.zimit-https.tls: true
      traefik.http.routers.zimit-https.tls.certresolver: prod
    environment:
      ZIMIT_API_URL: http://zimit_api:8000
    networks:
      - traefik
      - zimitfarm

  zimit_api:
    image: ghcr.io/openzim/zimit:zimit2
    container_name: zimit_api
    depends_on:
      - zimfarm_dispatcher
    command: python main.py
    environment:
      BINDING_HOST: 0.0.0.0
      INTERNAL_ZIMFARM_WEBAPI: http://zimfarm_dispatcher/v1
      _ZIMFARM_USERNAME: admin
      _ZIMFARM_PASSWORD: admin
      TASK_WORKER: worker
    volumes:
      - ../api/src:/app
    networks:
      - zimitfarm
    # ports:
    #   - 8000:8000

  zimfarm-ui:
    image: ghcr.io/openzim/zimfarm-ui:latest
    container_name: zimfarm-ui
    depends_on:
      - zimfarm_dispatcher
    labels:
      traefik.enable: true
      traefik.docker.network: traefik

      traefik.http.routers.zimfarm-http.rule: Host(`zimfarm.example.com`)
      traefik.http.routers.zimfarm-http.entrypoints: web
      traefik.http.routers.zimfarm-http.middlewares: https-redirect

      traefik.http.routers.zimfarm-https.rule: Host(`zimfarm.example.com`)
      traefik.http.routers.zimfarm-https.entrypoints: websecure
      traefik.http.routers.zimfarm-https.tls: true
      traefik.http.routers.zimfarm-https.tls.certresolver: prod

    environment:
      ZIMFARM_WEBAPI: http://zimfarm_dispatcher/v1
    networks:
      - traefik
      - zimitfarm
    ports:
      - 8888:80

  zimfarm_dispatcher:
    image: ghcr.io/openzim/zimfarm-dispatcher:latest
    container_name: zimfarm_dispatcher
    depends_on:
      - zimfarm_db
    environment:
      BINDING_HOST: 0.0.0.0
      JWT_SECRET: DH8kSNRdkEiJJCn2dOqfw
      POSTGRES_URI: postgresql+psycopg://zimfarm:ghRmSLSY8KpswzVq@zimfarm_db:5432/zimfarm
      ALEMBIC_UPGRADE_HEAD_ON_START: "1"
      ZIMIT_USE_RELAXED_SCHEMA: "y"
    networks:
      - zimitfarm
    # ports:
    #   - 80:80

  zimfarm_db:
    image: postgres:15.2-bullseye
    container_name: zimfarm_db
    environment:
      POSTGRES_DB: zimfarm
      POSTGRES_USER: zimfarm
      POSTGRES_PASSWORD: ghRmSLSY8KpswzVq
    volumes:
      - db:/var/lib/postgresql/data
    networks:
      - zimitfarm
    # ports:
    #   - 5432:5432

  zimfarm_receiver:
    image: ghcr.io/openzim/zimfarm-receiver:latest
    container_name: zimfarm_receiver
    ports:
      - 8222:22
    volumes:
      - ./receiver/create-warehouse-paths.sh:/contrib/create-warehouse-paths.sh
    environment:
      ZIMFARM_WEBAPI: http://zimfarm_dispatcher:8000/v1
    depends_on:
      - zimfarm_dispatcher
    networks:
      - zimitfarm

  zimfarm_worker-mgr:
    container_name: zimfarm_worker-mgr
    image: ghcr.io/openzim/zimfarm-worker-manager:latest
    depends_on:
      - zimfarm_dispatcher
    command: worker-manager --webapi-uri 'http://zimfarm_dispatcher/v1' --username test_worker --name test_worker
    environment:
      ZIMFARM_DISK: 16 GiB
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./test_worker-identity/id_rsa:/etc/ssh/keys/zimfarm
    networks:
      - zimitfarm

  zimfarm_task-worker:
    image: ghcr.io/openzim/zimfarm-task-worker:latest
    container_name: zimfarm_task-worker
    depends_on:
      - zimfarm_dispatcher
    command: task-worker --webapi-uri 'http://zimfarm_dispatcher/v1' --username test_worker --task-id <your_task_id>
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./test_worker-identity/id_rsa:/etc/ssh/keys/zimfarm
    environment:
      DEBUG: 1
      DOCKER_NETWORK: zimfarm
      ZIMFARM_DISK: 16 GiB
    networks:
      - zimitfarm

volumes:
  db:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: $VOLUMES_PATH_PREFIX/zimfarm/volumes/db

networks:
  traefik:
    external: true
  zimitfarm:
    name: zimitfarm

Questions:

1. Image for zimit API is this one ghcr.io/openzim/zimit:zimit2 ?
2. Are zimfarm_dispatcher and zimfarm_backend the same or 2 different containers?
3. For now I can run zimfarm-ui, zimfarm_dispatcher, zimfarm_receiver, zimfarm_db, zimit-ui. BUT zimit_api, zimfarm_worker-mgr, zimfarm_task-worker are falling with errors in logs:
   3.1. zimit_api: /usr/local/bin/entrypoint.sh: 3: exec: python: not found
   3.2. zimfarm_task-worker:

   usage: task-worker [-h] --task-id TASK_ID [--webapi-uri WEBAPI_URI] --username
                 USERNAME [--workdir WORKDIR]
   task-worker: error: argument --task-id: expected one argument
   

   How should worker-mgr provide task ID to task-worker? Or should worker-mgr create task-worker containers by himself, and I do not need zimfarm_task-worker in docker-compose file?
   3.3. zimfarm_worker-mgr: [2024-05-14 13:49:39,129: CRITICAL] private key is not a readable path. Where to get this private key, and for what is it used?
4. Is something else missing in my docker-compose.yml?

[benoit74]

Unfortunately, I won't have time to delve into all these deployment details.

But you should probably have a look at openzim/zimit-frontend#54 where I already gave lots of information to another user.

And also to https://github.com/openzim/zimfarm/blob/main/dev/docker-compose.yml

You can also have a look at https://github.com/kiwix/k8s/tree/main/zimfarm (our farm.openzim.org) and https://github.com/kiwix/k8s/tree/main/zimit (our farm.zimit.kiwix.org + zimit UI at zimit.kiwix.org).

We have to admit deploying zimfarm is not an easy feat at all, there are lots of moving pieces.