tuulos edited this page Sep 12, 2010 · 14 revisions

Main components of the Disco core

As of Disco 0.3, migration to the architecture below is still work in progress. Especially master and event server still need refactoring.

  • Master disco_server.erl:
    • controls available resources (nodes)
    • maintains the global state of running workers
    • handles fault-tolerance
  • Scheduler fair_scheduler*.erl
    • controls priority of all jobs and tasks
    • controls data locality and load balancing
  • Event (log) server event_server.erl
    • controls global events and statistics
    • controls logs for archived jobs
  • Job coordinator job_coordinator.erl
    • controls that all tasks belonging to a job are executed
    • handles events related to a job
  • Worker supervisor disco_worker.erl
    • controls a single running task
  • Web API web_server.erl, disco_web.erl, mochiweb/*.erl
    • Maps requests from the Web API to internal requests
  • DDFS ddfs/*.erl
    • Storage subsystem