A distributed make using celery for task distribution.
- Install puppet:
sudo apt-get install puppet
- Install puppetlabs-apt:
sudo puppet module install --modulepath=/usr/share/puppet/modules puppetlabs-apt
- Apply the puppet manifest in
deploy
sudo puppet apply local_intall.pp
Assuming the following directory structure:
- src
- celeryconfig.py
- makeparse.py
- logging.ini
- master.py
- result.py
- work.py
- Makefile
- premier.c
- Create
master_node
withlocalhost
as the content - Start
celery
celery worker -A work -l info
- Launch the main program with:
python master.py -f Makefile premier
- Reserve nodes on Grid'5000 with the script
reserve.sh
indeploy
directory.
./reserve.sh 4 1:30:00
for example, the commande below reserves 4 nodes for 1h30
:
- Automatically provision the nodes with the script
deploy.sh
indeploy
directory
./deploy.sh
This will create two files master_node
and worker_nodes
with respectively the
addresses of the machines acting as the master and the workers.
Running the program on Grid'5000 assumes the deployment described earlier.
NOTE: The program uses nfs
to share files between the nodes
Assuming the following directory structure:
- src
- celeryconfig.py
- makeparse.py
- logging.ini
- master.py
- result.py
- work.py
- Makefile
- premier.c
- master_node
- worker_nodes
- Start
celery
for allworker-node
inworker_nodes
ssh root@worker-node
celery worker -A work -linfo
- Launch the main program with:
python master.py -f Makefile premier
python master.py -h
displays the master.py
command line option. By default, running python master.py
without the -f
switch will use GNU-Makefile
or makefile
or Makefile
found in the current directory and will execute the first target in the makefile
. The -a
or --async
option allows to run asynchronously all the tasks without blocking the master
otherwise it will wait for the last task's completion.