-
Notifications
You must be signed in to change notification settings - Fork 129
User_Using your own compute workers
In these instructions, we guide you step-by-step to create your own "compute worker" (a server to which submissions of challenge participants are sent to be executed).
- Setting up a server as compute worker
- Hooking up the server to a queue
- Assigning a queue to a competition
First you need to have access to a server connected to the Internet having an IP address and into which you can remote login via ssh. If you do not have one, you need to create one. We provide instructions for creating Ubuntu Virtual machines (VM):
Then ssh into your machine and run the following commands:
-
Get docker (quick and dirty way; for a full clean installation see Docker Ubuntu instructions):
$ curl https://get.docker.com | sudo sh -
Add yourself to the docker group (so you don't nee to run as root):
$ sudo usermod -aG docker ubuntu -
Log out of your server and log back in. Then check your installation:
$ docker run hello-world -
Run this command to be able to run many containers at once:
$ sudo apt-get install docker-compose -
Clone the Codalab Competitions docker image:
$ git clone https://github.com/codalab/codalab-competitions.git -
Create and edit the configuration file
.env:$ cd codalab-competitions$ cp .env_sample .env$ vim .env
Note: vim is an editor coming with Ubuntu. If you rather use emacs, type first sudo apt-get install emacs.
-
Uncomment in
.envthe lineBROKER_URL=and paste your worker queue key:BROKER_URL=pyamqp://61fdd17e-fb04-4296-9d30-6cd3a69c8aa8:8cc2cd6e-ce72-4592-9483-fe6b6c8b2495@13.82.145.119/030e877d-5f61-479c-8c87-7da347828287
To get your key, see next section.
-
Get your worker to start computing (it will start listening to the queue via BROKER_URL and pick up jobs):
$ docker-compose up -d worker_compute -
Make a submission to your competition and check logs with to confirm it is working:
`$ docker-compose logs -f