-
Notifications
You must be signed in to change notification settings - Fork 2
Compute Nodes
Training neural networks requires a huge amount of resources (GPU, CPU, RAM, etc.) and your local computer might not be enough for your heavy tasks. We have several machines that you can access with high quality hardware for your experimentations. This section will provide information on the machine we own, and how to work with them.
Here is a list of the servers owned by Jean-François Lalonde at LVSN. Note that to have access you need to contact the IT service (aide-vision@gel.ulaval.ca) with JF in cc.
Currently, synchronising the processes between the students is simply done by communicating directly. If you need a resource (e.g. GPU), do not hesitate to ask on Slack (e.g. #jfgroup channel).
| Name | CPU Core | RAM | GPUs : RAM | CUDA |
|---|---|---|---|---|
| rachmaninoff | 12 | 64G | Tesla K40c : 12 G Tesla K40c : 12 G |
7.5 |
| mendelssohn | 20 | 126G | GeForce GTX TITAN X : 12 G Titan X (Pascal) : 12G |
10.0 |
| schubert | 12 | 126G | Titan X (Pascal) : 12 G Titan X (Pascal) : 12 G Titan Xp : 12 Gb |
10.0 |
| tchaikovsky | 24 | 126G | Titan V : 12G Titan RTX : 24G |
10.1 |
| liszt | 24 | 126G | Titan V : 12G Titan RTX : 24G |
10.1 |
Also there is no way to manage memory usage on the remote machines, so please be considerate of the space you are using as other might need the space for their experimentations. We recommend to use SSD memory only on a short period of time to accelerate your experiments, do not keep data in there, the data paths are there for this reason. Note that if space is needed, ask on Slack (e.g. #jfgroup). Note: the data on these machines are NOT BACKED UP.
| Name | SSD path : Total space | Data paths : Total space |
|---|---|---|
| rachmaninoff | /home/local : 239 G | /gel/rachmaninoff/data : 11T /home/debussy/sibelius : 5.8T /home/debussy/ravel : 16T |
| mendelssohn | /home-local : 235 G | /home-local2 : 1.8T |
| schubert | /home : 440 G | /home-local : 1T /home-local2 : 4.5T |
| tchaikovsky | /home-local : 325 G | /home-local2 : 3.7T |
| liszt | /home-local : 325 G | /home-local2 : 3.7T |
This is Marc-Andre's strategy to avoid blowing things up:
- My code is on the SSD (a few MB)
- The main dataset is on the SSD (about 10 GB). I only read it, so I cannot possibly fill the SSD up by accident. On the other hand, it is indeed faster when I train.
- Whatever I write (logs, checkpoints, intermediate results, etc.) goes on the HDD, the rationale being that here I can mess up something and just write hundreds of GBs of data by accident. If this is the case, I would rather have it done on the HDD, because 1) it takes much longer to fill so I have higher chances to uncover the problem before and 2) it should not prevent others to run or synchronize their code.
TL;DR: whatever I read, I put it on the SSD (unless unreasonably large). Whatever I write, I put it on the HDD. I call it minimizing the surprise.
Rachmaninoff is used as a webserver. If you want to expose files to the world, you can create a soft link of your files to the path /home/local/server_public/skymangler/imgstorage and then access it with a link such as http://rachmaninoff.gel.ulaval.ca/static/your_file_name
The following sections is a list of tools useful when working on remote machines. Note that this guide is just to make you aware of these tools, read their documentation for more information.
To log on a machine you may use ssh (change for yours, and for one of the machine name):
ssh <idul>@<servername>.gel.ulaval.ca
If you are outside of ulaval's network you can use the university's vpn or tunnel through pika.gel.ulaval.ca.
ssh -t <idul>@pika.gel.ulaval.ca ssh <idul>@<servername>.gel.ulaval.ca
Note that you may have to setup an ssh key.
Useful when you want to do a plain copy of your file/folder:
scp <source> <destination>
example usage to copy from your local computer to a server:
scp /your/local/file.txt <idul>@<servername>.gel.ulaval.ca:/the/server/path
A more powerful tool is rsync that transfer only the changes in the files. Ideal if you just want to sync a few changed files without recopying everything.
rsync <options> <source> <destination>
example usage to sync a folder to the remote machine (many options are possible, read the doc!):
rsync -avzh /your/local/folder <idul>@<servername>.gel.ulaval.ca:/path/to/folder
Docker is probably the most important software to know in this document, so we dedicated its own page. Docker can be seen as a fast and efficient way of deploying virtual machines. Docker is fast because it does not virtualize the hardware but only the OS (info). Here we use it mainly to isolate our project dependencies, it gives great freedom over what library you can install and makes your experiment easier to setup and use in the future.
If you ssh in a remote machine, you might need multiple terminals, or want to keep the terminal alive even if you lose your connection. You can use tmux:
Create a session :
tmux new -s <session-name>
Use ctrl-b + c for a new sub terminal and ctrl-b + n to navigate between them. ctrl-d or ctrl-x will kill the sub-terminal.
If you have a long process to execute (e.g. uncompress a large database) you might want to shut your connection. To keep your terminal (and its process) running, you can detach from your tmux session with ctrl-b + d and reattach to it with:
tmux a -t <session-name>
You can list active session with
tmux ls
Most of the cases, we don't have root access to these computers. If you want to install libraries under your folder, forget apt-get, you will need to compile them locally.
In the repo, we provide some scripts to install python, tmux, openexr, etc.
- Inspired by Gabriel Leclerc's documentation.