Skip to content

RWTH GPU Cluster stuff

Jump to bottom
SamSweere edited this page May 22, 2020 · 2 revisions

How to get Jupyter Notebook working on the Aachen cluster

Update by Sam

  • First connect to the vpn vpn.rwth-aachen.de with your user ID.
  • Connect to cluster using ssh:
ssh -l hl669674 login-g.hpc.itc.rwth-aachen.de

Where at hl669674 you put your user ID.

  • Clone the git repository in a new folder. Make sure not to use the original folder name since that will make the file-path length later on too long.
  • Go to the folder where you want to run your code.
  • Start virtual python environment:
python -m virtualenv -p python3.6 venv
  • Activate that environment:
source venv/bin/activate
  • Install Jupyter notebook:
pip install jupyterlab
  • Save the following script by using for example nano: nano jupyter_script.sh
  • CHANGE THE PATH NAME IN THE SCRIPT:
#!/usr/local_rwth/bin/zsh

#SBATCH --job-name=iwmpm-jupyterlab
#SBATCH --output=logs/output_%J.txt
#SBATCH --time=08:00:00
#SBATCH --cpus-per-task=24
#SBATCH --mem-per-cpu=4G
#SBATCH --gres=gpu:1

module switch intel gcc # This is recommended
module load cuda/100
module load cudnn/7.4

source $HOME/RP1/cleaned_src/venv/bin/activate

node=$(hostname -s)
port=8989

echo -e "
The server is $node.
The port is $port

To connect to the Jupyter instance you need to first set up a ssh tunnel from the node to your own computer.
Running this command does just that:
ssh -N -L ${port}:${node}:${port} ${USER}@login.hpc.itc.rwth-aachen.de
and log in with your password."

jupyter lab --no-browser --ip=$node --port=$port
  • Make a logs folder:
mkdir logs
  • Start a sbatch with the Jupyter notebook in it. In the previous script you can set the desired runtime. Note that setting a long runtime has the risk of having to wait for a long time until your sbatch will be executed (in our case the jupyter notebook):
sbatch jupyter_script.sh
  • Check if the sbatch is accepted and running (might have to wait a bit):
squeue -u hl669674
  • If it is running check the logs folder for the output of the scipt. This will contain the ipadress you can run in your local browser (while still being connected to the vpn) that runs Jupyter Notebook:
cat logs/<the log name>
  • Take the Jupyter Notebook link that has the IP-address in it (the other ones didn't work for me).
  • If everything went well you should see the Jupyter Notebook now! Note that the notebook will close automatically after the runtime of the sbatch has ran out (depending on what you set it to).

Disconnecting

  • If we are done we want to disconnect since we do not get unlimited runtime on the cluster.
  • To cancel an sbatch:
scancel <job id>
  • Deactivate the virtual environment:
deacactive
  • To deconect from the server:
logout

Copy files from cluster:

scp -r . hl669674@login-g.hpc.itc.rwth-aachen.de:~/TextMiningCourse/ABSA-PyTorch/

Possible scripts:

#!/usr/local_rwth/bin/zsh

#SBATCH --job-name=train-pong-iterative-agent
#SBATCH --output=logs/train_pong_%J.txt
#SBATCH --time=23:59:00
#SBATCH --cpus-per-task=4
#SBATCH --mem-per-cpu=32G
#SBATCH --gres=gpu:pascal:1

module load cuda
module load cudnn/7.4

source ../../venv/bin/activate

cd ../../WorldModels

while true; do
echo "Starting training."

python train.py --game_str "Pong" --base_dir "/hpcwork/hl669674/Pong" --file_name "Pong"
done

echo "Script completed."

### Time your job needs to execute, e. g. 2 days
#SBATCH --time=96:00:00

### Account type
#SBATCH --account=um_dke

### Memory your job needs per node
#SBATCH --mem=40G

### Use one node for parallel jobs on shared-memory systems
#SBATCH --nodes=1

### Number of threads to use
#SBATCH --cpus-per-task=12

### Number of hyperthreads per core
#SBATCH --ntasks-per-core=1

### Tasks per node (for shared-memory parallelisation, use 2)
#SBATCH --ntasks-per-node=1

### The last part consists of regular shell commands:
### Set the number of threads in your cluster environment to the value specified above
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

### Source bashrc
source ~/.bashrc

### Run train.py
### $1 is the argument for specifying the game which is used for the training
### $2 is the argument for specifying the log path
### $3 is the argument for specifying the importance of the reconstruction loss
python train.py $1 $2 $3

RWTH GPU Cluster Instructions

  • Connect Cisco AnyConnect VPN to vpn.rwth-aachen.de. User ID is vu304715. The default "webvpn" setting is fine.
  • Connect to cluster through SSH: ssh -X -l vu304715 login.hpc.itc.rwth-aachen.de
  • Optional: connect to GPU cluster to test scripts: ssh -X linuxnvc01. For some reason, the login-g node freezes if you try to access a GPU there...
  • Either submit the src/run_script.sh through bsub or execute directly on the GPU cluster.

For SLURM: ssh vu304715@login18-1.hpc.itc.rwth-aachen.de

DKE Commands:

r_batch_usage -p um_dke
bjobs -u all -q ih

For SLURM, this will be different:

sbatch <script_name>
squeue -u vu304715
scancel <job_id>

squeue --account=um_dke

Copying files

To copy data files to or from the cluster:

# FROM LOCAL TO SERVER
## Copy data to thesis repo
scp -r . vu304715@copy18-1.hpc.itc.rwth-aachen.de:~/Thesis_repo/src/data/
scp pong.zip vu304715@copy18-1.hpc.itc.rwth-aachen.de:/work/vu304715/data

### Copy files 0 - 799 to /work/data (doesn't work on windows, but I can use WinSCP then, although the speed is fucking slow (500kb instead of 4mb/s))
scp -r '(rnn_input_|rnn_output_)([0-9]|[0-9][0-9]|[0-7][0-9][0-9]).npy' vu304715@copy18-1.hpc.itc.rwth-aachen.de:/work/vu304715/data/sticky_75/

scp data/zips/catcher/bugfix.zip vu304715@copy18-1.hpc.itc.rwth-aachen.de:/work/vu304715/data/zips/catcher
scp data/zips/catcher/zero_angle.zip vu304715@copy18-1.hpc.itc.rwth-aachen.de:/work/vu304715/data/zips/catcher

### Copy weights from local to server
scp -r ./vae/final_weights.h5 'vu304715@copy18-1.hpc.itc.rwth-aachen.de:~/Thesis_repo/src/vae/'
scp -r ./rnn/final_weights.h5 'vu304715@copy18-1.hpc.itc.rwth-aachen.de:~/Thesis_repo/src/rnn/'

# Copy checkpoint files that are stored with LFS to cluster so we don't get some file corrupt error.
scp -r * 'vu304715@copy18-1.hpc.itc.rwth-aachen.de:~/Thesis_repo/src/dopamine_experiments/catcher/iterative_all_models_no_kl/checkpoints'


# FROM SERVER TO LOCAL
## Copy data from server to local machine
scp -r vu304715@copy18-1.hpc.itc.rwth-aachen.de:/work/vu304715/data/sticky_75 .
scp -r 'vu304715@copy18-1.hpc.itc.rwth-aachen.de:/work/vu304715/data/sticky_75/rnn_*' .
### Copy files 1400 - 1999
scp -r 'vu304715@copy18-1.hpc.itc.rwth-aachen.de:/work/vu304715/data/sticky_75/(rnn_input_|rnn_output_)(1[4-9][0-9][0-9]).npy' .

## Copy weights from server to local machine
scp -r 'vu304715@copy18-1.hpc.itc.rwth-aachen.de:~/Thesis_repo/src/rnn/sticky_75/' .
scp -r 'vu304715@copy18-1.hpc.itc.rwth-aachen.de:~/Thesis_repo/src/vae/sticky_75/(weights|training_history)[5-9][0-9].*' .

## Controller files and log files
scp -r vu304715@copy18-1.hpc.itc.rwth-aachen.de:~/Thesis_repo/src/controller/sticky_75_64_continued_2 ./controller
scp -r vu304715@copy18-1.hpc.itc.rwth-aachen.de:~/Thesis_repo/src/log/sticky_75_64_continued_2 ./log

scp vu304715@copy18-1.hpc.itc.rwth-aachen.de:/work/vu304715/data/zips/catcher/angle.zip ./data/zips/catcher
scp vu304715@copy18-1.hpc.itc.rwth-aachen.de:/work/vu304715/data/zips/catcher/multi.zip ./data/zips/catcher

scp -r vu304715@copy18-1.hpc.itc.rwth-aachen.de:~/Thesis_repo/src/dopamine_experiments/catcher/persistent_memory/dodge/agent_memory.zip dopamine_experiments/catcher/persistent_memory/dodge/


scp vu304715@copy18-1.hpc.itc.rwth-aachen.de:~/Thesis_repo/src/dopamine_experiments/catcher/persistent_memory/15.zip .
scp vu304715@copy18-1.hpc.itc.rwth-aachen.de:~/Thesis_repo/src/dopamine_experiments/catcher/persistent_memory/15_5/agent_memory.zip .

scp vu304715@copy18-1.hpc.itc.rwth-aachen.de:/work/vu304715/data/zips/zipped_experiments/persistent_mem_dodge_debug.zip .


scp vu304715@login18-g-1.hpc.itc.rwth-aachen.de:~/Iterative-Model-based-Reinforcement-Learning/src/vae/pong_softmax/final_weights.h5 .

Commands for my python scripts

To run the entire pipeline, execute the following in order, either on the cluster or locally:

Locally

Car Racing (discrete)

python 01_generate_data.py car_racing_discrete --total_episodes 2000 --batch_size 2000 --start_batch 0 --time_steps 1000 --sticky_actions 0.75 --data_dir ./data/car_racing/sticky_75
python 02_train_vae.py --start_batch 0 --new_model --files_per_batch 40 --data_dir ./data/car_racing/sticky_75 --weights_dir ./vae/car_racing/sticky_75
python 03_generate_rnn_data.py --start_batch 0 --max_batch 1999 --data_dir ./data/car_racing/sticky_75 --weights_dir ./vae/car_racing/sticky_75
python 04_train_rnn.py --start_batch 0 --max_batch 1999 --new_model --data_dir ./data/car_racing/sticky_75 --weights_dir ./rnn/car_racing/sticky_75

# For Deep Q-learning
python -um dopamine.discrete_domains.train --base_dir=dopamine_experiments/temp --gin_files='dopamine/agents/custom/configs/wm_car_racing_test.gin'
# For evolution
mpirun -np 17 xvfb-run -s "-screen 0 1400x900x24" python3 05_train_controller.py car_racing_discrete --num_worker 16 --num_worker_trial 2 --num_episode 4 --max_length 1000 --eval_steps 25

Catcher

# bugfix catcher
python 01_generate_data.py catcher --total_episodes 2000 --batch_size 2000 --start_batch 0 --time_steps 500 --sticky_actions 0 --data_dir ./data/catcher/bugfix
# From scratch
python 02_train_vae.py --start_batch 0 --max_batch 1999 --files_per_batch 80 --new_model --data_dir ./data/catcher/bugfix --weights_dir ./vae/catcher/bugfix

# To continue from the 11th batch (of size 120)
python 02_train_vae.py --start_batch 400 --max_batch 1999 --files_per_batch 80 --load_weights 4 --data_dir ./data/catcher/bugfix --weights_dir ./vae/catcher/bugfix

python 03_generate_rnn_data.py --start_batch 0 --max_batch 1999 --data_dir ./data/catcher/bugfix --weights_dir ./vae/catcher/bugfix

python 04_train_rnn.py --start_batch 0 --max_batch 1999 --new_model --data_dir ./data/catcher/bugfix --weights_dir ./rnn/catcher/bugfix --min_sequence_length 500

# For Deep Q-learning
python -um dopamine.discrete_domains.train --base_dir=dopamine_experiments/temp --gin_files='dopamine/agents/custom/configs/catcher_no_rnn.gin'
python -um dopamine.discrete_domains.train --base_dir=dopamine_experiments/temp --gin_files='dopamine/agents/custom/configs/catcher_visual_rnn.gin'

## For evolution
mpirun -np 4 python3 05_train_controller.py catcher --num_worker 3 --num_worker_trial 22 --num_episode 10 --optimizer pepg --sigma_init 0.3 --max_length 1000 --eval_steps 25 --eval_best 

## To visualize:
python model.py catcher --render_mode --filename ./controller/catcher/temp/catcher.cma.10.66.best_best.json
python model.py catcher --render_mode --filename dopamine_experiments/catcher/catcher_visual_rnn/weights.p

python -um dopamine.discrete_domains.evaluate --base_dir=dopamine_experiments/catcher/catcher_visual_rnn/ --gin_files='dopamine/agents/custom/configs/catcher_visual_rnn.gin' --gin_bindings='WMAgent.epsilon_eval=0' --gin_bindings='Runner.evaluation_steps=5000'


# To generate data for multiple environments (specified in config.py) and save as RNN data instead of raw observations:
python 01_generate_data.py catcher_angle --run_all_envs --use_vae ./vae/catcher/bugfix/final_weights.h5 --total_episodes 2000 --batch_size 2000 --start_batch 0 --time_steps 500 --sticky_actions 0 --data_dir ./data/catcher/angle

# Retraining RNN for a new environment
python retrain_rnn.py --env_name catcher_angle_-45 --data_dir ./data/catcher/angle --load_weights ./rnn/catcher/bugfix/final_weights.h5 --save_weights ./rnn/catcher/angle_retrained/-45 --min_sequence_length 500
# DQN for new angled environment with retrained RNN:
./scripts/catcher_angle_dqn_retrain.sh -45

# Train RNN from scratch for new environment
./scripts/catcher_angle_rnn_scratch.sh -45
# DQN for new angled environment from scratch
./scripts/catcher_angle_dqn_scratch.sh -45

# All DQN experiments( trained + scratch):
python -u run_DQN_experiments.py catcher_angle -45

# Training mixed RNN on multiple environments at once
python 04_train_rnn.py --new_model --env_prefix catcher_angle_ --experiment_suffixes "0, -45, 15, 60, -30, -75, 75" --data_dir ./data/catcher/angle --weights_dir ./rnn/catcher/multi-angle --min_sequence_length 500

Iterative agent

python -um dopamine.discrete_domains.train_iterative --base_dir=dopamine_experiments/catcher/test/ --gin_files='dopamine/agents/custom/configs/catcher_iterative.gin'

python3 -um dopamine.discrete_domains.train_iterative --base_dir=dopamine_experiments/catcher/test/ --gin_files='dopamine/agents/custom/configs/catcher_iterative_debug.gin'

python3 run_DQN_experiments.py iterative_catcher_angle_dqn "-45, 15, 60, dodge"

python -um dopamine.discrete_domains.train_iterative --base_dir=dopamine_experiments/catcher/test/ --gin_files='dopamine/agents/custom/configs/catcher_iterative_debug.gin'

python -um dopamine.discrete_domains.evaluate --base_dir=dopamine_experiments/catcher --gin_files='dopamine/agents/custom/configs/catcher_iterative_debug.gin' --gin_bindings='WMAgent.epsilon_eval=0' --gin_bindings='Runner.evaluation_steps=5000'

Atari

python3 -um dopamine.discrete_domains.train_iterative --base_dir=dopamine_experiments/atari/ --gin_files='dopamine/agents/custom/configs/atari_debug.gin'

On the cluster

bsub < scripts/generate_data_script.sh

bsub < scripts/train_vae_gpu_script.sh

bsub < scripts/generate_rnn_data_gpu_script.sh

bsub < scripts/train_rnn_gpu_script.sh

# For deep q-learning
bsub < scripts/train_controller_gpu_script.sh

# For evolution
bsub < scripts/evolve_controller_cpu_script.sh

-q and -a

Production queue (-q) seems to start around 21:00. Before that, it says jobs cannot schedule because of time window or something.

Graphical interface

If a GUI is needed (e.g. to render the box2d games and collect data), it's possible to go to https://cluster-x2.rz.rwth-aachen.de:3443 and choose XFCE. I'm not sure whether this is limited to ~10-20 minutes because it's interactive, but it probably is. For now, the easiest way to collect rendering data is therefore on my own pc, after which I can upload it to the server. However, this may be a problem if we have to train and evaluate an agent, since it will need to interact with the rendered environment for longer than 10-20 minutes. Turns out this is fairly easy to solve using the FastX client, and then forwarding the bsub with #BSUB -XF.

If the agent doesn't contain an RNN though (of maybe if it does, but it doesn't output the full history of values like in the Kung Fu Panda text sequence generator I made for ACML), it might work fine on my desktop with PlaidML.

Clone this wiki locally