Skip to content

AWS Cluster Setup

Jump to bottom
Sakshi Sharma edited this page Aug 7, 2026 · 2 revisions

Initial Cluster Setup

Step 1: Start ec2 instances

  • In AWS console, start ec2 instances (e.g. m5d.2xlarge) with required private IPs setup for each node

Step 2: Setup SSH keys

  • The Coordinator node should be able to ssh into all other nodes. Follow instructions: Setup SSH key.

  • Also manually add the public key to github account so we can clone the repo.

Step 3: Setup the Coordinator node

The Coordinator node requires manual setup. Follow the instructions on the Experiment Environment Setup page:

  1. Setup firewall rules
  2. Mount to local SSD (required for ec2 instances)
  3. Environment setup
  4. Install Golang

Step 4: Clone the system repo

  • Clone the system repo in Coordinator node
# Clone the git repo to under the SSD folder
cd /home/ubuntu/ssd
git clone git@github.com:CASP-Systems-BU/koala.git

Step 5: Setup all other nodes

  • Modify the /home/ubuntu/ssd/koala/scripts/aws/instance.txt file to include all nodes except the Coordinator node (current node).

  • Automatically setup all these nodes

cd /home/ubuntu/ssd/koala/scripts/aws/
python3 setup_cluster.py
  • Now your cluster is ready to use!

Stop Cluster

To stop the cluster, we should persist everything from SSD to EBS

Step 1: Persist all other nodes except Coordinator

cd /home/ubuntu/ssd/koala/scripts/aws/
python3 persist_ssd_before_stop.py

Step 2: Persist the Coordinator node

cd /home/ubuntu/
sudo rsync -a --delete --remove-source-files --exclude='lost+found' /home/ubuntu/ssd/ /home/ubuntu/ebs/ && \
sudo find /home/ubuntu/ssd -mindepth 1 -type d -empty ! -path '/home/ubuntu/ssd/lost+found' -delete && \
sync

Step 3: Stop ec2 instances

Stop all ec2 instances in the AWS console.

Resume Cluster

To resume a stopped cluster, we should re-mount SSD and move everything from EBS back to SSD.

Step 1: Restart ec2 instances

  • Restart all ec2 instances in the AWS console.

Step 2: Resume the Coordiantor node

  • Manually re-mount SSD
sudo mkfs -t ext4 -F /dev/nvme1n1
sudo mount /dev/nvme1n1 /home/ubuntu/ssd
sudo chmod o+w /home/ubuntu/ssd
sudo chown -R ubuntu /home/ubuntu/ssd
df -h
  • Move everything in EBS back to SSD
sudo rsync -a /home/ubuntu/ebs/ /home/ubuntu/ssd/

Step 3: Resume all other nodes

cd /home/ubuntu/ssd/koala/scripts/aws/
python3 recover_ssd_after_restart.py

Clone this wiki locally