HPC

HPC Account Creation and Troubleshooting

Instructions for Requesting a New HPC Account

• Go to The Spot (https://thespot.mskcc.org/esc/) and open a new ticket under: "Services by Department > IT > Research Applications and Services > High Performance Computing (HPC) > HPC Account Change Request"
  • Request type is "New Account"
  • Cost center is 50661 for Chan Lab
  • Lab Admin is Christina Marciante
  • Save your ticket id (something like RITM0305932) for later
• If you have not already done so, request a MSKCC Slack account through The Spot under: "Services by Department > IT > Research * Applications and Services > Research Computing > Request to Create a Box, Slack or GSUITE account."
  • This is processed automatically, typically within a few minutes.
• On the MSKCC Slack, Message Alejandro Krauskopf and message your HPC Account Change Request ticket id to expedite approval
• If you do not have an MSK e-mail address or access to VPN, contact Christina, who can set you up with work-around access to HPC.

HPC Debugging and Troubleshooting

The following channels exist on the MSKCC Slack workspace to deal with issues relating to HPC services and are the fastest modes of communication with the HPC team:

• hpc_mskcc: Support for all things HPC.
• hpc-ondemand: Support for any On Demand service provided by HPC (e.g. Lilac On Demand).

Iris

SSH

Enter the following in your command line interface:

ssh <MSKusername>@islogin01.mskcc.org

When prompted, enter your MSK password to log in to the cluster.

Storage

• Home Directory: Located at /home/$USER, this directory has a limit of 100GB.
• Compute Storage: Located at /data1/chanj3, the current quota for the lab is set at 20TB.
• Software Storage: Located at /usersoftware/chanj3, your lab has 100GB space allocated.
• Scratch Space: Located at /scratch/chanj3, your lab has 1TB of Flash storage allocated.

Data transfer

This is the data transfer node which you can utilize for copying and transferring files: isxfer01.mskcc.org

Open OnDemand

For any applications that need a graphical interface, we highly recommend using our web-based tool Open OnDemand via this link: https://isvood001.mskcc.org/pun/sys/dashboard; this allows users to launch different applications within an interactive job on the cluster.

User Documentation

For more detailed information about using the cluster, please feel free to refer to the user documentation attached here

Transfer data between Lilac and Iris

Generate SSH key pair

To transfer data from Lilac to Iris, you'd first need to generate an SSH key pair in your home directory on Iris itself, and then add that newly created public key to your authorized_keys file on Lilac.

• First on the Iris transfer node:

ssh <MSKusername>@isxfer01.mskcc.org
ssh-keygen

Saving it under the default location under /home/<MSKusername>/.ssh/id_rsa should be ok, or you could rename it as desired.

• Copy the generated public key. On the Lilac transfer node

ssh <MSKusername>@lilac-xfer01.mskcc.org
vim /home/<MSKusername>/.ssh/authorized_keys

And paste the public key output from before and save

• You should then be able to ssh from isxfer01 to lilac-xfer01

ssh <MSKusername>@isxfer01.mskcc.org

• And once logged in to isxfer01 by

ssh <MSKusername>@lilac-xfer01.mskcc.org

You should in theory be good to go to initiate the rsync!

rysnc

Doing any transfers within a screen session on isxfer01. I also recommend logging the verbose output of the rsync - you could create a "logs" folder or something similar under your home directory or /data1/chanj3.

ssh <MSKusername>@isxfer01.mskcc.org
screen -S rsync_lilac_to_iris

Screen session should then open

rsync -avLP <MSKusername>@lilac-xfer01:/path/to/source/data/in/lilac /path/to/destination/in/iris > /home/<MSKusername>/logs/rsync_lilac_to_iris.log

Then pressing ctrl+A+D to detach from the screen session, and tail -f /home//logs/rsync_lilac_to_iris.log to inspect the progress of the rsync.