Skip to content

SMaHT Kopah Quickstart

Jump to bottom
Shane Neph edited this page Dec 15, 2025 · 36 revisions

⚠️ Examples use Youngjun Kwon's uwnetid (yjkwon5) - replace yjkwon5 with your uwnetid.

📑 Table of Contents

🟣 Working effectively on hyak

  1. Your home directory is tiny by today's standards, and you will run into problems. Take the time to set up a real workspace by following these instructions. but replace the directory shown in: mkdir -p /mmfs1/gscratch/stergachislab/${USER} with our smaht directory: mkdir -p /mmfs1/gscratch/smaht/${USER}.

  2. Install conda (miniconda3). Follow the installer instructions for Linux. When you select the install location, choose somewhere other than your home directory (the default). I would choose: /mmfs1/gscratch/smaht/${USER}/tools/miniconda3

🟣 Install rclone

conda create -n rclone
conda activate rclone
conda install conda-forge::rclone

🟣 Configure rclone

Create ~/.config/rclone/rclone.conf

[k_smaht]
type = s3
provider = Ceph
access_key_id = <find key in ~/kopah_smaht>
secret_access_key = <find key in ~/kopah_smaht>
endpoint = https://s3.kopah.uw.edu

[k_smaht_reader]
type = s3
provider = Ceph
access_key_id = <find key in ~/kopah_smaht_reader>
secret_access_key = <find key in ~/kopah_smaht_reader>
endpoint = https://s3.kopah.uw.edu

💡 k_ prefix helps you remember these are Kopah keys.
💡 You can rename k_smaht_reader to k_sr or something else short if you prefer.
💡 You may not yet have ~/kopah_smaht_reader but we can get those for you.

🔵 Interface to download Kopah data

Find and download data: https://s3.kopah.uw.edu/smaht/index.html

🟢 List content on Kopah

rclone lsf k_smaht:smaht/

rclone listing commands

  • rclone lsf — lists files and directories at the given level (non-recursive)
  • rclone lsd — lists only directories at the given level (non-recursive)
  • rclone ls — recursively lists all files starting at the given level, including file sizes

🟢 Create your first bucket; add content

(using yjkwon5 as our example person)

echo "hello world" > hello.txt
rclone copy hello.txt k_smaht:smaht/analysis/yjkwon5/
rclone ls k_smaht:smaht/analysis/yjkwon5

🟢 Copy data from Kopah to your cwd

rm -f hello.txt # verify nothing local
rclone copy k_smaht:smaht/analysis/yjkwon5/hello.txt .

🟢 Copy files and folders to Kopah

Copy bam files, dereferenced symlinks to bam files, or entire folders to Kopah

rclone copy . k_smaht:smaht/analysis/yjkwon5/proj1/ --include "*.bam"
rclone copy -L . k_smaht:smaht/analysis/yjkwon5/proj2/dereferenced-symlinks/ --include "*.bam"
rclone copy myfolder/ k_smaht:smaht/analysis/yjkwon5/proj3/

💡 add --progress --stats=10s to end of copy command to see upload/download progress.

🟢 Pipe data directly into Kopah

You can pipe data directly to Kopah without creating a local file.

echo "hello world!" | rclone rcat k_smaht:smaht/analysis/yjkwon5/proj1/ok.txt
./my-script.sh input.bam | rclone rcat k_smaht:smaht/analysis/yjkwon5/proj2/output.bam

🟢 Stream data from Kopah

rclone cat k_smaht:smaht/analysis/yjkwon5/proj1/ok.txt # prints hello world!
rclone cat k_smaht:smaht/analysis/yjkwon5/proj2/output.bam | ft validate

🟢 Copying symlinks

Upload actual (dereferenced) files:

rclone copy -L path-to-hyak-symlink k_smaht:smaht/analysis/yjkwon5/my-fancy-directory/

Upload as pseudo-symlinks:

rclone copy path-to-hyak-symlink k_smaht:smaht/analysis/yjkwon5/dir-with-symlink-files/ --links

🟢 Delete files and folders

Delete one file:

rclone delete k_smaht:smaht/analysis/yjkwon5/subfolder1/bye.txt

Delete recursively:

rclone purge k_smaht:smaht/analysis/yjkwon5/subfolder2

🟢 List all buckets

rclone lsd k_smaht:

🟢 Check bucket or dir size

rclone size k_smaht:smaht/
rclone size k_smaht:smaht/analysis/yjkwon5/

🟢 Create a presigned URL

Create a hyperlink to private data that expires:

rclone link k_smaht:smaht/data/hic/SMaHT_001/lib_208871_sample_1676947_S1_L008_R2_001.fastq.gz

For a whole folder:

tar -czf everything.tgz <my-directory>
rclone copy everything.tgz k_smaht:smaht/analysis/yjkwon5/proj1/
rm everything.tgz # local
rclone link k_smaht:smaht/analysis/yjkwon5/proj1/everything.tgz

🟢 Mount directories

Put in your ~/.bashrc (then start a new terminal)

export PATH=${PATH}:/mmfs1/gscratch/smaht/bin

Mount or unmount Kopah smaht bucket (in cwd):

mount-kopah
unmount-kopah

You can add a directory to specify where to mount/unmount:

mount-kopah proj5
unmount-kopah proj5

🔵 Space allocations

Capacity can increase.

User Quota Used
smaht 230TB/209TiB 125TiB

Clone this wiki locally