Skip to content

SMaHT Kopah Quickstart

Jump to bottom
Shane Neph edited this page Dec 2, 2025 · 36 revisions

📑 Table of Contents

Install rclone

conda create -n rclone
conda activate rclone
conda install conda-forge::rclone

Configure rclone

Create ~/.config/rclone/rclone.conf

[k_smaht]
type = s3
provider = Ceph
access_key_id = <find key in ~/kopah_uwnetid>
secret_access_key = <find key in ~/kopah_uwnetid>
endpoint = https://s3.kopah.uw.edu

[k_smaht_reader]
type = s3
provider = Ceph
access_key_id = <find key in ~/kopah_smaht_reader>
secret_access_key = <find key in ~/kopah_smaht_reader>
endpoint = https://s3.kopah.uw.edu

💡 k_ prefix helps you remember these are Kopah keys.
💡 You can rename k_smaht_reader to k_sr or something else short if you prefer.

List content on Kopah under bucket 'smaht'

rclone lsf k_smaht:smaht/

rclone listing commands

  • rclone lsf — lists files and directories at the given level (non-recursive)
  • rclone lsd — lists only directories at the given level (non-recursive)
  • rclone ls — recursively lists all files starting at the given level, including file sizes

Copy up your first content; create your first folder (using yjkwon5 as our example person)

echo "hello world" > hello.txt
rclone copy hello.txt k_smaht:smaht/analysis/yjkwon5/
rclone ls k_smaht:smaht/analysis/yjkwon5

Copy data from Kopah to your cwd

rm hello.txt # verify nothing local
rclone copy k_smaht:smaht/analysis/yjkwon5 .

Copy files and folders to Kopah

Copy bam files, dereferenced symlinks to bam files, or entire folders to Kopah

rclone copy . k_smaht:smaht/analysis/yjkwon5/proj1/ --include "*.bam"
rclone copy -L . k_smaht:smaht/analysis/yjkwon5/proj2/dereferenced-symlinks/ --include "*.bam"
rclone copy myfolder/ k_smaht:smaht/analysis/yjkwon5/proj3/

💡 add --progress --stats=10s to end of copy command to see upload/download progress.

Pipe data directly into Kopah

You can pipe data directly to Kopah without creating a local file.

echo "hello world!" | rclone rcat k_smaht:smaht/analysis/yjkwon5/proj1/ok.txt
./my-script.sh input.bam | rclone rcat k_smaht:smaht/analysis/yjkwon5/proj2/output.bam

Stream data from Kopah

rclone cat k_smaht:smaht/analysis/yjkwon5/proj1/ok.txt # prints hello world!
rclone cat k_smaht:smaht/analysis/yjkwon5/proj2/output.bam | ft validate

Copying symlinks

Upload actual files:

rclone copy -L path-to-hyak-symlink k_smaht:smaht/analysis/yjkwon5/my-fancy-directory/

Upload as pseudo-symlinks:

rclone copy path-to-hyak-symlink k_smaht:smaht/analysis/yjkwon5/dir-with-symlink-files/ --links

Delete files and folders

Delete one file:

rclone delete k_smaht:smaht/analysis/yjkwon5/subfolder1/bye.txt

Delete recursively:

rclone purge k_smaht:smaht/analysis/yjkwon5/subfolder2

List all buckets

rclone lsd k_smaht:

Check bucket size

rclone size k_smaht:smaht/

Create a presigned URL

rclone link k_smaht:smaht/data/hic/SMaHT_001/lib_208871_sample_1676947_S1_L008_R2_001.fastq.gz

For a whole folder:

tar -czf everything.tgz <my-directory>
rclone copy everything.tgz k_smaht:smaht/analysis/yjkwon5/proj1/
rm everything.tgz # local
rclone link k_smaht:smaht/analysis/yjkwon5/proj1/everything.tgz

Mount directories

Run this script to mount read-only smaht bucket and subfolders on hyak. unmount bucket by running again with -u option.

#!/bin/tcsh -ef
# author : sjn

set buckets = (smaht)
set reader = k_smaht

set opts = "[-u|--unmount]"
set usage = `echo "$0:t $opts"`

@ do_unmount = 0
foreach param (`seq 1 $#argv`)
  set val = $argv[$param]
  if ( "--help" == "$val" ) then
    echo "$usage"
    exit 0
  else if ( "-u" == "$val" || "--unmount" == "$val" ) then
    @ do_unmount = 1
  else
    printf "Unknown option %s\n\n" $val
    echo "$usage"
    exit -1
  endif
end

set here = `pwd`

foreach b ($buckets)
  set target = "$here/mnt.$b"
  mkdir -p $target
  @ is_mounted = `mount | awk -v path="$target" '$0 ~ path { found=1 } END { print found }'`

  if ( $do_unmount == 0 ) then # mount
    if ( $is_mounted != 0 ) then
      echo "Already mounted: $here/mnt.$b"
    else
      rclone mount $reader":"$b $target \
        --read-only \
        --daemon \
        --vfs-cache-mode=off \
        --dir-cache-time=5m \
        --attr-timeout=60s \
        --log-file=$here/rclone.mnt.$b.log \
        --log-level=INFO
    endif
  else # do_unmount
    if ( $is_mounted != 0 ) then
      fusermount -u $target
    endif
  endif
end

exit 0

Space allocations

We have 180 TB total which can increase.

User Quota Used
smaht 180TB/163TiB 57TiB

Clone this wiki locally