Skip to content

Kopah Quickstart

Jump to bottom
Shane Neph edited this page Nov 26, 2025 · 117 revisions

⚠️ Examples use Sohny's uwnetid (mhsohny) - replace mhsohny with your uwnetid.

📑 Table of Contents

Install rclone

conda create -n rclone
conda activate rclone
conda install conda-forge::rclone

Configure rclone

Create ~/.config/rclone/rclone.conf (replace mhsohny with your uwnetid):

[k_mhsohny]
type = s3
provider = Ceph
access_key_id = <find key in ~/kopah_uwnetid>
secret_access_key = <find key in ~/kopah_uwnetid>
endpoint = https://s3.kopah.uw.edu

[k_stergachis_reader]
type = s3
provider = Ceph
access_key_id = <find key in ~/kopah_stergachis_reader>
secret_access_key = <find key in ~/kopah_stergachis_reader>
endpoint = https://s3.kopah.uw.edu

💡 k_ prefix helps you remember these are Kopah keys.
💡 You can rename k_stergachis_reader to k_sr or something else short if you prefer.

Read from Kopah

rclone lsf k_stergachis_reader:userprod/

Should see:

data-store/
production/
web/

rclone listing commands

  • rclone lsf — lists files and directories at the given level (non-recursive)
  • rclone lsd — lists only directories at the given level (non-recursive)
  • rclone ls — recursively lists all files starting at the given level, including file sizes

Create your first bucket; add content

echo "hello world" > hello.txt
echo "bye world" > bye.txt
rclone copy hello.txt k_mhsohny:mhsohny/
rclone copy bye.txt k_mhsohny:mhsohny/subfolder1/
rclone ls k_mhsohny:mhsohny

Copy data from Kopah to your cwd

rm hello.txt
rclone copy k_mhsohny:mhsohny/hello.txt .

Copy bam files to Kopah

rclone copy . k_mhsohny:mhsohny/data/ --include "*.bam"
rclone copy -L . k_mhsohny:mhsohny/data/dereferenced-symlinks/ --include "*.bam"
rclone copy myfolder/ k_mhsohny:mhsohny/myNewFolder/

💡 add --progress --stats=10s to end of copy command to see upload/download progress.

Copying symlinks

Upload actual files:

rclone copy -L path-to-klone-symlink k_mhsohny:mhsohny/my-fancy-directory/

Upload as pseudo-symlinks:

rclone copy path-to-klone-symlink k_mhsohny:mhsohny/dir-with-symlink-files/ --links

Delete files and folders

Delete one file:

rclone delete k_mhsohny:mhsohny/subfolder1/bye.txt

Delete recursively:

rclone purge k_mhsohny:mhsohny/subfolder2/

List all buckets

rclone lsd k_mhsohny:

Check bucket size

rclone size k_mhsohny:mhsohny/

Create a presigned URL

rclone link k_mhsohny:mhsohny/web/private/path-to-file.bam

For a whole folder:

tar -czf everything.tgz <my-directory>
rclone copy everything.tgz k_mhsohny:mhsohny/web/private/
rm everything.tgz
rclone link k_mhsohny:mhsohny/web/private/everything.tgz

Configure s3cmd

  • s3cmd is installed at /usr/bin/s3cmd
    (Do not download from conda).

  • You must create a ~/.s3cfg file for s3cmd to work.

    [default]
    host_base = s3.kopah.uw.edu
    host_bucket = s3.kopah.uw.edu/%(bucket)
    use_https = True
    public_url_use_https = True
    # Login credentials
    access_key = <find key in ~/kopah_uwnetid>
    secret_key = <find key in ~/kopah_uwnetid>

⚠️ Remarkably, it is imperative that the comment line '# Login credentials' is present as shown

Public bucket/folder

Kopah is private by default. You can share private data via presigned URLs as described above.
You can also make a public bucket or subfolder using an S3 bucket policy.

Here's a sample policy that allows public access to:

  • s3://sjn/web/public/

There is a distinction between 'get' and 'list'. In this example, I allow 'GetObject' but not 'ListBucket' to sjn/web/public/. If I send a hyperlink to a file, someone can see it in their browser. But, they cannot navigate through my directories. This json file also shows that stergachis_reader can list private content in sjn/web/private/.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PublicReadGetObject",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::sjn/web/public/*"
    },
    {
      "Sid": "AllowListForPrivate",
      "Effect": "Allow",
      "Principal": {
        "AWS": "stergachis_reader"
      },
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::sjn",
      "Condition": {
        "StringLike": {
          "s3:prefix": "web/private/*"
        }
      }
    }
  ]
}

See your current bucket policy:
s3cmd info s3://mhsohny

Set your bucket policy:
s3cmd setpolicy public-policy.json s3://mhsohny

Main bucket layout conventions

Create your main bucket named after your uwnetid by copying up your first file if needed.

rclone copy myfile.txt k_mhsohny:mhsohny/

Create a policy that allows stergachis_reader to read and download data from your bucket. Save as policy.json.

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "AllowReaderListBucket",
          "Effect": "Allow",
          "Principal": { "AWS": "arn:aws:iam:::user/stergachis_reader" },
          "Action": [
            "s3:GetObject",
            "s3:ListBucket"
          ],
          "Resource": [
            "arn:aws:s3:::mhsohny",
            "arn:aws:s3:::mhsohny/*"
          ]
        }
      ]
    }

update policy:

s3cmd setpolicy policy.json s3://mhsohny
s3cmd setacl s3://mhsohny --acl-grant=read:stergachis_reader

Now others can read data in your bucket using k_stergachis_reader:

rclone lsf k_stergachis_reader:mhsohny

IGV data steps

  1. Create bucket & set permissions (see above).

  2. Upload data:

     rclone copy my-fave.bam k_mhsohny:mhsohny/igv/

  3. Update ACLs:

     s3cmd setacl "s3://mhsohny/igv/*" --acl-grant=read:stergachis_reader

    or recursively:

     s3cmd setacl "s3://mhsohny/igv/" --acl-grant=read:stergachis_reader --recursive

  4. On your local machine, set up credentials for stergachis_reader as your [default] in ~/.aws/credentials.

Mount directories

Modify variables $buckets and $reader near the top of this script and mount (read-only) directories that you want to access like they live on Klone

#!/bin/tcsh -ef
# author : sjn

set buckets = (sjn smaht stergachis userprod)
set reader = k_stergachis_reader

set opts = "[-u|--unmount]"
set usage = `echo "$0:t $opts"`

@ do_unmount = 0
foreach param (`seq 1 $#argv`)
  set val = $argv[$param]
  if ( "--help" == "$val" ) then
    echo "$usage"
    exit 0
  else if ( "-u" == "$val" || "--unmount" == "$val" ) then
    @ do_unmount = 1
  else
    printf "Unknown option %s\n\n" $val
    echo "$usage"
    exit -1
  endif
end

set here = `pwd`

foreach b ($buckets)
  set target = "$here/mnt.$b"
  mkdir -p $target
  @ is_mounted = `mount | awk -v path="$target" '$0 ~ path { found=1 } END { print found }'`

  if ( $do_unmount == 0 ) then # mount
    if ( $is_mounted != 0 ) then
      echo "Already mounted: $here/mnt.$b"
    else
      rclone mount $reader":"$b $target \
        --read-only \
        --daemon \
        --vfs-cache-mode=off \
        --dir-cache-time=5m \
        --attr-timeout=60s \
        --log-file=$here/rclone.mnt.$b.log \
        --log-level=INFO
    endif
  else # do_unmount
    if ( $is_mounted != 0 ) then
      fusermount -u $target
    endif
  endif
end

exit 0

Space allocations

We have 600 TB total. Allocations can change easily through UW-IT.

These are the current allocations and space used for lab-wide accounts.

User Quota Used
absterga 2TB/1.8TiB --
asedeno 10TB/9TiB --
bmallo 5TB/4.5TiB --
bohaczuk 8TB/7.2TiB --
dubocd 5TB/4.5TiB --
mhsohny 10TB/9TiB --
minkinaa 5TB/4.5TiB --
mvollger 1TB/0.9TiB --
olivecb 5TB/4.5TiB --
sjn 5TB/4.5TiB --
smaht 180TB/163TiB 27TiB
stergachislab 97TB/88TiB 70TiB
swansoe 5TB/4.5TiB --
userprod 257TB/233TiB 121TiB
yhhc 5TB/4.5TiB --

Clone this wiki locally