Skip to content

Kopah Quickstart

Jump to bottom
Shane Neph edited this page Jul 11, 2025 · 117 revisions

Kopah Quickstart

⚠️ If a comment starts with # ***, it is important and requires your attention.

Kopah is private by default. You can share private data via presigned URLs as described below.
You can also make a public bucket or subfolder (not covered here yet).
Feel free to improve this document after feedback.

📑 Table of Contents

Install rclone

conda create -n rclone
conda activate rclone
conda install conda-forge::rclone

Configure rclone

Create ~/.config/rclone/rclone.conf:

[k_mhsohny]
type = s3
provider = Ceph
access_key_id = <find key in ~/kopah_uwnetid>
secret_access_key = <find key in ~/kopah_uwnetid>
endpoint = https://s3.kopah.uw.edu

[k_stergachis_reader]
type = s3
provider = Ceph
access_key_id = <find key in ~/kopah_stergachis_reader>
secret_access_key = <find key in ~/kopah_stergachis_reader>
endpoint = https://s3.kopah.uw.edu

💡 k_ prefix helps you remember these are Kopah keys.

Read from Kopah

rclone lsf k_stergachis_reader:userprod/

Should see:

data-store/
production/
web/

Other useful:

rclone lsd k_stergachis_reader:userprod/
rclone ls  k_stergachis_reader:userprod/

Create your first bucket; add content

echo "hello world" > hello.txt
echo "bye world" > bye.txt
rclone copy hello.txt k_mhsohny:mhsohny/
rclone copy bye.txt k_mhsohny:mhsohny/subfolder1/
rclone ls k_mhsohny:mhsohny

Copy data from Kopah to your cwd

rm hello.txt
rclone copy k_mhsohny:mhsohny/hello.txt .

Copy bam files to Kopah

rclone copy . k_mhsohny:mhsohny/data/ --include "*.bam"
rclone copy myfolder/ k_mhsohny:mhsohny/myfolder/

Delete files and folders

Delete one file:

rclone delete k_mhsohny:mhsohny/subfolder1/bye.txt

Delete recursively:

rclone purge k_mhsohny:mhsohny/subfolder2/

Copying symlinks

Upload actual files:

rclone copy -L path-to-klone-symlink k_mhsohny:mhsohny/my-fancy-directory/

Upload as pseudo-symlinks:

rclone copy path-to-klone-symlink k_mhsohny:mhsohny/dir-with-symlink-files/ --links

Create a presigned URL

rclone link k_mhsohny:mhsohny/web/private/path-to-file.bam

For a whole folder:

tar -czf everything.tgz <my-directory>
rclone copy everything.tgz k_mhsohny:mhsohny/web/private/
rm everything.tgz
rclone link k_mhsohny:mhsohny/web/private/everything.tgz

Main bucket layout conventions

Create your main bucket named after your uwnetid & set permissions:

rclone copy some-file k_mhsohny:mhsohny/
s3cmd setacl s3://mhsohny --acl-grant=read:stergachis_reader

Then others can read:

rclone lsf k_stergachis_reader:mhsohny

List all buckets

rclone lsd k_mhsohny:

Check bucket size

rclone size k_mhsohny:mhsohny/

IGV data steps

  1. Create bucket & set permissions (see above).

  2. Upload data:

     rclone copy my-fave.bam k_mhsohny:mhsohny/igv/

  3. Update ACLs:

     s3cmd setacl "s3://mhsohny/igv/*" --acl-grant=read:stergachis_reader

    or recursively:

     s3cmd setacl "s3://mhsohny/igv/" --acl-grant=read:stergachis_reader --recursive

Feel free to edit or extend!

Clone this wiki locally