Kopah Quickstart

⚠️ Examples use Sohny's uwnetid (mhsohny) - replace mhsohny with your uwnetid.

📑 Table of Contents

Install rclone
Configure rclone
List content on Kopah
Create your first bucket; add content
Copy data from Kopah to your cwd
Copy files and folders to Kopah
Pipe data directly into Kopah
Stream data from Kopah
Copying symlinks
Delete files and folders
List all buckets
Check bucket or dir size
Configure s3cmd
Main bucket layout conventions
Create a presigned URL
Public bucket/folder
IGV data steps
Mount directories
Space allocations

🟣 Install rclone

There is a version already available at: /usr/bin/rclone
You can also add via conda if you prefer

conda create -n rclone
conda activate rclone
conda install conda-forge::rclone

🟣 Configure rclone

Create ~/.config/rclone/rclone.conf (replace mhsohny with your uwnetid):

[k_mhsohny]
type = s3
provider = Ceph
access_key_id = <find key in ~/kopah_uwnetid>
secret_access_key = <find key in ~/kopah_uwnetid>
endpoint = https://s3.kopah.uw.edu
use_already_exists = false
compress = true

[k_stergachis_reader]
type = s3
provider = Ceph
access_key_id = <find key in ~/kopah_stergachis_reader>
secret_access_key = <find key in ~/kopah_stergachis_reader>
endpoint = https://s3.kopah.uw.edu
use_already_exists = false
compress = true

💡 k_ prefix helps you remember these are Kopah keys.
💡 You can rename k_stergachis_reader to k_sr or something else short if you prefer.

⚠️ Kopah sometimes downloads plain text/html files as gzipped files. 'compress = true' prevents that annoyance.

🟢 List content on Kopah

rclone lsf k_stergachis_reader:userprod/

Should see (at least):

data-store/
production/
web/

rclone listing commands

rclone lsf — lists files and directories at the given level (non-recursive)
rclone lsd — lists only directories at the given level (non-recursive)
rclone lsl — recursively lists all files starting at the given level, including file sizes and timestamps

💡 Adding --list-fast option helps speed up things if listing out a huge number of items.

🟢 Create your first bucket; add content

echo "hello world" > hello.txt
echo "bye world" > bye.txt
rclone copyto hello.txt k_mhsohny:mhsohny/hello.txt
rclone copyto bye.txt k_mhsohny:mhsohny/subfolder1/bye.txt
rclone lsf k_mhsohny:mhsohny

🟢 Copy data from Kopah to your cwd

rm -f hello.txt # verify nothing local
rclone copy k_mhsohny:mhsohny/hello.txt .

🟢 Copy files and folders to Kopah

Copy bam files, dereferenced symlinks to bam files, or entire folders to Kopah

rclone copy . k_mhsohny:mhsohny/data/ --include "*.bam"
rclone copy -L . k_mhsohny:mhsohny/data/dereferenced-symlinks/ --include "*.bam"
rclone copy myfolder/ k_mhsohny:mhsohny/myNewFolder/

💡 add --progress --stats=10s to end of copy command to see upload/download progress.

🟢 Pipe data directly into Kopah

You can pipe data directly to Kopah without creating a local file.

echo "hello world!" | rclone rcat k_mhsohny:mhsohny/projects/proj1/ok.txt
./my-script.sh input.bam | rclone rcat k_mhsohny:mhsohny/projects/proj2/output.bam

🟢 Stream data from Kopah

rclone cat k_mhsohny:mhsohny/projects/proj1/ok.txt # prints hello world!
rclone cat k_mhsohny:mhsohny/projects/proj2/output.bam | ft validate

🟢 Copying symlinks

Upload actual (dereferenced) files:

rclone copy -L path-to-hyak-symlink k_mhsohny:mhsohny/my-fancy-directory/

Upload as pseudo-symlinks:

rclone copy path-to-hyak-symlink k_mhsohny:mhsohny/dir-with-symlink-files/ --links

Download a directory that includes pseudo-symlinks and automatically convert them back to symlinks:

rclone copy k_mhsohny:mhsohny/dir-with-symlink-files/ . --links

🟢 Delete files and folders

Delete one file:

rclone delete k_mhsohny:mhsohny/subfolder1/bye.txt

Delete recursively:

rclone purge k_mhsohny:mhsohny/subfolder2/

🟢 List all buckets

rclone lsd k_mhsohny:

🟢 Check bucket or dir size

rclone size k_mhsohny:mhsohny/

🟣 Configure s3cmd

s3cmd is installed at /usr/bin/s3cmd
(Do not download from conda).
You must create a ~/.s3cfg file for s3cmd to work.

    [default]
    host_base = s3.kopah.uw.edu
    host_bucket = s3.kopah.uw.edu/%(bucket)
    use_https = True
    public_url_use_https = True
    # Login credentials
    access_key = <find key in ~/kopah_uwnetid>
    secret_key = <find key in ~/kopah_uwnetid>

⚠️ Remarkably, it is imperative that the comment line '# Login credentials' is present as shown

🟣 Main bucket layout conventions

Create your main bucket named after your uwnetid by copying up your first file if needed.

rclone copy myfile.txt k_mhsohny:mhsohny/

Create a policy that allows stergachis_reader to read and download data from your bucket. Save as policy.json.

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "AllowReaderListBucket",
          "Effect": "Allow",
          "Principal": { "AWS": "arn:aws:iam:::user/stergachis_reader" },
          "Action": [
            "s3:GetObject",
            "s3:ListBucket"
          ],
          "Resource": [
            "arn:aws:s3:::mhsohny",
            "arn:aws:s3:::mhsohny/*"
          ]
        }
      ]
    }

update policy:

s3cmd setpolicy policy.json s3://mhsohny
s3cmd setacl s3://mhsohny --acl-grant=read:stergachis_reader

Now others (in our lab) can read data in your bucket using k_stergachis_reader:

rclone lsf k_stergachis_reader:mhsohny

🟢 Create a presigned URL

Link to private data which is good for 1 week by default.

rclone link k_mhsohny:mhsohny/web/private/path-to-file.bam

For a whole folder:

tar -czf everything.tgz <my-directory>
rclone copy everything.tgz k_mhsohny:mhsohny/web/private/
rm everything.tgz # local
rclone link k_mhsohny:mhsohny/web/private/everything.tgz

🟣 Public bucket/folder

Kopah is private by default. You can share private data via presigned URLs as described above.
You can also make a public bucket or subfolder using an S3 bucket policy.

Here's a sample policy that allows public access to:

s3://sjn/web/public/

There is a distinction between 'get' and 'list'. In this example, I allow 'GetObject' but not 'ListBucket' to sjn/web/public/. If I send a hyperlink to a file, someone can see it in their browser. But, they cannot navigate through my directories. This json file also shows that stergachis_reader can list private content in any of my buckets/folders.

Change sjn to your uwnetid.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PublicReadGetObject",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::sjn/web/public/*"
    },
    {
      "Sid": "AllowReaderListBucket",
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam:::user/stergachis_reader" },
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::sjn",
        "arn:aws:s3:::sjn/*"
      ]
    }
  ]
}

See your current bucket policy:
s3cmd info s3://sjn

Set your bucket policy:
s3cmd setpolicy public-policy.json s3://sjn

🟣 IGV data steps

💡 You will need IGV version >= 2.19.2. Download the latest version.
💡 If you want to view an IGV session file downloaded from Prod-Reporter, you only need to do step 4 here.

Create bucket & set permissions (see above).

Upload data:

 rclone copy my-fave.bam k_mhsohny:mhsohny/igv/

Update ACLs:

 s3cmd setacl "s3://mhsohny/igv/*" --acl-grant=read:stergachis_reader

or recursively:

 s3cmd setacl "s3://mhsohny/igv/" --acl-grant=read:stergachis_reader --recursive

On your local machine, set up credentials for stergachis_reader. IGV uses 2 files in ~/.aws/ on your local machine.

a) create ~/.aws/config with the following content:
```
[default]
region = us-west-2
output = json
endpoint_url = https://s3.kopah.uw.edu
```
b) create ~/.aws/credentials with the following content:
(Two keys are the same as k_stergachis_reader in Configure rclone)
```
[default]
aws_access_key_id = <your stergachis_reader Access Key>
aws_secret_access_key = <your stergachis_reader Secret Key>
```

🟢 Mount directories

Put in your ~/.bashrc (then start a new terminal)

export PATH=${PATH}:/mmfs1/gscratch/stergachislab/bin

Mount or unmount Kopah buckets (in cwd):

kopah-mount
kopah-unmount

The default buckets are your uw-netid, userprod, and stergachis. You can pass in any bucket name(s) to override:

kopah-mount hprc stergachis
kopah-unmount hprc stergachis

🔵 Space allocations

We have 900 TB total. Allocations can change easily through UW-IT.

You can check how much allocation you have used by typing (replace mhsohny with your uwnetid):

rclone size k_mhsohny:

These are the current allocations and space used for lab-wide accounts.

User	Quota	Used
absterga	0TB/0TiB	--
asedeno	75TB/68.2TiB	--
blcs	2TB/1.8TiB	--
bmallo	2TB/1.8TiB	--
bohaczuk	2TB/1.8TiB	--
claflamm	2TB/1.8TiB	--
chiyy	2TB/1.8TiB	--
dubocd	3TB/2.7TiB	--
hprc	50TB/45.4TiB	--
mhsohny	20TB/18TiB	--
minkinaa	5TB/4.5TiB	--
mjohnso8	2TB/1.8TiB	--
mvollger	2TB/1.8TiB	--
olivecb	2TB/1.8TiB	--
pguptax	1TB/0.9TiB	--
prod-reporter	1TB/0.9TiB	--
sjn	14TB/12.7TiB	--
smaht	385TB/350TiB	250TiB
stergachis	52TB/47TiB	37TiB
swansoe	2TB/1.8TiB	--
tdreal	1TB/0.9TiB	--
userprod	250TB/227TiB	205TiB
yhhc	5TB/4.5TiB	--

Uh oh!

Kopah Quickstart

📑 Table of Contents

🟣 Install rclone

🟣 Configure rclone

🟢 List content on Kopah

rclone listing commands

🟢 Create your first bucket; add content

🟢 Copy data from Kopah to your cwd

🟢 Copy files and folders to Kopah

🟢 Pipe data directly into Kopah

🟢 Stream data from Kopah

🟢 Copying symlinks

🟢 Delete files and folders

🟢 List all buckets

🟢 Check bucket or dir size

🟣 Configure s3cmd

🟣 Main bucket layout conventions

🟢 Create a presigned URL

🟣 Public bucket/folder

🟣 IGV data steps

🟢 Mount directories

🔵 Space allocations

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally