Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update object storage access #23

Merged
merged 4 commits into from
Oct 10, 2022
Merged

Update object storage access #23

Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
bd43b99
update object storage access
sebastian-luna-valero Sep 27, 2022
c6f0fb4
amend OS_STORAGE_URL
sebastian-luna-valero Sep 28, 2022
00682a8
explain why we use openstack commands and another project
guillaumeeb Oct 7, 2022
9965a78
Update link to fedcloud
guillaumeeb Oct 7, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 32 additions & 22 deletions EGI-CLI-Swift-S3.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Using fedcloud and openstack clients
# Using fedcloud and openstack clients to access CESNET object storage

## First install required Python packages

Expand Down Expand Up @@ -27,47 +27,57 @@ fedcloud token check

Last command should return a valid token.

## Use Openstack
## Use Openstack without fedcloudclient to access a specific project storage

It is possible to use `openstack` command through `fedcloudclient` to interact with
Openstack object storage, as [documented here](https://docs.egi.eu/users/data/storage/object-storage/#access-via-rclone).
guillaumeeb marked this conversation as resolved.
Show resolved Hide resolved

However, this works well when there is only one project associated with a
Virtual Organization (i.e. a one to one mapping between the two).
We currently have one Virtual Organization (vo.pangeo.eu) and two OpenStack projects associated with it.
We have created _vo.pangeo.eu-swift_, a new, separate OpenStack project to allow normal users
(i.e. non admin users) of the _vo.pangeo.eu_ VO to work with an object store.
This way, we should be able to set more fine grained authorizations on buckets.
That's why we need to use `openstack` commands in replacement of the `fedcloudclient`.

Please configure these environment variables:
```
export OS_AUTH_URL=https://identity.cloud.muni.cz/v3
export OS_AUTH_TYPE=v3oidcaccesstoken
export OS_PROTOCOL=openid
export OS_IDENTITY_PROVIDER=egi.eu
export OS_ACCESS_TOKEN=$OIDC_ACCESS_TOKEN
export OS_PROJECT_ID=57102d3e06b7476088fe4924370ae170
export OS_STORAGE_URL=https://object-store.cloud.muni.cz/swift/v1
```

Then the following command should work:
```
fedcloud openstack --site CESNET-MCC --vo vo.pangeo.eu container list
openstack container list
guillaumeeb marked this conversation as resolved.
Show resolved Hide resolved
```

## Retrieve Openstack Swift credentials
## Retrieve Openstack token for Swift

```
# get OS_AUTH_URL
$ fedcloud openstack --site CESNET-MCC --vo vo.pangeo.eu catalog show keystone

# get OS_AUTH_TOKEN
$ fedcloud openstack --site CESNET-MCC --vo vo.pangeo.eu token issue \
-c id \
-f value

# get OS_STORAGE_URL for your site and Virtual Organisation
$ fedcloud openstack --site CESNET-MCC --vo vo.pangeo.eu catalog show swift
$ openstack token issue -c id -f value
```

You'll need OS_AUTH_TOKEN and OS_STORAGE_URL in order to interact with Swift using Zarr.

OS_STORAGE_URL is actually always https://object-store.cloud.muni.cz/swift/v1 for CESNET. So what you really need above is OS_AUTH_TOKEN.

You'll need `OS_AUTH_TOKEN` and `OS_STORAGE_URL` in order to interact with Swift using Zarr.

## Retrieve S3 credentials

CESNET provides the following self-service to get S3 credentials:
https://docs.cloud.muni.cz/cloud/advanced-features/#s3-credentials.

Using `fedcloudclient` you can do:
Please run:
```
fedcloud openstack --site CESNET-MCC --vo vo.pangeo.eu ec2 credentials create
fedcloud openstack --site CESNET-MCC --vo vo.pangeo.eu ec2 credentials list
openstack ec2 credentials create
openstack ec2 credentials list
```

Once you've created a credential, you can retrieve it with the `list` command. Do not use create if you already have one.

This will provide `access` and `secret` keys. The `endpoint` URL is: https://object-store.cloud.muni.cz/.
This will provide `access` and `secret` keys. The `endpoint` URL is: `https://object-store.cloud.muni.cz/`.

__Be really careful of what you do with your credentials, e.g. avoid living them into notebooks.__
__Be really careful of what you do with your credentials, e.g. avoid living them into notebooks.__