Skip to content

Commit

Permalink
Add more notes about the buckets in the playbook
Browse files Browse the repository at this point in the history
For now, keep the info about the "old" tenant buckets for upload tests.
Despite that we will switch to use the new non-tenant buckets, these
buckets might be useful again in the future.
  • Loading branch information
hsinfang committed Feb 2, 2024
1 parent b83d8d5 commit 072d87c
Showing 1 changed file with 8 additions and 2 deletions.
10 changes: 8 additions & 2 deletions doc/playbook.rst
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,13 @@ The bucket ``rubin-pp-dev-users`` holds:

* ``rubin-pp-dev-users/unobserved/`` contains raw files that the upload script(s) can draw from to create incoming raws.

``rubin-pp-dev`` has had notifications configured for it; these publish to a Kafka topic.
``rubin-pp-dev`` has had notifications configured for it; these publish to the Kafka topic ``prompt-processing-dev``.
The notifications can be viewed at `Kafdrop <https://k8s.slac.stanford.edu/usdf-prompt-processing-dev/kafdrop/`_.
.

The buckets ``rubin:rubin-pp`` and ``rubin:rubin-pp-users`` are also for Prompt Processing development previously used by the testers.
``rubin:rubin-pp-users`` contains an older version of the development central repository.
``rubin:rubin-pp`` has notifications configured to publish to the Kafka topic ``rubin-prompt-processing``.

The default Rubin users' setup on ``rubin-devl`` includes an AWS credential file at the environment variable ``AWS_SHARED_CREDENTIALS_FILE`` and a default profile without read permission to the prompt processing buckets.
A separate credential for prompt processing developers is at `vault <https://vault.slac.stanford.edu/ui/vault/secrets/secret/show/rubin/usdf-prompt-processing-dev/s3-buckets>`_ and can be set up as another credential profile for Butler or command line tools such as AWS Command Line Interface and MinIO Client.
Expand All @@ -104,7 +110,7 @@ The AWS CLI can be used to inspect non-tenenat buckets:

You must pass the ``--endpoint-url`` argument even if you have ``S3_ENDPOINT_URL`` defined.

Some of the prompt processing buckets are Ceph tenant buckets and require a tenant prefix, which violates the bucket name standard and is not supported by AWS CLI.
If a bucket is a Ceph tenant bucket and requires a tenant prefix, the bucket name violates the standard and is not supported by AWS CLI.
The MinIO Client ``mc`` tool may be used.
One version can be accessed at ``/sdf/group/rubin/sw/bin/mc`` at USDF.
To inspect buckets with the MinIO Client ``mc`` tool, first set up an alias (e.g. ``usdf-pp``) and then can use commands:
Expand Down

0 comments on commit 072d87c

Please sign in to comment.