Skip to content

Commit

Permalink
Add more notes about the buckets in the playbook
Browse files Browse the repository at this point in the history
For now, keep the info about the "old" dev tenant buckets.
Despite that we are switching to use the new non-tenant buckets,
these buckets might be useful again in the future.
  • Loading branch information
hsinfang committed Feb 15, 2024
1 parent 8c898ca commit 1dde00f
Showing 1 changed file with 13 additions and 3 deletions.
16 changes: 13 additions & 3 deletions doc/playbook.rst
Original file line number Diff line number Diff line change
Expand Up @@ -121,10 +121,20 @@ The bucket ``rubin-pp-dev-users`` holds:

* ``rubin-pp-dev-users/unobserved/`` contains raw files that the upload script(s) can draw from to create incoming raws.

``rubin-pp-dev`` has had notifications configured for it; these publish to a Kafka topic.
``rubin-pp-dev`` has had notifications configured for it; these publish to the Kafka topic ``prompt-processing-dev``.
The notifications can be viewed at `Kafdrop <https://k8s.slac.stanford.edu/usdf-prompt-processing-dev/kafdrop>`_.
.

The buckets ``rubin:rubin-pp`` and ``rubin:rubin-pp-users`` are also for Prompt Processing development and previously used by the testers.
``rubin:rubin-pp-users`` contains an older version of the development central repository.
``rubin:rubin-pp`` has notifications configured to publish to the Kafka topic ``rubin-prompt-processing``.

The Ceph user ``rubin-prompt-processing`` owns the ``rubin:rubin-pp`` and ``rubin:rubin-pp-users`` buckets, and the newer Ceph user ``prompt-processing-dev`` owns the ``rubin-pp-dev`` and ``rubin-pp-dev-users`` buckets.
We are in the process to deprecate the ``rubin-prompt-processing`` user as it has a more restrictive permission than ``prompt-processing-dev``.

The default Rubin users' setup on ``rubin-devl`` includes an AWS credential file at the environment variable ``AWS_SHARED_CREDENTIALS_FILE`` and a default profile without read permission to the prompt processing buckets.
A separate credential for prompt processing developers is at `vault <https://vault.slac.stanford.edu/ui/vault/secrets/secret/show/rubin/usdf-prompt-processing-dev/s3-buckets>`_ and can be set up as another credential profile for Butler or command line tools such as AWS Command Line Interface and MinIO Client.
A separate credential for prompt processing developers as the Ceph user ``prompt-processing-dev`` (new) or ``rubin-prompt-processing`` (old) is at `vault <https://vault.slac.stanford.edu/ui/vault/secrets/secret/show/rubin/usdf-prompt-processing-dev/s3-buckets>`_.
The credential can be set up as another credential profile for Butler or command line tools such as AWS Command Line Interface and MinIO Client.
One way to set up this profile is with the AWS CLI:

.. code-block:: sh
Expand All @@ -145,7 +155,7 @@ The AWS CLI can be used to inspect non-tenenat buckets:

You must pass the ``--endpoint-url`` argument even if you have ``S3_ENDPOINT_URL`` defined.

Some of the prompt processing buckets are Ceph tenant buckets and require a tenant prefix, which violates the bucket name standard and is not supported by AWS CLI.
If a bucket is a Ceph tenant bucket and requires a tenant prefix, the bucket name violates the standard and is not supported by AWS CLI.
The MinIO Client ``mc`` tool may be used.
One version can be accessed at ``/sdf/group/rubin/sw/bin/mc`` at USDF.
To inspect buckets with the MinIO Client ``mc`` tool, first set up an alias (e.g. ``usdf-pp``) and then can use commands:
Expand Down

0 comments on commit 1dde00f

Please sign in to comment.