Skip to content

DM-41591: Include a visit with no template coverage in the upload.py test #108

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Dec 13, 2023
Merged

DM-41591: Include a visit with no template coverage in the upload.py test #108

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
471f72b
Update playbook on USDF S3 buckets
hsinfang Oct 18, 2023
6f2ff63
Update playbook on the unobserved data for testers
hsinfang Dec 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 15 additions & 6 deletions doc/playbook.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,8 @@ This is the only situation in which a change to ``BASE_TAG_LIST`` should be comm
Buckets
=======

`This document <https://confluence.lsstcorp.org/display/LSSTOps/USDF+S3+Bucket+Organization>`_ describes the overall organization of S3 buckets and access at USDF.

The bucket ``rubin:rubin-pp`` holds incoming raw images.

The bucket ``rubin:rubin-pp-users`` holds:
Expand All @@ -80,7 +82,7 @@ The bucket ``rubin:rubin-pp-users`` holds:
``rubin:rubin-pp`` has had notifications configured for it; these publish to a Kafka topic.

The default Rubin users' setup on ``rubin-devl`` includes an AWS credential file at the environment variable ``AWS_SHARED_CREDENTIALS_FILE`` and a default profile without read permission to the prompt processing buckets.
A separate credential for prompt processing developers is at `vault <https://vault.slac.stanford.edu/ui/vault/secrets/secret/show/rubin/usdf-prompt-processing-dev/s3-buckets>`_ and can be set up as another credential profile for Butler or AWS Command Line Interface.
A separate credential for prompt processing developers is at `vault <https://vault.slac.stanford.edu/ui/vault/secrets/secret/show/rubin/usdf-prompt-processing-dev/s3-buckets>`_ and can be set up as another credential profile for Butler or command line tools such as AWS Command Line Interface and MinIO Client.
One way to set up this profile is with the AWS CLI:

.. code-block:: sh
Expand All @@ -101,8 +103,10 @@ The AWS CLI can be used to inspect non-tenenat buckets:

You must pass the ``--endpoint-url`` argument even if you have ``S3_ENDPOINT_URL`` defined.

The prompt processing buckets are Ceph tenant buckets and require a tenant prefix.
To inspect them with the MinIO Client ``mc`` tool, first set up an alias (e.g. ``usdf-pp``) and then can use commands:
Some of the prompt processing buckets are Ceph tenant buckets and require a tenant prefix, which violates the bucket name standard and is not supported by AWS CLI.
The MinIO Client ``mc`` tool may be used.
One version can be accessed at ``/sdf/group/rubin/sw/bin/mc`` at USDF.
To inspect buckets with the MinIO Client ``mc`` tool, first set up an alias (e.g. ``usdf-pp``) and then can use commands:

.. code-block:: sh

Expand Down Expand Up @@ -236,11 +240,16 @@ Sample command line:
.. code-block:: sh

python upload.py HSC 3
python upload.py LATISS 2
python upload.py LATISS 3

This script draws images stored in the ``rubin:rubin-pp-users`` bucket.
For HSC, 4 groups, in total 10 raw files, are curated.
For LATISS, 2 groups, in total 2 raw fits files and their corresponding json metadata files, are curated.

* For HSC, 4 groups, in total 10 raw files, are curated.
They are the COSMOS data as curated in `ap_verify_ci_cosmos_pdr2 <Rhttps://github.com/lsst/ap_verify_ci_cosmos_pdr2>`_.
* For LATISS, 3 groups, in total 3 raw fits files and their corresponding json metadata files, are curated.
One of the files, the unobserved group `2023-10-11T01:45:47.810`, has modified RA at a location with no templates.
Astrometry is also expected to fail in WCS fitting.
This visit can test pipeline fallback features.

``python/tester/upload_hsc_rc2.py``: Command line argument is the number of groups of images to send.

Expand Down