From 94ead1d6f2ea20e8148ad444b6fa46fae1c6b037 Mon Sep 17 00:00:00 2001 From: Krzysztof Findeisen Date: Mon, 17 Jun 2024 16:45:06 -0700 Subject: [PATCH 1/8] Update Cassandra examples for new dev keyspaces. Keyspaces like pp_apdb_latiss are now reserved for production; development spaces are suffixed with _dev. --- doc/playbook.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/playbook.rst b/doc/playbook.rst index 45559590..d8c10a38 100644 --- a/doc/playbook.rst +++ b/doc/playbook.rst @@ -528,11 +528,11 @@ To set up a new keyspace and connection, use: .. code-block:: sh apdb-cli create-cassandra sdfk8sk001.sdf.slac.stanford.edu sdfk8sk004.sdf.slac.stanford.edu \ - pp_apdb_latiss pp_apdb_latiss-dev.py --replication-factor=3 --enable-replica + pp_apdb_latiss_dev pp_apdb_latiss-dev.py --user apdb --replication-factor=3 --enable-replica apdb-cli create-cassandra sdfk8sk001.sdf.slac.stanford.edu sdfk8sk004.sdf.slac.stanford.edu \ - pp_apdb_hsc pp_apdb_hsc-dev.py --replication-factor=3 --enable-replica + pp_apdb_hsc_dev pp_apdb_hsc-dev.py --user apdb --replication-factor=3 --enable-replica apdb-cli create-cassandra sdfk8sk001.sdf.slac.stanford.edu sdfk8sk004.sdf.slac.stanford.edu \ - pp_apdb_lsstcomcamsim pp_apdb_lsstcomcamsim-dev.py --replication-factor=3 --enable-replica + pp_apdb_lsstcomcamsim_dev pp_apdb_lsstcomcamsim-dev.py --user apdb --replication-factor=3 --enable-replica Here ``sdfk8sk001.sdf.slac.stanford.edu`` and ``sdfk8sk004.sdf.slac.stanford.edu`` are two nodes within the Prompt Processing allocation, which are the ``contact_points`` used for the initial connection. All of the available nodes will be used. From a588df58f0cd606d6537b5d1f912827bdf72f428 Mon Sep 17 00:00:00 2001 From: Krzysztof Findeisen Date: Tue, 18 Jun 2024 12:10:17 -0700 Subject: [PATCH 2/8] Update db-auth guidelines to reflect existence of dev and prod APDBs. Having all keyspaces use the apdb credentials is potentially problematic. The revised example leaves room for prod APDB configs later, even though those are outside the scope of the Playbook. --- doc/playbook.rst | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/doc/playbook.rst b/doc/playbook.rst index d8c10a38..63cc5114 100644 --- a/doc/playbook.rst +++ b/doc/playbook.rst @@ -495,9 +495,14 @@ Add the following to your ``db-auth.yaml``, replacing ``PORT`` and ``PASSWORD`` .. code-block:: sh - - url: cassandra://sdfk8sk001.sdf.slac.stanford.edu:PORT/* + # Cassandra dev APDBs + - url: cassandra://sdfk8sk001.sdf.slac.stanford.edu:PORT/pp_apdb_*_dev username: apdb password: PASSWORD + # Workaround for list-cassandra not having keyspace-agnostic credentials, MUST go after all other entries + - url: cassandra://sdfk8sk001.sdf.slac.stanford.edu:PORT/* + username: ANY_CASSANDRA_ACCOUNT + password: PASSWORD and execute ``chmod 0600 ~/.lsst/db-auth.yaml``. From 275b4fcacc2bbdfd77eb6731675978131097702e Mon Sep 17 00:00:00 2001 From: Krzysztof Findeisen Date: Tue, 18 Jun 2024 12:10:38 -0700 Subject: [PATCH 3/8] Clean up .pgpass guidelines. Adding comments to the file makes it easier to understand. Users should not be using the service accounts instead of the developer account. --- doc/playbook.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/playbook.rst b/doc/playbook.rst index 63cc5114..d570b9e0 100644 --- a/doc/playbook.rst +++ b/doc/playbook.rst @@ -471,10 +471,10 @@ For passwordless login, create a ``~/.pgpass`` file with contents: .. code-block:: + # Dev APDBs usdf-prompt-processing-dev.slac.stanford.edu:5432:lsst-devl:rubin:PASSWORD - usdf-prompt-processing-dev.slac.stanford.edu:5432:ppcentralbutler:latiss_prompt:PASSWORD - usdf-prompt-processing-dev.slac.stanford.edu:5432:ppcentralbutler:hsc_prompt:PASSWORD - usdf-prompt-processing-dev.slac.stanford.edu:5432:ppcentralbutler:lsstcomcamsim_prompt:PASSWORD + # Dev central repo + usdf-prompt-processing-dev.slac.stanford.edu:5432:ppcentralbutler:pp:PASSWORD and execute ``chmod 0600 ~/.pgpass``. From ea687016cc95549c2c225a458e5e74fcc50ae428 Mon Sep 17 00:00:00 2001 From: Krzysztof Findeisen Date: Tue, 18 Jun 2024 12:15:50 -0700 Subject: [PATCH 4/8] Document using db-auth for Butler credentials. This allows developers to use db-auth-only setups if they don't need to work with Postgres. --- doc/playbook.rst | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/doc/playbook.rst b/doc/playbook.rst index d570b9e0..3266e1b8 100644 --- a/doc/playbook.rst +++ b/doc/playbook.rst @@ -473,7 +473,7 @@ For passwordless login, create a ``~/.pgpass`` file with contents: # Dev APDBs usdf-prompt-processing-dev.slac.stanford.edu:5432:lsst-devl:rubin:PASSWORD - # Dev central repo + # Dev central repo, can also go in db-auth (see below) usdf-prompt-processing-dev.slac.stanford.edu:5432:ppcentralbutler:pp:PASSWORD and execute ``chmod 0600 ~/.pgpass``. @@ -499,6 +499,10 @@ Add the following to your ``db-auth.yaml``, replacing ``PORT`` and ``PASSWORD`` - url: cassandra://sdfk8sk001.sdf.slac.stanford.edu:PORT/pp_apdb_*_dev username: apdb password: PASSWORD + # Dev central repo, can also go in .pgpass (see above) + - url: postgresql://usdf-prompt-processing-dev.slac.stanford.edu/ppcentralbutler + username: pp + password: PASSWORD # Workaround for list-cassandra not having keyspace-agnostic credentials, MUST go after all other entries - url: cassandra://sdfk8sk001.sdf.slac.stanford.edu:PORT/* username: ANY_CASSANDRA_ACCOUNT @@ -506,8 +510,6 @@ Add the following to your ``db-auth.yaml``, replacing ``PORT`` and ``PASSWORD`` and execute ``chmod 0600 ~/.lsst/db-auth.yaml``. -The credentials for the Butler (``ppcentralbutler``) can be stored in a ``~/.pgpass`` file as above, or in ``db-auth.yaml``. - Creating an APDB ---------------- From 17f317090724566817af1062cbde06c0a9576807 Mon Sep 17 00:00:00 2001 From: Krzysztof Findeisen Date: Tue, 18 Jun 2024 15:49:13 -0700 Subject: [PATCH 5/8] Write up how the APDB index works. We want labels for all our APDBs for ease of analysis and management, even if Prompt Processing itself can't read the index. --- doc/playbook.rst | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/doc/playbook.rst b/doc/playbook.rst index 3266e1b8..b092c790 100644 --- a/doc/playbook.rst +++ b/doc/playbook.rst @@ -545,6 +545,17 @@ Here ``sdfk8sk001.sdf.slac.stanford.edu`` and ``sdfk8sk004.sdf.slac.stanford.edu All of the available nodes will be used. In the above example, ``pp_apdb_latiss`` is the Cassandra keyspace (similar to schema for Postgres), and ``pp_apdb_latiss-dev.py`` is the usual APDB config. +The APDB Index +-------------- + +Standard APDBs, including those used by Prompt Processing, are registered in the file pointed to by ``$DAX_APDB_INDEX_URI``. +This file is **not** visible from Prompt Processing pods, but can be used to operate on existing DBs from ``sdfrome``. +For example, the ``dev`` LATISS APDB is registered under ``pp-dev:latiss``, and ``Apdb`` calls and ``apdb-cli`` commands can substitute ``label:pp-dev:latiss`` for the config URI everywhere except database creation. + +In most cases, there is no need to edit the registry. +If you are creating a genuinely new APDB (for example, for a new instrument), add its entry(ies) to the file. +All Prompt Processing APDBs store their config file on S3, so that the file is visible to the pods. + Resetting the APDB ------------------ From 9563151cd81690a9956dc91181a57a10bded26e4 Mon Sep 17 00:00:00 2001 From: Krzysztof Findeisen Date: Tue, 18 Jun 2024 16:54:23 -0700 Subject: [PATCH 6/8] Tweak Cassandra docs. Fixed some typos and removed the dashboard link, which is now in Slack with the others. --- doc/playbook.rst | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/doc/playbook.rst b/doc/playbook.rst index b092c790..d7863306 100644 --- a/doc/playbook.rst +++ b/doc/playbook.rst @@ -481,16 +481,12 @@ and execute ``chmod 0600 ~/.pgpass``. Cassandra ^^^^^^^^^ -We have a Cassandra cluster at the USDF on dedicated hardware, that is currently deployed in parallell across 12 nodes. +We have a Cassandra cluster at the USDF on dedicated hardware, that is currently deployed in parallel across 12 nodes. Of those, 6 are reserved for Andy Salnikov's development and testing, and 6 are available for Prompt Processing. -The status of the cluster can be monitored with the `cassandra_dashboard`_ in grafana. - -.. _cassandra_dashboard: https://grafana.slac.stanford.edu/d/d7d52e6b-e376-49dc-8ef8-e4742dd229a9/cassandra-system-metrics?orgId=1&refresh=5m - The nodes available for Prompt Processing are ``sdfk8sk001`` through ``sdfk8sk006``. To access the Cassandra cluster, you must add credentials to your ``~/.lsst/db-auth.yaml``. -The appropriate credentials are stored in the SLAC Vault, under ``rubin/usdf-apdb-dev/cassandra``. +The appropriate credentials are stored in the `SLAC Vault `_. Add the following to your ``db-auth.yaml``, replacing ``PORT`` and ``PASSWORD`` from the Vault: .. code-block:: sh From c6569930b2a4ac9c96eb1f1d06041daa6dde282b Mon Sep 17 00:00:00 2001 From: Krzysztof Findeisen Date: Tue, 18 Jun 2024 17:38:02 -0700 Subject: [PATCH 7/8] Add --init-only to pipetask command for updating dataset types. --init-only guarantees that dataset types are registered, and it's much faster than actually running the pipeline. --- doc/playbook.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/playbook.rst b/doc/playbook.rst index d7863306..ce10db3e 100644 --- a/doc/playbook.rst +++ b/doc/playbook.rst @@ -265,7 +265,7 @@ One raw was ingested, visit-defined, and kept in the development central repo, s .. code-block:: sh apdb-cli create-sql "sqlite:///apdb.db" apdb_config.py - pipetask run -b s3://rubin-pp-dev-users/central_repo -i LATISS/raw/all,LATISS/defaults,LATISS/templates -o u/username/collection -d "detector=0 and instrument='LATISS' and exposure=2023082900500 and visit_system=0" -p $PROMPT_PROCESSING_DIR/pipelines/LATISS/ApPipe.yaml -c parameters:apdb_config=apdb_config.py -c diaPipe:doPackageAlerts=False --register-dataset-types + pipetask run -b s3://rubin-pp-dev-users/central_repo -i LATISS/raw/all,LATISS/defaults,LATISS/templates -o u/username/collection -d "detector=0 and instrument='LATISS' and exposure=2023082900500 and visit_system=0" -p $PROMPT_PROCESSING_DIR/pipelines/LATISS/ApPipe.yaml -c parameters:apdb_config=apdb_config.py -c diaPipe:doPackageAlerts=False --register-dataset-types --init-only .. TODO: update pipetask call after DM-43416 From ba2db4a78f4996f8eaea6692c2fb0bd6689120a9 Mon Sep 17 00:00:00 2001 From: Krzysztof Findeisen Date: Thu, 20 Jun 2024 10:30:09 -0700 Subject: [PATCH 8/8] Mention upload_from_repo.py when a run is too large for upload.py. This message is directed towards new users, who might not know what to do when they want to run, say, 10 visits. --- python/tester/upload.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/python/tester/upload.py b/python/tester/upload.py index a43c8fb0..953b5afd 100644 --- a/python/tester/upload.py +++ b/python/tester/upload.py @@ -348,7 +348,8 @@ def upload_from_raws(kafka_url, instrument, raw_pool, src_bucket, dest_bucket, n """ if n_groups > len(raw_pool): raise ValueError(f"Requested {n_groups} groups, but only {len(raw_pool)} " - "unobserved raw groups are available.") + "unobserved raw groups are available." + "For large-scale tests, consider upload_from_repo.py.") for i, true_group in enumerate(itertools.islice(raw_pool, n_groups)): group = increment_group(instrument, group_base, i)