Small changes in Chained Datastore

Some minor changes were made in chained datastore to support the read only SasquatchDatastore going into analysis_tools. A note of the datastore was added to the datastore document.
lsst · May 3, 2023 · b2c8c2c · b2c8c2c
1 parent f655029
commit b2c8c2c
Show file tree

Hide file tree

Showing 2 changed files with 30 additions and 3 deletions.
diff --git a/doc/lsst.daf.butler/datastores.rst b/doc/lsst.daf.butler/datastores.rst
@@ -112,3 +112,20 @@ The only constraint is that all the datasets in registry are associated with at
 
 `~datastores.chainedDatastore.ChainedDatastore` has a ``datastores`` key that contains a list of datastore configurations that can match the ``datastore`` contents from other datastores.
 Additionally, a `~datastores.chainedDatastore.ChainedDatastore` can also support ``constraints`` definitions.
+
+Sasquatch Datastore
+===================
+
+The `lsst.analysis.tools` package implements a special kind of datastore to facilitate  uploading `~lsst.analysis.tools.interface.MetricMeasurementBundle`\ s to a Sasquatch instance.
+This `~lsst.analysis.tools.interface.datastore.SasquatchDatastore` is currently write only and is meant to aid dispatching `~lsst.analysis.tools.interface.MetricMeasurementBundle`\ s anytime such a dataset is put with the butler.
+Often times this datastore will be used in conjunction with both a `datastores.chainedDatastore.ChainedDatastore` and a `datastores.fileDatastore.FileDatastore`.
+In such a setup, the `~lsst.analysis.tools.interface.MetricMeasurementBundle` will be uploaded to the Sasquatch database, and then persisted to a file based location the butler can retrieve from.
+
+The supported configurations are
+
+**restProxyUrl**
+    The url where a http rest api based kafka proxy to the Sasquatch database can be found.
+**accessToken**
+    An access token that is used to authenticate to the rest api server.
+**namespace**
+    The namespace within a Sasquatch database where metrics will be uploaded.
diff --git a/python/lsst/daf/butler/datastores/chainedDatastore.py b/python/lsst/daf/butler/datastores/chainedDatastore.py
@@ -411,7 +411,9 @@ def put(self, inMemoryDataset: Any, ref: DatasetRef) -> None:
         npermanent = 0
         nephemeral = 0
         for datastore, constraints in zip(self.datastores, self.datastoreConstraints):
-            if constraints is not None and not constraints.isAcceptable(ref):
+            if (
+                constraints is not None and not constraints.isAcceptable(ref)
+            ) or not datastore.constraints.isAcceptable(ref):
                 log.debug("Datastore %s skipping put via configuration for ref %s", datastore.name, ref)
                 continue
 
@@ -570,7 +572,11 @@ def getManyURIs(
             if not missing_refs:
                 break
             for datastore in self.datastores:
-                got_uris = datastore.getManyURIs(missing_refs, p, allow_missing=True)
+                try:
+                    got_uris = datastore.getManyURIs(missing_refs, p, allow_missing=True)
+                except NotImplementedError:
+                    # some datastores may not implement generating URIs
+                    continue
                 missing_refs -= got_uris.keys()
                 uris.update(got_uris)
                 if not missing_refs:
@@ -745,7 +751,11 @@ def retrieveArtifacts(
                 # caching datastore since using an on-disk local
                 # cache is exactly what we should be doing.
                 continue
-            datastore_refs = {ref for ref in pending if datastore.exists(ref)}
+            try:
+                datastore_refs = {ref for ref in pending if datastore.exists(ref)}
+            except NotImplementedError:
+                # Some datastores may not support retrieving artifacts
+                continue
 
             if datastore_refs:
                 grouped_by_datastore[number] = datastore_refs