Skip to content

Commit

Permalink
Also allow storage class conversion on butler.put
Browse files Browse the repository at this point in the history
The infrastructure already existed to support it and there
is no particular reason why get should convert but put
should not. Doing this provides a way for newer classes
to be used in code but whilst still supporting old
dataset type definitions.
  • Loading branch information
timj committed Jan 7, 2022
1 parent b8699d7 commit 40d1c76
Show file tree
Hide file tree
Showing 4 changed files with 18 additions and 1 deletion.
2 changes: 2 additions & 0 deletions python/lsst/daf/butler/configs/storageClasses.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,8 @@ storageClasses:
pytype: lsst.skymap.BaseSkyMap
PropertySet:
pytype: lsst.daf.base.PropertySet
converters:
lsst.pipe.base.TaskMetadata: lsst.daf.base.PropertySet.from_mapping
PropertyList:
pytype: lsst.daf.base.PropertyList
IsrCalib:
Expand Down
4 changes: 4 additions & 0 deletions python/lsst/daf/butler/datastores/fileDatastore.py
Original file line number Diff line number Diff line change
Expand Up @@ -1058,6 +1058,10 @@ def _write_in_memory_to_artifact(self, inMemoryDataset: Any, ref: DatasetRef) ->
info : `StoredFileInfo`
Information describing the artifact written to the datastore.
"""
# May need to coerce the in memory dataset to the correct
# python type.
inMemoryDataset = ref.datasetType.storageClass.coerce_type(inMemoryDataset)

location, formatter = self._prepare_for_put(inMemoryDataset, ref)
uri = location.uri

Expand Down
2 changes: 2 additions & 0 deletions tests/config/basic/storageClasses.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ storageClasses:
delegate: lsst.daf.butler.tests.MetricsDelegate
parameters:
- slice
converters:
dict: lsst.daf.butler.tests.MetricsExample.makeFromDict
StructuredData:
# Data from a simple Python class
pytype: lsst.daf.butler.tests.MetricsExample
Expand Down
11 changes: 10 additions & 1 deletion tests/test_butler.py
Original file line number Diff line number Diff line change
Expand Up @@ -1352,7 +1352,7 @@ def testPruneDatasets(self):
butler.pruneDatasets([ref1, ref2, ref3], purge=True, unstore=True)

def testPytypeCoercion(self):
"""Test python type coercion on Butler.get"""
"""Test python type coercion on Butler.get and put."""

# Store some data with the normal example storage class.
storageClass = self.storageClassFactory.getStorageClass("StructuredDataNoComponents")
Expand All @@ -1366,6 +1366,15 @@ def testPytypeCoercion(self):
datasetType_ori = butler.registry.getDatasetType(datasetTypeName)
self.assertEqual(datasetType_ori.storageClass.name, "StructuredDataNoComponents")

# Put a dict and this should coerce to a MetricsExample
test_dict = {"summary": {"a": 1}, "output": {"b": 2}}
metric_ref = butler.put(test_dict, datasetTypeName, dataId=dataId, visit=424)
test_metric = butler.getDirect(metric_ref)
self.assertEqual(test_metric.summary, test_dict["summary"])
self.assertEqual(test_metric.output, test_dict["output"])
# Remove the dataset because another will be stored later in the test.
butler.pruneDatasets([metric_ref], unstore=True, purge=True)

# Now need to hack the registry dataset type definition.
# There is no API for this.
manager = butler.registry._managers.datasets
Expand Down

0 comments on commit 40d1c76

Please sign in to comment.