DM-41164: Fix QG-load breakage from DM-41043.

[TallJimbo]

Checklist

• ran Jenkins
• added a release note for user-visible changes to doc/changes

[codecov]

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Comparison is base (15f4466) 87.74% compared to head (97581c4) 87.73%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #893      +/-   ##
==========================================
- Coverage   87.74%   87.73%   -0.02%     
==========================================
  Files         271      271              
  Lines       36358    36364       +6     
  Branches     7572     7573       +1     
==========================================
+ Hits        31903    31904       +1     
- Misses       3273     3278       +5     
  Partials     1182     1182              

Files	Coverage Δ
python/lsst/daf/butler/datastore/record_data.py	81.31% <28.57%> (-4.57%)	⬇️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[eigerx]

To set up the environment I did:

setup lsst_distrib -t d_latest
cd daf_butler/
git fetch
git checkout tickets/DM-41164
setup -j -r .
scons

Then in iPython I said:

from lsst.daf.butler import Butler
butler=Butler('/repo/main')
graph_uri='/sdf/group/rubin/shared/campaigns/HSC-RC2/cm_prod/submit/HSC/runs/RC2/w_2023_39/DM-40985/step1/group0/w00_000/HSC_runs_RC2_w_2023_39_DM-40985_step1_group0_w00_000.qgraph'
QuantumGraphExecutionReport.make_reports(butler, graph_uri)

This generated the error

File /sdf/group/rubin/sw/conda/envs/lsst-scipipe-7.0.1/share/eups/Linux64/utils/g384e8880d6+81bc2a20b4/python/lsst/utils/doImport.py:52, in doImport.<locals>.tryImport(module, fromlist, previousError)
     51 try:
---> 52     pytype = getattr(pytype, f)
     53 except AttributeError as e:

AttributeError: module 'lsst.daf.butler.core' has no attribute 'storedFileInfo'

The above exception was the direct cause of the following exception:

ImportError                               Traceback (most recent call last)
Cell In[7], line 1
----> 1 QuantumGraphExecutionReport.make_reports(butler, graph_uri)

File /sdf/group/rubin/sw/conda/envs/lsst-scipipe-7.0.1/share/eups/Linux64/pipe_base/gdde26871fd+6c82528add/python/lsst/pipe/base/execution_reports.py:376, in QuantumGraphExecutionReport.make_reports(cls, butler, graph)
    374 status_graph = networkx.DiGraph()
    375 if not isinstance(graph, QuantumGraph):
--> 376     qg = QuantumGraph.loadUri(graph)
    377 else:
    378     qg = graph

File /sdf/group/rubin/sw/conda/envs/lsst-scipipe-7.0.1/share/eups/Linux64/pipe_base/gdde26871fd+6c82528add/python/lsst/pipe/base/graph/graph.py:876, in QuantumGraph.loadUri(cls, uri, universe, nodes, graphID, minimumVersion)
    874 if uri.getExtension() in {".qgraph"}:
    875     with LoadHelper(uri, minimumVersion) as loader:
--> 876         qgraph = loader.load(universe, nodes, graphID)
    877 else:
    878     raise ValueError(f"Only know how to handle files saved as `.qgraph`, not {uri}")

File /sdf/group/rubin/sw/conda/envs/lsst-scipipe-7.0.1/share/eups/Linux64/pipe_base/gdde26871fd+6c82528add/python/lsst/pipe/base/graph/_loadHelpers.py:232, in LoadHelper.load(self, universe, nodes, graphID)
    229 # use the daf butler context vars to aid in ensuring deduplication in
    230 # object instantiation.
    231 runner = PersistenceContextVars()
--> 232 graph = runner.run(self.deserializer.constructGraph, nodeSet, _readBytes, universe)
    233 return graph

File /sdf/data/rubin/user/eiger/daf_butler/python/lsst/daf/butler/persistence_context.py:198, in PersistenceContextVars.run(self, function, *args, **kwargs)
    193 self._ctx = copy_context()
    194 # Type checkers seem to have trouble with a second layer nesting of
    195 # parameter specs in callables, so ignore the call here and explicitly
    196 # cast the result as we know this is exactly what the return type will
    197 # be.
--> 198 result = self._ctx.run(self._functionRunner, function, *args, **kwargs)  # type: ignore
    199 return cast(_T, result)

File /sdf/data/rubin/user/eiger/daf_butler/python/lsst/daf/butler/persistence_context.py:164, in PersistenceContextVars._functionRunner(self, function, *args, **kwargs)
    161     self._tokens[name] = attribute.set({})
    163 # Call the supplied function and record the result
--> 164 result = function(*args, **kwargs)
    166 # Reset all the context variables back to the state they were in before
    167 # this function was run.
    168 persistenceVars = self._getContextVars()

File /sdf/group/rubin/sw/conda/envs/lsst-scipipe-7.0.1/share/eups/Linux64/pipe_base/gdde26871fd+6c82528add/python/lsst/pipe/base/graph/_versionDeserializers.py:617, in DeserializerV3.constructGraph(self, nodes, _readBytes, universe)
    612             datasetDict.addProducer(DatasetTypeName(typeName), recreatedTaskDef)
    614 # reconstitute the node, passing in the dictionaries for the
    615 # loaded TaskDefs and dimension records. These are used to ensure
    616 # that each unique record is only loaded once
--> 617 qnode = QuantumNode.from_simple(nodeDeserialized, loadedTaskDef, universe)
    618 container[qnode.nodeId] = qnode
    619 taskToQuantumNode[loadedTaskDef[nodeTaskLabel]].add(qnode)

File /sdf/group/rubin/sw/conda/envs/lsst-scipipe-7.0.1/share/eups/Linux64/pipe_base/gdde26871fd+6c82528add/python/lsst/pipe/base/graph/quantumNode.py:155, in QuantumNode.from_simple(cls, simple, taskDefMap, universe, recontitutedDimensions)
    148 if recontitutedDimensions is not None:
    149     warnings.warn(
    150         "The recontitutedDimensions argument is now ignored and may be removed after v26",
    151         category=FutureWarning,
    152         stacklevel=find_outside_stacklevel("lsst.pipe.base"),
    153     )
    154 return QuantumNode(
--> 155     quantum=Quantum.from_simple(simple.quantum, universe),
    156     taskDef=taskDefMap[simple.taskLabel],
    157     nodeId=simple.nodeId,
    158 )

File /sdf/data/rubin/user/eiger/daf_butler/python/lsst/daf/butler/_quantum.py:447, in Quantum.from_simple(cls, simple, universe, reconstitutedDimensions)
    445 datastore_records: dict[str, DatastoreRecordData] | None = None
    446 if simple.datastoreRecords is not None:
--> 447     datastore_records = {
    448         datastore_name: DatastoreRecordData.from_simple(record_data)
    449         for datastore_name, record_data in simple.datastoreRecords.items()
    450     }
    452 quant = Quantum(
    453     taskName=simple.taskName,
    454     dataId=dataId,
   (...)
    458     datastore_records=datastore_records,
    459 )
    460 return quant

File /sdf/data/rubin/user/eiger/daf_butler/python/lsst/daf/butler/_quantum.py:448, in <dictcomp>(.0)
    445 datastore_records: dict[str, DatastoreRecordData] | None = None
    446 if simple.datastoreRecords is not None:
    447     datastore_records = {
--> 448         datastore_name: DatastoreRecordData.from_simple(record_data)
    449         for datastore_name, record_data in simple.datastoreRecords.items()
    450     }
    452 quant = Quantum(
    453     taskName=simple.taskName,
    454     dataId=dataId,
   (...)
    458     datastore_records=datastore_records,
    459 )
    460 return quant

File /sdf/data/rubin/user/eiger/daf_butler/python/lsst/daf/butler/datastore/record_data.py:243, in DatastoreRecordData.from_simple(cls, simple, universe, registry)
    241 for class_name, table_data in simple.records.items():
    242     try:
--> 243         klass = doImportType(class_name)
    244     except ImportError:
    245         # Prior to DM-41043 we were embedding a lsst.daf.butler.core
    246         # path in the serialized form, which we never wanted; fix this
    247         # one case.
    248         if class_name.endswith("StoredDatastoreItemInfo"):

File /sdf/group/rubin/sw/conda/envs/lsst-scipipe-7.0.1/share/eups/Linux64/utils/g384e8880d6+81bc2a20b4/python/lsst/utils/doImport.py:114, in doImportType(importable)
     90 def doImportType(importable: str) -> type:
     91     """Import a python type given an importable string and return it.
     92 
     93     Parameters
   (...)
    112         item could not be retrieved from the imported module.
    113     """
--> 114     imported = doImport(importable)
    115     if isinstance(imported, types.ModuleType):
    116         raise TypeError(f"Import of {importable} returned a module and not a type.")

File /sdf/group/rubin/sw/conda/envs/lsst-scipipe-7.0.1/share/eups/Linux64/utils/g384e8880d6+81bc2a20b4/python/lsst/utils/doImport.py:69, in doImport(importable)
     67 while moduleComponents:
     68     try:
---> 69         pytype = tryImport(".".join(moduleComponents), infileComponents, previousError)
     70         if not infileComponents and hasattr(pytype, moduleComponents[-1]):
     71             # This module has an attribute with the same name as the
     72             # module itself (like doImport.doImport, actually!).
     73             # If that attribute was lifted to the package, we should
     74             # return the attribute, not the module.
     75             try:

File /sdf/group/rubin/sw/conda/envs/lsst-scipipe-7.0.1/share/eups/Linux64/utils/g384e8880d6+81bc2a20b4/python/lsst/utils/doImport.py:55, in doImport.<locals>.tryImport(module, fromlist, previousError)
     53     except AttributeError as e:
     54         extra = f"({previousError})" if previousError is not None else ""
---> 55         raise ImportError(
     56             f"Could not get attribute '{f}' from '{module}' when importing '{importable}' {extra}"
     57         ) from e
     58 return pytype

ImportError: Could not get attribute 'storedFileInfo' from 'lsst.daf.butler.core' when importing 'lsst.daf.butler.core.storedFileInfo.StoredFileInfo' (No module named 'lsst.daf.butler.core.storedFileInfo')```

[eigerx]

Success!

Out[5]: QuantumGraphExecutionReport(tasks={'isr': TaskExecutionReport(failed={}, failed_upstream={}, output_datasets={'postISRCCD': DatasetTypeExecutionReport(missing_failed=set(), missing_not_produced={}, missing_upstream_failed=set(), n_produced=8858), 'isr_metadata': DatasetTypeExecutionReport(missing_failed=set(), missing_not_produced={}, missing_upstream_failed=set(), n_produced=8858), 'isr_log': DatasetTypeExecutionReport(missing_failed=set(), missing_not_produced={}, missing_upstream_failed=set(), n_produced=8858)}), 'characterizeImage': TaskExecutionReport(failed={}, failed_upstream={}, output_datasets={'icSrc': DatasetTypeExecutionReport(missing_failed=set(), missing_not_produced={}, missing_upstream_failed=set(), n_produced=8858), 'icExpBackground': DatasetTypeExecutionReport(missing_failed=set(), missing_not_produced={}, missing_upstream_failed=set(), n_produced=8858), 'icExp': DatasetTypeExecutionReport(missing_failed=set(), missing_not_produced={}, missing_upstream_failed=set(), n_produced=8858), 'characterizeImage_metadata': DatasetTypeExecutionReport(missing_failed=set(), missing_not_produced={}, missing_upstream_failed=set(), n_produced=8858), 'characterizeImage_log': DatasetTypeExecutionReport(missing_failed=set(), missing_not_produced={}, missing_upstream_failed=set(), n_produced=8858)}), 'calibrate': TaskExecutionReport(failed={}, failed_upstream={}, output_datasets={'srcMatch': DatasetTypeExecutionReport(missing_failed=set(), missing_not_produced={}, missing_upstream_failed=set(), n_produced=8858), 'src': DatasetTypeExecutionReport(missing_failed=set(), missing_not_produced={}, missing_upstream_failed=set(), n_produced=8858), 'calexpBackground': DatasetTypeExecutionReport(missing_failed=set(), missing_not_produced={}, missing_upstream_failed=set(), n_produced=8858), 'srcMatchFull': DatasetTypeExecutionReport(missing_failed=set(), missing_not_produced={}, missing_upstream_failed=set(), n_produced=8858), 'calexp': DatasetTypeExecutionReport(missing_failed=set(), missing_not_produced={}, missing_upstream_failed=set(), n_produced=8858), 'calibrate_metadata': DatasetTypeExecutionReport(missing_failed=set(), missing_not_produced={}, missing_upstream_failed=set(), n_produced=8858), 'calibrate_log': DatasetTypeExecutionReport(missing_failed=set(), missing_not_produced={}, missing_upstream_failed=set(), n_produced=8858)}), 'writePreSourceTable': TaskExecutionReport(failed={}, failed_upstream={}, output_datasets={'preSource': DatasetTypeExecutionReport(missing_failed=set(), missing_not_produced={}, missing_upstream_failed=set(), n_produced=8858), 'writePreSourceTable_metadata': DatasetTypeExecutionReport(missing_failed=set(), missing_not_produced={}, missing_upstream_failed=set(), n_produced=8858), 'writePreSourceTable_log': DatasetTypeExecutionReport(missing_failed=set(), missing_not_produced={}, missing_upstream_failed=set(), n_produced=8858)}), 'transformPreSourceTable': TaskExecutionReport(failed={}, failed_upstream={}, output_datasets={'preSourceTable': DatasetTypeExecutionReport(missing_failed=set(), missing_not_produced={}, missing_upstream_failed=set(), n_produced=8858), 'transformPreSourceTable_metadata': DatasetTypeExecutionReport(missing_failed=set(), missing_not_produced={}, missing_upstream_failed=set(), n_produced=8858), 'transformPreSourceTable_log': DatasetTypeExecutionReport(missing_failed=set(), missing_not_produced={}, missing_upstream_failed=set(), n_produced=8858)})})```