DM-39582: Decrease memory usage and load times when reading graphs

[natelust]

Checklist

• ran Jenkins
• added a release note for user-visible changes to doc/changes

[andy-slac]

Looks good.

[andy-slac]

Probably removal.md instead of api.md?

[andy-slac]

I do not entirely understand PersistenceContextVars/context vars, would it be nice if it could work like a regular context manager, e.g.:

with PersistenceContextVars():
   return self.deserializer.constructGraph(nodeSet, _readBytes, universe)

but may be it's not possible?

[natelust]

I worked on that for a while, and the outcome kept coming out more awkward, and not at all nice like that. I think it is mostly because context vars are more geared toward asyncio, callback programming style.

[codecov]

Codecov Report

Patch coverage: 66.66% and project coverage change: -0.07 ⚠️

Comparison is base (806fb6d) 82.82% compared to head (44f5f6c) 82.75%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #348      +/-   ##
==========================================
- Coverage   82.82%   82.75%   -0.07%     
==========================================
  Files          66       66              
  Lines        7242     7237       -5     
  Branches     1410     1291     -119     
==========================================
- Hits         5998     5989       -9     
- Misses        999     1000       +1     
- Partials      245      248       +3     

Impacted Files	Coverage Δ
python/lsst/pipe/base/executionButlerBuilder.py	66.21% <0.00%> (ø)
python/lsst/pipe/base/graph/_implDetails.py	44.69% <ø> (ø)
python/lsst/pipe/base/graphBuilder.py	63.68% <ø> (ø)
...ython/lsst/pipe/base/script/transfer_from_graph.py	12.50% <0.00%> (ø)
...ython/lsst/pipe/base/tests/mocks/_pipeline_task.py	34.54% <0.00%> (+0.61%)	⬆️
python/lsst/pipe/base/pipeline.py	66.05% <25.00%> (ø)
python/lsst/pipe/base/connections.py	77.85% <37.50%> (-0.28%)	⬇️
python/lsst/pipe/base/_quantumContext.py	86.39% <60.00%> (ø)
python/lsst/pipe/base/graph/quantumNode.py	88.70% <85.71%> (-0.58%)	⬇️
python/lsst/pipe/base/_instrument.py	84.92% <100.00%> (ø)
... and 5 more

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.