DM-40441: first batch of deprecations from RFC-949 #287
Conversation
The same code was already being called in the end, but now we're bypassing the soon-to-be-deprecated backwards-compatibility shim.
These have apparently been there from the beginning, but seem to have never been used and never documented.
This also deprecates iterable-of-TaskDef inputs to SimplePipelineExecutor.
This deprecates the 'builder' argument to make_quantum_graph in favor of the new 'builder_class' argument.
Since the QG is now frequently responsible for carrying information about registry dataset types to executing code, we don't want it to be legal to change the registry definition after making a QG and then run that QG. We also can't really stop it most of the time, but we can at least make these tests not do it.
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #287 +/- ##
==========================================
- Coverage 87.29% 86.83% -0.47%
==========================================
Files 49 49
Lines 4472 4542 +70
Branches 771 780 +9
==========================================
+ Hits 3904 3944 +40
- Misses 415 441 +26
- Partials 153 157 +4 ☔ View full report in Codecov by Sentry. |
| ): | ||
| raise | ||
| expected = self.full_butler.registry.getDatasetType(dataset_type.name) | ||
| raise ConflictingDefinitionError( |
There was a problem hiding this comment.
Since you are raising the same exception can you instead annotate the original exception?
except ConflictingDefinitionError as e:
expected = self.full_butler.registry.getDatasetType(dataset_type.name)
e.add_note(
f"DatasetType definition in registry has changed since the QuantumGraph was built: "
f"{dataset_type} (graph) != {expected} (registry)."
)
raise?
There was a problem hiding this comment.
I gave this a try, but that changes which message appears as the primary one, and in this case we want the new message to be more prominent than the original.
| except ConflictingDefinitionError: | ||
| if not _check_compatibility( |
There was a problem hiding this comment.
I assume the compatibility check is now done lower down and I haven't read the pipe_base changes yet.
There was a problem hiding this comment.
Yes, specifically in PipelineGraph.resolve.
| if builder: | ||
| warnings.warn( | ||
| "The 'builder' argument to SeparablePipelineBuilder.make_quantum_graph " | ||
| "is deprecated in favor of 'builder_class', and will be removed after v26.", |
There was a problem hiding this comment.
You want to remove it all before 27 comes out?
There was a problem hiding this comment.
No, just an old branch I forgot to update.
| quantum_graph = graph_builder.makeGraph( | ||
| pipeline, collections=butler.collections, run=butler.run, userQuery=where, bind=bind | ||
| # TODO: disable this block and adjust docs and annotations | ||
| # on DM-40443. |
There was a problem hiding this comment.
Maybe DM-40443 should be made a blocker of DM-37430? (the v27 pipelines release ticket)
There was a problem hiding this comment.
Given the errors I made in version numbers on this ticket, I'm setting the release ticket to block DM-40443 instead.
requirements.txt
Outdated
| @@ -5,7 +5,7 @@ pydantic >=2,<3.0 | |||
| networkx | |||
| git+https://github.com/lsst/resources@main#egg=lsst-resources | |||
There was a problem hiding this comment.
Please modernize all these when you go in to remove the ticket branch.
lsst-resources @ git+https://github.com/lsst/resources@main
lsst-daf-butler @ git+https://github.com/lsst/daf_butler@main
lsst-utils @ git+https://github.com/lsst/utils@main
lsst-pipe-base @ git+https://github.com/lsst/pipe_base@main
lsst-pex-config @ git+https://github.com/lsst/pex_config@main
TallJimbo
force-pushed
the
tickets/DM-40441
branch
from
2269c20
to
3c77a63
Compare
...instead of PipelineDatasetTypes. This includes a check on changes to registry dataset types between QG generation and execution, but only when executing with full butler.
Calling code in this package will be updated to avoid the new warning in a later commit.
Calling code in this package will be updated to avoid the new warning in a later commit.
This should eliminate all new deprecation warnings from internal ctrl_mpexec calls.
It's easier to deprecate the whole function than it is to migrate its return type.
I'm deferring actually removing this usage to DM-40639, since there's an opportunity to easily make a nice improvement to this code at the same time, but only after DM-39779 merges (and I don't want to increase the scope of this ticket further).
TallJimbo
force-pushed
the
tickets/DM-40441
branch
from
3c77a63
to
977d342
Compare
Co-authored-by: Tim Jenness <tjenness@lsst.org>
TallJimbo
force-pushed
the
tickets/DM-40441
branch
2 times, most recently
from
94a2200
to
734861c
Compare
TallJimbo
force-pushed
the
tickets/DM-40441
branch
from
734861c
to
fab8c00
Compare
Requires lsst/pipe_base#411
Checklist
added a release note for user-visible changes to(thedoc/changespipe_basechangelog entry covers these changes, too)