Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-38952: revive pipeline mock system and move it here (from ctrl_mpexec) #339

Merged
merged 10 commits into from Jun 9, 2023

Conversation

TallJimbo
Copy link
Member

@TallJimbo TallJimbo commented May 26, 2023

Checklist

  • ran Jenkins
  • added a release note for user-visible changes to doc/changes

@codecov
Copy link

codecov bot commented May 26, 2023

Codecov Report

Patch coverage: 61.61% and project coverage change: +0.07 🎉

Comparison is base (e026563) 82.54% compared to head (7dcc699) 82.62%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #339      +/-   ##
==========================================
+ Coverage   82.54%   82.62%   +0.07%     
==========================================
  Files          61       66       +5     
  Lines        6865     7124     +259     
  Branches     1363     1387      +24     
==========================================
+ Hits         5667     5886     +219     
- Misses        925      993      +68     
+ Partials      273      245      -28     
Impacted Files Coverage Δ
python/lsst/pipe/base/graphBuilder.py 63.91% <33.33%> (-0.35%) ⬇️
...ython/lsst/pipe/base/tests/mocks/_pipeline_task.py 34.23% <34.23%> (ø)
...ython/lsst/pipe/base/tests/mocks/_storage_class.py 51.77% <51.77%> (ø)
python/lsst/pipe/base/pipeline.py 65.98% <87.50%> (+0.92%) ⬆️
...ython/lsst/pipe/base/tests/mocks/_data_id_match.py 94.91% <94.91%> (ø)
python/lsst/pipe/base/_dataset_handle.py 98.87% <100.00%> (+0.01%) ⬆️
python/lsst/pipe/base/tests/mocks/__init__.py 100.00% <100.00%> (ø)
tests/test_dataid_match.py 100.00% <100.00%> (ø)
tests/test_testUtils.py 99.04% <100.00%> (+0.94%) ⬆️

... and 31 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

Copy link
Member

@timj timj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great.

doc/changes/DM-38952.feature.md Outdated Show resolved Hide resolved
doc/lsst.pipe.base/testing-pipelines-with-mocks.rst Outdated Show resolved Hide resolved
doc/lsst.pipe.base/testing-pipelines-with-mocks.rst Outdated Show resolved Hide resolved
doc/lsst.pipe.base/testing-pipelines-with-mocks.rst Outdated Show resolved Hide resolved
# docstring is inherited from base class
return value

def visitRangeLiteral(self, start: int, stop: int, stride: Optional[int], node: Node) -> Any:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
def visitRangeLiteral(self, start: int, stop: int, stride: Optional[int], node: Node) -> Any:
def visitRangeLiteral(self, start: int, stop: int, stride: int | None, node: Node) -> Any:

Just in case you want to take a quick modernization pass on this file -- there aren't many old-style annotations (changing to visit_range_literal might be a step too far).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point; I've updated the annotations. Changing the method names would require changing them in a bunch of places in daf_butler, too (though I guess they're at least internal APIs), so I agree that's a step too far.

python/lsst/pipe/base/tests/mocks/_pipeline_task.py Outdated Show resolved Hide resolved
python/lsst/pipe/base/tests/mocks/_pipeline_task.py Outdated Show resolved Hide resolved
python/lsst/pipe/base/tests/mocks/_pipeline_task.py Outdated Show resolved Hide resolved
if not isinstance(refs, list):
refs = [refs]
inputs_list = []
for ref in refs:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
for ref in refs:
for ref in lsst.utils.iteration.ensure_iterable(refs):

Then you don't need the 2 lines converting to a list above. Similarly in the outputRefs loop below.

python/lsst/pipe/base/tests/mocks/_storage_class.py Outdated Show resolved Hide resolved
This has bitrotted and does not work anymore, but before fixing that
I wanted to get a clean transfer of what we had before with only
import/formatting changes.

I haven't moved the mock for ButlerQuantumContext that these classes
assume, because I'll be modifying them to use the MockStorageClass and
MockDataset classes instead.
This seems to have been introduced by switching derivedComponents to
Mapping instead of dict, but that shouldn't have mattered.
This allows the instrument embedded in a pipeline to be passed into
the graph generation algorithm even when the pipeline is converted
to a sequence of TaskDef objects in advance.
@TallJimbo TallJimbo merged commit b6be565 into main Jun 9, 2023
9 of 10 checks passed
@TallJimbo TallJimbo deleted the tickets/DM-38952 branch June 9, 2023 01:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants