Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DM-37703: Use InMemoryDatasetHandle and drop DatasetRef from test #80

Merged
merged 5 commits into from
Apr 19, 2023
Merged

DM-37703: Use InMemoryDatasetHandle and drop DatasetRef from test #80

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
084f43d
Use InMemoryDatasetHandle rather than mocked implementation
timj Apr 18, 2023
0c284b3
Use a resolved DatasetRef
timj Apr 18, 2023
f84f60b
Remove the DatasetRef completely from test
timj Apr 19, 2023
a28f6bc
Update pre-commit configuration
timj Apr 19, 2023
cce5369
Add flake8 to pre-commit
timj Apr 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
10 changes: 7 additions & 3 deletions .pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v2.3.0
rev: v4.4.0
hooks:
- id: check-yaml
- id: end-of-file-fixer
- id: trailing-whitespace
- repo: https://github.com/psf/black
rev: 22.3.0
rev: 23.3.0
hooks:
- id: black
# It is recommended to specify the latest version of Python
Expand All @@ -15,7 +15,11 @@ repos:
# https://pre-commit.com/#top_level-default_language_version
language_version: "python3.10"
- repo: https://github.com/pycqa/isort
rev: 5.10.1
rev: 5.12.0
hooks:
- id: isort
name: isort (python)
- repo: https://github.com/PyCQA/flake8
rev: 6.0.0
hooks:
- id: flake8
36 changes: 3 additions & 33 deletions tests/test_catalogMatch.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,34 +29,8 @@
import pandas as pd
from lsst.analysis.tools.tasks import CatalogMatchConfig, CatalogMatchTask
from lsst.daf.base import PropertyList
from lsst.daf.butler import DatasetRef, DatasetType, DimensionUniverse, StorageClass
from lsst.meas.algorithms import ReferenceObjectLoader


class MockSourceTableRef:
"""Replicate functionality of `DeferredDatasetHandle`"""

def __init__(self, source_table, ref=None, dataId=None):
self.source_table = source_table
self.ref = ref
self.dataId = dataId

def get(self, parameters={}, **kwargs):
"""Retrieve the specified dataset using the API of the Gen3 Butler.
Parameters
----------
parameters : `dict`, optional
Parameter dictionary. Supported key is ``columns``.
Returns
-------
dataframe : `pandas.DataFrame`
dataframe, cut to the specified columns.
"""
if "columns" in parameters:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't look like this parameter is ever used because it looks like this code has been copied from an implementation that had been used for pandas DataFrame objects -- the tests pass and SimpleCatalog is not configured to understand a columns parameter.

_columns = parameters["columns"]
return self.source_table[_columns]
else:
return self.source_table.copy()
from lsst.pipe.base import InMemoryDatasetHandle


class MockDataId:
Expand All @@ -65,10 +39,6 @@ class MockDataId:
def __init__(self, region):
self.region = region

datasetDimensions = DimensionUniverse().extract(["htm7"])
datasetType = DatasetType("gaia_dr2_20200414", datasetDimensions, StorageClass("SimpleCatalog"))
self.ref = DatasetRef(datasetType, {"htm7": "mockRefCat"})
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Turns out this ref wasn't needed at all so I simply deleted it.



class TestCatalogMatch(unittest.TestCase):
"""Test CatalogMatchTask"""
Expand Down Expand Up @@ -143,7 +113,7 @@ def _make_refCat(self, starIds, starRas, starDecs, poly):
-------
refDataId : MockDataId
Object that replicates the functionality of a dataId
deferredRefCat : MockSourceTableRef
deferredRefCat : InMemoryDatasetHandle
Object that replicates the functionality of a `DeferredDatasetRef`
"""
refSchema = afwTable.SimpleTable.makeMinimalSchema()
Expand All @@ -160,7 +130,7 @@ def _make_refCat(self, starIds, starRas, starDecs, poly):
record.setDec(lsst.geom.Angle(starDecs[i], lsst.geom.degrees))
record.set(fluxKey, 1)
refDataId = MockDataId(poly)
deferredRefCat = MockSourceTableRef(refCat, ref=refDataId.ref)
deferredRefCat = InMemoryDatasetHandle(refCat, storageClass="SimpleCatalog", htm7="mockRefCat")
return refDataId, deferredRefCat

def _make_objectCat(self, starIds, starRas, starDecs):
Expand Down